1 Introduction

The low-level libxenctrl library currently has an unstable API and ABI and some of the hypervisor interfaces which it exposes are similarly unstable.

However several external projects use some of these interfaces (at least: qemu and kexec-tools), which presents problems for distros and other consumers. In particular the need for spurious rebuilds of those components against newer versions of Xen and difficulty supporting parallel installation of different versions of Xen (useful during upgrade).

This document considers whether parts of libxenctrl can be split out into new libraries with more useful API and ABI guarantees.

2 ABI/API Compatibility Classes

Compatibility opportunities:

• LAPI -- Library API
• LABI -- Library ABI
• HABI -- Hypervisor ABI (includes ioctls).

Each can either be Stable (S), Unstable (U) or don't care (x, because made moot by higher level interface class, e.g. no real point from an application PoV to a Stable HABI behind a unstable LABI).

Stable vs. Unstable is across major hypervisor version bump, always aim to be stable across point releases.

For libraries Stable means SONAME major component, but forward compatible only (i.e. old app on new library works, new app on old library may not link due to e.g. new symbols. This is the normal SONAME expectation). XXX find a link to the sort of scheme I mean.

HABI may include ioctls used to access those ABIs, typically these are already required to be stable by the relevant OS maintainers.

A library interface may fall into one of these categories (I expect there are others and we may not want any library to use some of even these):

• Unstable LAPI (Uxx)
  • The Wild West
  • Current Examples: libxenctrl
• Stable LAPI, Unstable LABI (SUx)
  • Requires application rebuild for a new Xen version, but not application source code changes.
  • Current Examples: libxenlight
• Stable LAPI, Stable LABI, Unstable HABI (SSU)
  • Library can be switched out via dynamic linking across hypervisor upgrade (mechanism TBD, pos. distro specific, e.g. symlink switched on boot). Requires application/daemon restart but not rebuild (but changing hypervisor version involves a reboot anyway).
  • Current Examples: None??
• Stable LAPI, Stable LABI, Stable HABI (SSS)
  • Applications linked again a library will function against any hypervisor version.
  • Current Examples: None??

3 Goal

Provide SSU or SSS interfaces for major external consumers of current libxenctrl functionality.

Out of scope (for now): SSU or SSS interfaces for consumers of libxenlight. Rationale: Lets focus on fixing external consumers of libxenctrl first.

4 Known External Consumers of libxenctrl

• qemu
• kexec tools
• in guest tools e.g. users of libxenstore, libvchan, and by extension grant table and event channel functionality. NB: libxenstore is already SSU or SSS (XXX?)

5 libxenguest

Uses some of the same sources as libxenctrl. Could do with some rationalisation of what lives where.

Not a candidate for API/ABI stability at this point.

6 Stable libraries

These libraries will provide a stable API to all consumers going forward. They can be mixed and matched with Xen version.

6.1 libxentoollog: logging

The following do not depend on any underlying Xen functionality.

6.2 libxenevtchn: event channels

Interacting with Event Channels via /dev/xen/evtchn.

6.3 libxengnttab: grant tables

Interacting with Grant Tables via /dev/xen/gnt{shr,alloc}.

xengnt???_*_notify also use IOCTL_GNTALLOC_SET_UNMAP_NOTIFY, IOCTL_GNTALLOC_SET_UNMAP_NOTIFY, etc

6.4 libxencall: making hypercalls

Making hypercalls and allocating suitable memory to use as hypercall pointer ("GUEST_HANDLE") arguments.

6.5 libxenforeignmemory: privileged foreign mappings

Mapping foreign guest memory.

6.6 libxendevicemodel: functionality for x86/HVM device models

7 Unstable libraries

These libraries do not provide a stable interface and are required to be updated in lockstep with the hypervisor.

7.1 libxenctrl

Gathered by:

$ objdump -T tools/libxc/libxenctrl.so| awk '$4 ~ /\.text/ { print $7 }' | sort -u

The following proposes some functional groupings via some proposed split library names. In some cases we may also wish to consider replacing an API with one which can be properly maintained going forwards. e.g.:

• perhaps replacing domctl's used by qemu with new stable hypercall ABIs and reflecting that in new library APIs.
• perhaps exposing more constrained versions of some broad interfaces for external users.

In addition to these there are many inlines in xenctrl.h. Where possible these should become either private functions or regular functions exported by the library.

Notes:

• Things marked "private" are generally defined in xc_private.h or are otherwise supposedly internal but leaked.
• Things marked "external, via ..." are generally expected to be used via ... (often a macro).
• There are lots of static functions in the xc_* namespace.

7.1.1 Logging

The following are already logically somewhat separate. They do not depend on any underlying Xen functionality.

7.1.2 Basic interface

- xc_interface_close
- xc_interface_open

7.1.3 Event channels

Interacting with Event Channels via __HYPERVISOR_event_channel_op (EVTCHNOP_*).

7.1.4 Grant tables

Interacting with Grant Tables via __HYPERVISOR_grant_table_op (GNTTABOP_*)`.

xc_*_notify also use IOCTL_GNTALLOC_SET_UNMAP_NOTIFY, IOCTL_GNTALLOC_SET_UNMAP_NOTIFY, etc

7.1.5 Hypercalls

7.1.5.1 Wrappers

7.1.5.2 Hypercall buffers

Allocate and manange (and attempt to enforce the use of) memory suitable for passing to a hypercall as a GUEST_HANDLE argument.

The following are external via a corresponding xc_ function:

The following are private:

• xc__hypercall_bounce_post
• xc__hypercall_bounce_pre
• xc__hypercall_buffer_cache_release

7.1.6 Mapping foreign memory

All implemented in terms of xenforeignmemory_map.

7.1.7 Managing guest memory map

7.1.8 Kexec

Functions specifically for use by kexec tools

7.1.9 Unclassified remains

Note: Not everything which belongs in one of the above categorisations has actually been moved to a suitable home.

Symbols:

- bitmap_64_to_byte
- bitmap_byte_to_64
- discard_file_cache
- read_exact
- write_exact
- writev_exact
- xc_add_mmu_update
- xc_alloc_mmu_updates
- xc_assign_device
- xc_assign_dt_device
- xc_availheap
- xc_clear_domain_pages
- xc_clear_last_error
- xc_copy_to_domain_page
- xc_core_arch_auto_translated_physmap
- xc_core_arch_get_scratch_gpfn
- xc_core_arch_gpfn_may_present
- xc_core_arch_map_p2m
- xc_core_arch_map_p2m_writable
- xc_core_arch_memory_map_get
- xc_core_shdr_get
- xc_core_shdr_set
- xc_cpumap_alloc
- xc_cpumap_clearcpu
- xc_cpumap_setcpu
- xc_cpumap_testcpu
- xc_cpu_offline
- xc_cpu_online
- xc_cpupool_addcpu
- xc_cpupool_create
- xc_cpupool_destroy
- xc_cpupool_freeinfo
- xc_cpupool_getinfo
- xc_cpupool_infofree
- xc_cpupool_movedomain
- xc_cpupool_removecpu
- xc_cputopoinfo
- xc_deassign_device
- xc_deassign_dt_device
- xc_disable_turbo
- xc_domain_bind_pt_irq
- xc_domain_bind_pt_isa_irq
- xc_domain_bind_pt_pci_irq
- xc_domain_bind_pt_spi_irq
- xc_domain_cacheflush
- xc_domain_claim_pages
- xc_domain_create
- xc_domain_create_config
- xc_domain_debug_control
- xc_domain_decrease_reservation
- xc_domain_decrease_reservation_exact
- xc_domain_destroy
- xc_domain_disable_migrate
- xc_domain_dumpcore
- xc_domain_dumpcore_via_callback
- xc_domain_get_cpu_usage
- xc_domain_get_guest_width
- xc_domain_getinfo
- xc_domain_getinfolist (`XEN_SYSCTL_getdomaininfolist`). Used by qemu-builder
- xc_domain_get_machine_address_size
- xc_domain_get_pod_target
- xc_domain_get_tsc_info
- xc_domain_hvm_getcontext
- xc_domain_hvm_getcontext_partial
- xc_domain_hvm_setcontext
- xc_domain_increase_reservation
- xc_domain_increase_reservation_exact
- xc_domain_iomem_permission
- xc_domain_ioport_mapping
- xc_domain_ioport_permission
- xc_domain_irq_permission
- xc_domain_maximum_gpfn
- xc_domain_max_vcpus
- xc_domain_memory_exchange_pages
- xc_domain_memory_mapping
- xc_domain_node_getaffinity
- xc_domain_node_setaffinity
- xc_domain_nr_gpfns
- xc_domain_p2m_audit
- xc_domain_pause
- xc_domain_pin_memory_cacheattr (`XEN_DOMCTL_pin_mem_cacheattr`). Used by qemu-dm.
- xc_domain_resume
- xc_domain_send_trigger
- xc_domain_set_access_required
- xc_domain_setdebugging
- xc_domain_sethandle
- xc_domain_set_machine_address_size
- xc_domain_set_max_evtchn
- xc_domain_setmaxmem (`XEN_DOMCTL_max_mem`). Used by qemu-dm.
- xc_domain_set_memmap_limit
- xc_domain_set_memory_map
- xc_domain_set_pod_target
- xc_domain_set_target
- xc_domain_set_time_offset
- xc_domain_set_tsc_info
- xc_domain_set_virq_handler
- xc_domain_setvnuma
- xc_domain_shutdown (`SCHEDOP_remote_shutd`).
- xc_domain_subscribe_for_suspend
- xc_domain_suppress_spurious_page_faults
- xc_domain_unbind_msi_irq
- xc_domain_unbind_pt_irq
- xc_domain_unbind_pt_spi_irq
- xc_domain_unpause
- xc_domain_update_msi_irq
- xc_enable_turbo
- xc_error_code_to_desc
- xc_ffs16
- xc_ffs32
- xc_ffs64
- xc_ffs8
- xc_flask_access
- xc_flask_add_device
- xc_flask_add_iomem
- xc_flask_add_ioport
- xc_flask_add_pirq
- xc_flask_avc_cachestats
- xc_flask_avc_hashstats
- xc_flask_context_to_sid
- xc_flask_del_device
- xc_flask_del_iomem
- xc_flask_del_ioport
- xc_flask_del_pirq
- xc_flask_getavc_threshold
- xc_flask_getbool_byid
- xc_flask_getbool_byname
- xc_flask_getenforce
- xc_flask_load
- xc_flask_op
- xc_flask_policyvers
- xc_flask_relabel_domain
- xc_flask_setavc_threshold
- xc_flask_setbool
- xc_flask_setenforce
- xc_flask_sid_to_context
- xc_flush_mmu_updates
- xc_get_cpufreq_avgfreq
- xc_get_cpufreq_para
- xc_get_cpuidle_max_cstate
- xc_getcpuinfo
- xc_get_cpumap_size
- xc_get_device_group
- xc_get_last_error
- xc_get_machine_memory_map. Used by kexec-tools
- xc_get_max_cpus. Used by kexec-tools
- xc_get_max_nodes
- xc_get_mem_access
- xc_get_nodemap_size
- xc_get_online_cpus
- xc_get_pfn_list
- xc_get_pfn_type_batch
- xc_get_tot_pages
- xc_get_vcpu_migration_delay
- xc_hvm_inject_msi (`HVMOP_inject_msi`). Used by qemu-dm.
- xc_hvm_inject_trap
- xc_hvm_modified_memory (`HVMOP_modified_memory`). Used by qemu-dm.
- xc_hvm_param_get (`HVMOP_get_param`)
- xc_hvm_param_set (`HVMOP_set_param`)
- xc_hvm_set_isa_irq_level (`HVMOP_set_isa_irq_level`). Used by qemu-dm.
- xc_hvm_set_mem_type (`HVMOP_set_mem_type`). Used by qemu-dm.
- xc_hvm_set_pci_intx_level (`HVMOP_set_pci_intx_level`). Used by qemu-dm.
- xc_hvm_set_pci_link_route (`HVMOP_set_pci_link_route`). Used by qemu-dm.
- xc_hvm_track_dirty_vram (`HVMOP_track_dirty_vram`). Used by qemu-dm.
- xc_lockprof_query
- xc_lockprof_query_number
- xc_lockprof_reset
- xc_machphys_mfn_list
- xc_make_page_below_4G
- xc_map_domain_meminfo
- xc_maximum_ram_page
- xc_mca_op
- xc_mem_access_disable_emulate
- xc_mem_access_enable_emulate
- xc_memalign
- xc_mem_paging_disable
- xc_mem_paging_enable
- xc_mem_paging_evict
- xc_mem_paging_load
- xc_mem_paging_nominate
- xc_mem_paging_prep
- xc_mem_paging_resume
- xc_memshr_add_to_physmap
- xc_memshr_audit
- xc_memshr_control
- xc_memshr_debug_gfn
- xc_memshr_debug_gref
- xc_memshr_domain_resume
- xc_memshr_nominate_gfn
- xc_memshr_nominate_gref
- xc_memshr_ring_disable
- xc_memshr_ring_enable
- xc_memshr_share_gfns
- xc_memshr_share_grefs
- xc_mmuext_op
- xc_monitor_disable
- xc_monitor_enable
- xc_monitor_mov_to_msr
- xc_monitor_resume
- xc_monitor_singlestep
- xc_monitor_software_breakpoint
- xc_monitor_write_ctrlreg
- xc_nodemap_alloc
- xc_numainfo
- xc_pcitopoinfo
- xc_perfc_query
- xc_perfc_query_number
- xc_perfc_reset
- xc_physdev_map_pirq
- xc_physdev_map_pirq_msi
- xc_physdev_pci_access_modify
- xc_physdev_unmap_pirq
- xc_physinfo
- xc_pm_get_cxstat
- xc_pm_get_max_cx
- xc_pm_get_max_px
- xc_pm_get_pxstat
- xc_pm_reset_cxstat
- xc_pm_reset_pxstat
- xc_psr_cmt_attach
- xc_psr_cmt_detach
- xc_psr_cmt_enabled
- xc_psr_cmt_get_data
- xc_psr_cmt_get_domain_rmid
- xc_psr_cmt_get_l3_cache_size
- xc_psr_cmt_get_l3_event_mask
- xc_psr_cmt_get_l3_upscaling_factor
- xc_psr_cmt_get_total_rmid
- xc_readconsolering
- xc_report
- xc_report_error
- xc_report_progress_single
- xc_report_progress_step
- xc_reportv
- xc_resource_op
- xc_sched_arinc653_schedule_get
- xc_sched_arinc653_schedule_set
- xc_sched_credit2_domain_get
- xc_sched_credit2_domain_set
- xc_sched_credit_domain_get
- xc_sched_credit_domain_set
- xc_sched_credit_params_get
- xc_sched_credit_params_set
- xc_sched_id
- xc_sched_rtds_domain_get
- xc_sched_rtds_domain_set
- xc_sedf_domain_get
- xc_sedf_domain_set
- xc_send_debug_keys
- xc_set_broken_page_p2m
- xc_set_cpufreq_gov
- xc_set_cpufreq_para
- xc_set_cpuidle_max_cstate
- xc_set_mem_access
- xc_set_progress_prefix
- xc_set_sched_opt_smt
- xc_set_vcpu_migration_delay
- xc_shadow_control
- xc_sharing_freed_pages
- xc_sharing_used_frames
- xc_strerror
- xc_tbuf_disable
- xc_tbuf_enable
- xc_tbuf_get_size
- xc_tbuf_set_cpu_mask
- xc_tbuf_set_evt_mask
- xc_tbuf_set_size
- xc_test_assign_device
- xc_test_assign_dt_device
- xc_tmem_auth
- xc_tmem_control
- xc_tmem_control_oid
- xc_tmem_restore
- xc_tmem_restore_extra
- xc_tmem_save
- xc_tmem_save_done
- xc_tmem_save_extra
- xc_translate_foreign_address
- xc_unmap_domain_meminfo
- xc_vcpu_getaffinity
- xc_vcpu_getcontext
- xc_vcpu_getinfo
- xc_vcpu_setaffinity
- xc_vcpu_setcontext
- xc_version. Used by kexec-tools
- xc_vm_event_control
- xc_vm_event_enable
- xc_watchdog

7.2 libxenguest

Gathered by:

$ objdump -T tools/libxc/libxenguest.so| awk '$4 ~ /\.text/ { print $7 }' | sort -u

- arch_setup_bootearly
- arch_setup_bootlate
- arch_setup_meminit
- cr3_to_mfn
- csum_page
- dhdr_type_to_str
- dump_bad_pseudophysmap_entry
- elf_access_ok
- elf_access_unsigned
- elf_call_log_callback
- elf_check_broken
- elf_get_ptr
- elf_init
- elf_is_elfbinary
- elf_load_binary
- elf_lookup_addr
- elf_mark_broken
- elf_memcpy_safe
- elf_memset_safe
- elf_note_desc
- elf_note_name
- elf_note_next
- elf_note_numeric
- elf_note_numeric_array
- elf_parse_binary
- elf_parse_bsdsyms
- elf_phdr_by_index
- elf_phdr_count
- elf_phdr_is_loadable
- elf_round_up
- elf_section_end
- elf_section_name
- elf_section_start
- elf_segment_end
- elf_segment_start
- elf_set_log
- elf_shdr_by_index
- elf_shdr_by_name
- elf_shdr_count
- elf_strfmt
- elf_strval
- elf_sym_by_index
- elf_sym_by_name
- elf_xen_parse
- elf_xen_parse_features
- elf_xen_parse_guest_info
- elf_xen_parse_note
- handle_tsc_info
- lz4_decompress_unknownoutputsize
- mfn_in_pseudophysmap
- mfn_to_cr3
- mfn_to_pfn
- pin_table
- populate_pfns
- rec_type_to_str
- write_split_record
- write_tsc_info
- x86_pv_domain_info
- x86_pv_map_m2p
- xc_await_suspend
- xc_compression_add_page
- xc_compression_compress_pages
- xc_compression_create_context
- xc_compression_free_context
- xc_compression_reset_pagebuf
- xc_compression_uncompress_page
- xc_cpuid_apply_policy
- xc_cpuid_check
- xc_cpuid_set
- xc_cpuid_to_str
- xc_domain_get_native_protocol
- xc_domain_restore
- xc_domain_save
- xc_dom_allocate
- xc_dom_alloc_page
- xc_dom_alloc_segment
- xc_dom_boot_domU_map
- xc_dom_boot_image
- xc_dom_boot_mem_init
- xc_dom_boot_xen_init
- xc_dom_build_image
- xc_dom_check_gzip
- xc_dom_compat_check
- xc_dom_devicetree_file
- xc_dom_devicetree_max_size
- xc_dom_devicetree_mem
- xc_dom_do_gunzip
- xc_dom_feature_translated
- xc_dom_find_arch_hooks
- xc_dom_gnttab_hvm_seed
- xc_dom_gnttab_init
- xc_dom_gnttab_seed
- xc_dom_kernel_check_size
- xc_dom_kernel_file
- xc_dom_kernel_max_size
- xc_dom_kernel_mem
- xc_dom_linux_build
- xc_dom_loginit
- xc_dom_log_memory_footprint
- xc_dom_malloc
- xc_dom_malloc_filemap
- xc_dom_malloc_page_aligned
- xc_dom_mem_init
- xc_dom_panic_func
- xc_dom_parse_image
- xc_dom_pfn_to_ptr
- xc_dom_pfn_to_ptr_retcount
- xc_dom_printf
- xc_dom_rambase_init
- xc_dom_ramdisk_check_size
- xc_dom_ramdisk_file
- xc_dom_ramdisk_max_size
- xc_dom_ramdisk_mem
- xc_dom_register_arch_hooks
- xc_dom_register_external
- xc_dom_register_loader
- xc_dom_release
- xc_dom_strdup
- xc_dom_try_gunzip
- xc_dom_unmap_all
- xc_dom_unmap_one
- xc_dom_update_guest_p2m
- xc_elf_set_logfile
- xc_exchange_page
- xc_get_bit_size
- xc_hvm_build
- xc_hvm_build_target_mem
- xc_inflate_buffer
- xc_linux_build
- xc_linux_build_mem
- xc_map_m2p
- xc_mark_page_offline
- xc_mark_page_online
- xc_query_page_offline_status
- xc_read_image
- xc_try_lz4_decode

Use privcmd and evtchn device handles:

- xc_suspend_evtchn_init_exclusive. Used by xen-hptool and libxl.
- xc_suspend_evtchn_init_sane. Unused
- xc_suspend_evtchn_release. Used by xen-hptool and libxl.

8 Consumers of libxenctrl functionality

8.1 qemu

$ nm tools/qemu-xen-dir-remote/i386-softmmu/qemu-system-i386 | grep \\bU.xc_

- xc_domain_add_to_physmap
- xc_domain_bind_pt_pci_irq
- xc_domain_create
- xc_domain_destroy
- xc_domain_getinfo
- xc_domain_ioport_mapping
- xc_domain_max_vcpus
- xc_domain_memory_mapping
- xc_domain_pin_memory_cacheattr
- xc_domain_populate_physmap_exact
- xc_domain_setmaxmem
- xc_domain_shutdown
- xc_domain_unbind_msi_irq
- xc_domain_unbind_pt_irq
- xc_domain_unpause
- xc_domain_update_msi_irq
- xc_evtchn_alloc_unbound
- xenevtchn_bind_interdomain
- xenevtchn_close
- xenevtchn_fd
- xenevtchn_notify
- xenevtchn_open
- xenevtchn_pending
- xenevtchn_unbind
- xenevtchn_unmask
- xc_hvm_inject_msi
- xc_hvm_modified_memory
- xc_hvm_set_isa_irq_level
- xc_hvm_set_mem_type
- xc_hvm_set_pci_intx_level
- xc_hvm_set_pci_link_route
- xc_hvm_track_dirty_vram
- xc_interface_close
- xc_interface_open
- xc_linux_build
- xenforeignmemory_map
- xenforeignmemory_open
- xenforeignmemory_unmap
- xc_physdev_map_pirq
- xc_physdev_map_pirq_msi
- xc_physdev_unmap_pirq
- xendevicemodel_shutdown
- xendevicemodel_reboot
- xendevicemodel_s3_suspend
- xendevicemodel_s3_awaken

8.1.1 By functional area

As a reference, this is a split of the xc_ calls made by QEMU, based on functionality areas

"S" column is whether the interface to the left is a stable ABI or not.

8.1.1.1 Device Model

8.1.1.2 PCI Passthrough

NB: More might be used by libxl in the future e.g. for ARM or PVH passthrough?

8.1.1.3 PV backends

All stable interfaces:

* xenevtchn_bind_interdomain (`IOCTL_EVTCHN_BIND_INTERDOMAIN`)
* xenevtchn_notify (`IOCTL_EVTCHN_NOTIFY`)
* xenevtchn_open (`open(2)`)
* xenevtchn_pending (`read(2)`)
* xenevtchn_unbind
* xenevtchn_unmask (`write(2)`)
* xengnttab_close
* xengnttab_map_grant_ref
* xengnttab_munmap
* xengnttab_open
* xengnttab_set_max_grants
* xenforeignmemory_map
* xenforeignmemory_open
* xenforeignmemory_unmap

8.1.1.4 PV domain builder

No stable interfaces, obsolete and can be disabled during QEMU build

* xc_domain_create
* xc_domain_destroy
* xc_domain_getinfolist
* xc_domain_max_vcpus
* xc_domain_setmaxmem (`XEN_DOMCTL_max_mem`)
* xc_domain_unpause
* xc_evtchn_alloc_unbound
* xc_linux_build

8.2 kexec-tools

xc_kexec_exec N KEXEC_CMD_kexec Y None xc_kexec_get_range N KEXEC_CMD_kexec_get_range Y None xc_kexec_load N KEXEC_CMD_kexec_load Y None xc_kexec_unload N KEXEC_CMD_kexec_unload Y None

9 Foreign memory mapping functions

The xc_map_foreign_* functions were once a bit of a twisty maze, but now all are implemented in terms of a single underlying per-OS function.

osdep level interfaces:

compat means using osdep_map_foreign_batch layer

libxenctrl level interfaces:

osdep: * have been collapsed since osdep layer was removed.

xc_map_foreign_bulk can be an internal compat helper used by oses which have not implemented their own version. It used to use xc_map_foreign_batch which is now an osdep for those OSes which need it.

xc_map_foreign_bulk should become an underlying stable API.