From: Laine Stump Date: Fri, 4 Mar 2016 15:26:23 +0000 (-0500) Subject: conf: new pci controller model pci-expander-bus X-Git-Url: http://xenbits.xensource.com/gitweb?a=commitdiff_plain;h=52f3d0a4d2de2f3a633bd49b4ebc46a31329a04c;p=libvirt.git conf: new pci controller model pci-expander-bus This is a standard PCI root bus (not a bridge) that can be added to a 440fx-based domain. Although it uses a PCI slot, this is *not* how it is connected into the PCI bus hierarchy, but is only used for control. Each pci-expander-bus provides 32 slots (0-31) that can accept hotplug of standard PCI devices. The usefulness of pci-expander-bus relative to a pci-bridge is that the NUMA node of the bus can be specified with the subelement of . This gives guest-side visibility to the NUMA node of attached devices (presuming that management apps only assign a device to a bus that has a NUMA node number matching the node number of the device on the host). Each pci-expander-bus also has a "busNr" attribute. The expander-bus itself will take the busNr specified, and all buses that are connected to this bus (including the pci-bridge that is automatically added to any expander bus of model "pxb" (see the next commit)) will use busNr+1, busNr+2, etc, and the pci-root (or the expander-bus with next lower busNr) will use bus numbers lower than busNr. --- diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in index e3307eb049..43102c8d11 100644 --- a/docs/formatdomain.html.in +++ b/docs/formatdomain.html.in @@ -3102,12 +3102,13 @@ PCI controllers have an optional model attribute with possible values pci-root, pcie-root, pcie-root-port, pci-bridge, - dmi-to-pci-bridge, pcie-switch-upstream-port, or - pcie-switch-downstream-port. + dmi-to-pci-bridge, pcie-switch-upstream-port, + pcie-switch-downstream-port, or pci-expander-bus. (pci-root and pci-bridge since 1.0.5, pcie-root and dmi-to-pci-bridge since - 1.1.2, pcie-root-port, pcie-switch-upstream-port, and - pcie-switch-downstream-port since 1.2.19) + 1.1.2, pcie-root-port, pcie-switch-upstream-port, + pcie-switch-downstream-port since 1.2.19, + and pci-expander-bus since 1.3.4) The root controllers (pci-root and pcie-root) have an optional pcihole64 element specifying how big (in kilobytes, or in the unit specified by pcihole64's @@ -3131,9 +3132,9 @@

PCI controllers also have an optional - subelement <target> with the attributes - listed below. These are configurable items that 1) are visible - to the guest OS so must be preserved for guest ABI + subelement <target> with the attributes and + subelements listed below. These are configurable items that 1) + are visible to the guest OS so must be preserved for guest ABI compatibility, and 2) are usually left to default values or derived automatically by libvirt. In almost all cases, you should not manually add a <target> subelement @@ -3170,6 +3171,35 @@ which is visible to the virtual machine. If set, port must be between 0 and 255. +

busNr
+
+ pci-expander-bus controllers can have an + optional busNr attribute (1-254). This will be + the bus number of the new bus; All bus numbers between that + specified and 255 will be available only for assignment to + PCI/PCIe controllers plugged into the hierarchy starting with + this expander bus, and bus numbers less than the specified + value will be available to the next lower expander-bus (or the + root-bus if there are no lower expander buses). If you do not + specify a busNumber, libvirt will find the lowest existing + busNumber in all other expander buses (or use 256 if there are + no others) and auto-assign the busNr of that found bus - 2, + which provides one bus number for the pci-expander-bus and one + for the pci-bridge that is automatically attached to it (if + you plan on adding more pci-bridges to the hierarchy of the + bus, you should manually set busNr to a lower value). +
+
<node>
+
+ pci-expander-bus controllers can have an + optional <node> subelement within + the <target> subelement, which is used to + set the NUMA node reported to the guest OS for that bus - the + guest OS will then know that all devices on that bus are a + part of the specified NUMA node (it is up to the user of the + libvirt API to attach host devices to the correct + pci-expander-bus when assigning them to the domain). +

For machine types which provide an implicit PCI bus, the pci-root diff --git a/docs/schemas/domaincommon.rng b/docs/schemas/domaincommon.rng index c6cdc81556..c181713355 100644 --- a/docs/schemas/domaincommon.rng +++ b/docs/schemas/domaincommon.rng @@ -1786,6 +1786,8 @@ x3130-upstream xio3130-downstream + + pxb @@ -1808,7 +1810,16 @@ - + + + + + + + + + + @@ -1834,6 +1845,7 @@ pcie-root-port pcie-switch-upstream-port pcie-switch-downstream-port + pci-expander-bus diff --git a/src/conf/domain_addr.c b/src/conf/domain_addr.c index 7ee893f675..b8f897f173 100644 --- a/src/conf/domain_addr.c +++ b/src/conf/domain_addr.c @@ -51,7 +51,11 @@ virDomainPCIControllerModelToConnectType(virDomainControllerModelPCI model) return 0; case VIR_DOMAIN_CONTROLLER_MODEL_PCI_BRIDGE: - /* pci-bridge is treated like a standard PCI endpoint device, */ + case VIR_DOMAIN_CONTROLLER_MODEL_PCI_EXPANDER_BUS: + /* pci-bridge and pci-expander-bus are treated like a standard + * PCI endpoint device, because they can plug into any + * standard PCI slot. + */ return VIR_PCI_CONNECT_TYPE_PCI_DEVICE; case VIR_DOMAIN_CONTROLLER_MODEL_DMI_TO_PCI_BRIDGE: @@ -231,6 +235,13 @@ virDomainPCIAddressBusSetModel(virDomainPCIAddressBusPtr bus, bus->minSlot = 1; bus->maxSlot = VIR_PCI_ADDRESS_SLOT_LAST; break; + case VIR_DOMAIN_CONTROLLER_MODEL_PCI_EXPANDER_BUS: + bus->flags = (VIR_PCI_CONNECT_HOTPLUGGABLE | + VIR_PCI_CONNECT_TYPE_PCI_DEVICE); + bus->minSlot = 0; + bus->maxSlot = VIR_PCI_ADDRESS_SLOT_LAST; + break; + case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT: /* slots 1 - 31, no hotplug, PCIe endpoint device or * pcie-root-port only, unless the address was specified in diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index 12a3a509ee..b541b270fe 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -320,7 +320,8 @@ VIR_ENUM_IMPL(virDomainControllerModelPCI, VIR_DOMAIN_CONTROLLER_MODEL_PCI_LAST, "dmi-to-pci-bridge", "pcie-root-port", "pcie-switch-upstream-port", - "pcie-switch-downstream-port") + "pcie-switch-downstream-port", + "pci-expander-bus") VIR_ENUM_IMPL(virDomainControllerPCIModelName, VIR_DOMAIN_CONTROLLER_PCI_MODEL_NAME_LAST, @@ -329,7 +330,8 @@ VIR_ENUM_IMPL(virDomainControllerPCIModelName, "i82801b11-bridge", "ioh3420", "x3130-upstream", - "xio3130-downstream") + "xio3130-downstream", + "pxb") VIR_ENUM_IMPL(virDomainControllerModelSCSI, VIR_DOMAIN_CONTROLLER_MODEL_SCSI_LAST, "auto", @@ -1654,6 +1656,8 @@ virDomainControllerDefNew(virDomainControllerType type) def->opts.pciopts.chassisNr = -1; def->opts.pciopts.chassis = -1; def->opts.pciopts.port = -1; + def->opts.pciopts.busNr = -1; + def->opts.pciopts.numaNode = -1; break; case VIR_DOMAIN_CONTROLLER_TYPE_IDE: case VIR_DOMAIN_CONTROLLER_TYPE_FDC: @@ -7830,6 +7834,8 @@ virDomainControllerDefParseXML(xmlNodePtr node, char *chassisNr = NULL; char *chassis = NULL; char *port = NULL; + char *busNr = NULL; + int numaNode = -1; char *ioeventfd = NULL; xmlNodePtr saved = ctxt->node; int rc; @@ -7894,12 +7900,23 @@ virDomainControllerDefParseXML(xmlNodePtr node, chassisNr = virXMLPropString(cur, "chassisNr"); chassis = virXMLPropString(cur, "chassis"); port = virXMLPropString(cur, "port"); + busNr = virXMLPropString(cur, "busNr"); processedTarget = true; } } cur = cur->next; } + /* node is parsed differently from target attributes because + * someone thought it should be a subelement instead... + */ + rc = virXPathInt("string(./target/node)", ctxt, &numaNode); + if (rc == -2 || (rc == 0 && numaNode < 0)) { + virReportError(VIR_ERR_XML_ERROR, "%s", + _("invalid NUMA node in target")); + goto error; + } + if (queues && virStrToLong_ui(queues, NULL, 10, &def->queues) < 0) { virReportError(VIR_ERR_XML_ERROR, _("Malformed 'queues' value '%s'"), queues); @@ -8069,6 +8086,25 @@ virDomainControllerDefParseXML(xmlNodePtr node, goto error; } } + if (busNr) { + if (virStrToLong_i(busNr, NULL, 0, + &def->opts.pciopts.busNr) < 0) { + virReportError(VIR_ERR_XML_ERROR, + _("Invalid busNr '%s' in PCI controller"), + busNr); + goto error; + } + if (def->opts.pciopts.busNr < 0 || + def->opts.pciopts.busNr > 254) { + virReportError(VIR_ERR_XML_ERROR, + _("PCI controller busNr '%s' out of range " + "- must be 0-254"), + busNr); + goto error; + } + } + if (numaNode >= 0) + def->opts.pciopts.numaNode = numaNode; break; default: @@ -8087,6 +8123,7 @@ virDomainControllerDefParseXML(xmlNodePtr node, VIR_FREE(chassisNr); VIR_FREE(chassis); VIR_FREE(port); + VIR_FREE(busNr); VIR_FREE(ioeventfd); return def; @@ -19420,7 +19457,9 @@ virDomainControllerDefFormat(virBufferPtr buf, pciModel = true; if (def->opts.pciopts.chassisNr != -1 || def->opts.pciopts.chassis != -1 || - def->opts.pciopts.port != -1) + def->opts.pciopts.port != -1 || + def->opts.pciopts.busNr != -1 || + def->opts.pciopts.numaNode != -1) pciTarget = true; break; @@ -19456,7 +19495,19 @@ virDomainControllerDefFormat(virBufferPtr buf, if (def->opts.pciopts.port != -1) virBufferAsprintf(buf, " port='0x%x'", def->opts.pciopts.port); - virBufferAddLit(buf, "/>\n"); + if (def->opts.pciopts.busNr != -1) + virBufferAsprintf(buf, " busNr='%d'", + def->opts.pciopts.busNr); + if (def->opts.pciopts.numaNode == -1) { + virBufferAddLit(buf, "/>\n"); + } else { + virBufferAddLit(buf, ">\n"); + virBufferAdjustIndent(buf, 2); + virBufferAsprintf(buf, "%d\n", + def->opts.pciopts.numaNode); + virBufferAdjustIndent(buf, -2); + virBufferAddLit(buf, "\n"); + } } if (def->queues || def->cmd_per_lun || diff --git a/src/conf/domain_conf.h b/src/conf/domain_conf.h index 443a1d7954..76093e4b67 100644 --- a/src/conf/domain_conf.h +++ b/src/conf/domain_conf.h @@ -1,7 +1,7 @@ /* * domain_conf.h: domain XML processing * - * Copyright (C) 2006-2015 Red Hat, Inc. + * Copyright (C) 2006-2016 Red Hat, Inc. * Copyright (C) 2006-2008 Daniel P. Berrange * Copyright (c) 2015 SUSE LINUX Products GmbH, Nuernberg, Germany. * @@ -759,6 +759,7 @@ typedef enum { VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT_PORT, VIR_DOMAIN_CONTROLLER_MODEL_PCIE_SWITCH_UPSTREAM_PORT, VIR_DOMAIN_CONTROLLER_MODEL_PCIE_SWITCH_DOWNSTREAM_PORT, + VIR_DOMAIN_CONTROLLER_MODEL_PCI_EXPANDER_BUS, VIR_DOMAIN_CONTROLLER_MODEL_PCI_LAST } virDomainControllerModelPCI; @@ -770,6 +771,7 @@ typedef enum { VIR_DOMAIN_CONTROLLER_PCI_MODEL_NAME_IOH3420, VIR_DOMAIN_CONTROLLER_PCI_MODEL_NAME_X3130_UPSTREAM, VIR_DOMAIN_CONTROLLER_PCI_MODEL_NAME_XIO3130_DOWNSTREAM, + VIR_DOMAIN_CONTROLLER_PCI_MODEL_NAME_PXB, VIR_DOMAIN_CONTROLLER_PCI_MODEL_NAME_LAST } virDomainControllerPCIModelName; @@ -834,6 +836,11 @@ struct _virDomainPCIControllerOpts { * pcie-root-port/pcie-switch-downstream-port, -1 = unspecified */ int chassis; int port; + int busNr; /* used by pci-expander-bus, -1 == unspecified */ + /* numaNode is a *subelement* of target (to match existing + * item in memory target config) -1 == unspecified + */ + int numaNode; }; /* Stores the virtual disk controller configuration */ diff --git a/src/qemu/qemu_domain_address.c b/src/qemu/qemu_domain_address.c index 73a699ccee..d336b4b90f 100644 --- a/src/qemu/qemu_domain_address.c +++ b/src/qemu/qemu_domain_address.c @@ -1372,6 +1372,7 @@ qemuDomainPCIControllerSetDefaultModelName(virDomainControllerDefPtr cont) case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_SWITCH_DOWNSTREAM_PORT: *modelName = VIR_DOMAIN_CONTROLLER_PCI_MODEL_NAME_XIO3130_DOWNSTREAM; break; + case VIR_DOMAIN_CONTROLLER_MODEL_PCI_EXPANDER_BUS: case VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT: case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT: case VIR_DOMAIN_CONTROLLER_MODEL_PCI_LAST: @@ -1507,6 +1508,7 @@ qemuDomainAssignPCIAddresses(virDomainDefPtr def, if (options->port == -1) options->port = addr->slot; break; + case VIR_DOMAIN_CONTROLLER_MODEL_PCI_EXPANDER_BUS: case VIR_DOMAIN_CONTROLLER_MODEL_DMI_TO_PCI_BRIDGE: case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_SWITCH_UPSTREAM_PORT: case VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT: diff --git a/tests/qemuxml2argvdata/qemuxml2argv-pci-expander-bus.xml b/tests/qemuxml2argvdata/qemuxml2argv-pci-expander-bus.xml new file mode 100644 index 0000000000..79c959346d --- /dev/null +++ b/tests/qemuxml2argvdata/qemuxml2argv-pci-expander-bus.xml @@ -0,0 +1,167 @@ + + expander-test + 3ec6cbe1-b5a2-4515-b800-31a61855df41 + 219100 + 219100 + 16 + + hvm + + + + + + + + + + /usr/bin/qemu-system-x86_64 + + +

+ + + + + + 1 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + diff --git a/tests/qemuxml2xmloutdata/qemuxml2xmlout-pci-expander-bus.xml b/tests/qemuxml2xmloutdata/qemuxml2xmlout-pci-expander-bus.xml new file mode 100644 index 0000000000..946e94f0f0 --- /dev/null +++ b/tests/qemuxml2xmloutdata/qemuxml2xmlout-pci-expander-bus.xml @@ -0,0 +1,206 @@ + + expander-test + 3ec6cbe1-b5a2-4515-b800-31a61855df41 + 219100 + 219100 + 16 + + hvm + + + + + + + + + + + destroy + restart + destroy + + /usr/bin/qemu-system-x86_64 + + +
+ + + + + + 1 + +
+ + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + + +
+ + + diff --git a/tests/qemuxml2xmltest.c b/tests/qemuxml2xmltest.c index cbe6b7821c..78b6dbe15b 100644 --- a/tests/qemuxml2xmltest.c +++ b/tests/qemuxml2xmltest.c @@ -644,6 +644,9 @@ mymain(void) QEMU_CAPS_ICH9_AHCI, QEMU_CAPS_DEVICE_VIDEO_PRIMARY, QEMU_CAPS_VGA_QXL, QEMU_CAPS_DEVICE_QXL); + DO_TEST_FULL("pci-expander-bus", WHEN_ACTIVE, + QEMU_CAPS_DEVICE_PCI_BRIDGE, + QEMU_CAPS_DEVICE_PXB); DO_TEST_FULL("hostdev-scsi-lsi", WHEN_ACTIVE,