ia64/xen-unstable

changeset 19401:e2ada9d65bca

Xen: documentation for VT-d/SR-IOV

Add a section about how to use the SR-IOV device with VT-d.

Signed-off-by: Yu Zhao <yu.zhao@intel.com>
author Keir Fraser <keir.fraser@citrix.com>
date Thu Mar 19 10:10:59 2009 +0000 (2009-03-19)
parents 4616acf91797
children f02a528d2e56
files docs/misc/vtd.txt
line diff
     1.1 --- a/docs/misc/vtd.txt	Thu Mar 19 10:10:31 2009 +0000
     1.2 +++ b/docs/misc/vtd.txt	Thu Mar 19 10:10:59 2009 +0000
     1.3 @@ -26,7 +26,18 @@ title Xen-Fedora Core (2.6.18-xen)
     1.4          module /boot/vmlinuz-2.6.18.8-xen root=LABEL=/ ro xencons=ttyS console=tty0 console=ttyS0, pciback.hide=(01:00.0)(03:00.0)
     1.5          module /boot/initrd-2.6.18-xen.img
     1.6  
     1.7 -12) reboot system
     1.8 +    or use dynamic hiding via PCI backend sysfs interface:
     1.9 +        a) check if the driver has binded to the device
    1.10 +            ls -l /sys/bus/pci/devices/0000:01:00.0/driver
    1.11 +            ... /sys/bus/pci/devices/0000:01:00.0/driver -> ../../../../bus/pci/drivers/igb
    1.12 +        b) if yes, then unload the driver first
    1.13 +            echo -n 0000:01:00.0 >/sys/bus/pci/drivers/igb/unbind
    1.14 +        c) add the device to the PCI backend
    1.15 +            echo -n 0000:01:00.0 >/sys/bus/pci/drivers/pciback/new_slot
    1.16 +        d) let the PCI backend bind to the device
    1.17 +            echo -n 0000:01:00.0 >/sys/bus/pci/drivers/pciback/bind
    1.18 +
    1.19 +12) reboot system (not requires if you use the dynamic hiding method)
    1.20  13) add "pci" line in /etc/xen/hvm.conf for to assigned devices
    1.21          pci = [ '01:00.0', '03:00.0' ]
    1.22  15) start hvm guest and use "lspci" to see the passthru device and
    1.23 @@ -160,3 +171,82 @@ driver's view are different. As a result
    1.24  buffer specified by driver.
    1.25  
    1.26  Such devices assigned to HVM domain currently do not work.
    1.27 +
    1.28 +
    1.29 +Using SR-IOV with VT-d
    1.30 +--------------------------------
    1.31 +
    1.32 +The Single Root I/O Virtualization is a PCI Express feature supported by
    1.33 +some devices such as Intel 82576 which allows you to create virtual PCI
    1.34 +devices (Virtual Function) and assign them to the HVM guest.
    1.35 +
    1.36 +You can use latest lspci (v3.1 and above) to check if your PCIe device
    1.37 +supports the SR-IOV capability or not.
    1.38 +
    1.39 +  $ lspci -s 01:00.0 -vvv
    1.40 +
    1.41 +  01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
    1.42 +        Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
    1.43 +
    1.44 +        ...
    1.45 +
    1.46 +        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
    1.47 +                IOVCap: Migration-, Interrupt Message Number: 000
    1.48 +                IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
    1.49 +                IOVSta: Migration-
    1.50 +                Initial VFs: 8, Total VFs: 8, Number of VFs: 7, Function Dependency Link: 00
    1.51 +                VF offset: 128, stride: 2, Device ID: 10ca
    1.52 +                Supported Page Size: 00000553, System Page Size: 00000001
    1.53 +                VF Migration: offset: 00000000, BIR: 0
    1.54 +        Kernel driver in use: igb
    1.55 +
    1.56 +
    1.57 +The function that has the SR-IOV capability is also known as Physical
    1.58 +Function. You need the Physical Function driver (runs in the Dom0 and
    1.59 +controls the physical resources allocation) to enable the Virtual Function.
    1.60 +Following is the Virtual Functions associated with above Physical Function.
    1.61 +
    1.62 +  $ lspci | grep -e 01:1[01].[0246]
    1.63 +
    1.64 +  01:10.0 Ethernet controller: Intel Corporation Device 10ca (rev 01)
    1.65 +  01:10.2 Ethernet controller: Intel Corporation Device 10ca (rev 01)
    1.66 +  01:10.4 Ethernet controller: Intel Corporation Device 10ca (rev 01)
    1.67 +  01:10.6 Ethernet controller: Intel Corporation Device 10ca (rev 01)
    1.68 +  01:11.0 Ethernet controller: Intel Corporation Device 10ca (rev 01)
    1.69 +  01:11.2 Ethernet controller: Intel Corporation Device 10ca (rev 01)
    1.70 +  01:11.4 Ethernet controller: Intel Corporation Device 10ca (rev 01)
    1.71 +
    1.72 +We can tell that Physical Function 01:00.0 has 7 Virtual Functions (01:10.0,
    1.73 +01:10.2, 01:10.4, 01:10.6, 01:11.0, 01:11.2, 01:11.4). And the Virtual
    1.74 +Function PCI Configuration Space looks just like normal PCI device.
    1.75 +
    1.76 +  $ lspci -s 01:10.0 -vvv
    1.77 +
    1.78 +  01:10.0 Ethernet controller: Intel Corporation 82576 Gigabit Virtual Function
    1.79 +        Subsystem: Intel Corporation Gigabit Virtual Function
    1.80 +        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    1.81 +        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    1.82 +        Region 0: [virtual] Memory at d2840000 (64-bit, non-prefetchable) [size=16K]
    1.83 +        Region 3: [virtual] Memory at d2860000 (64-bit, non-prefetchable) [size=16K]
    1.84 +        Capabilities: [70] MSI-X: Enable+ Mask- TabSize=3
    1.85 +                Vector table: BAR=3 offset=00000000
    1.86 +                PBA: BAR=3 offset=00002000
    1.87 +        Capabilities: [a0] Express (v2) Endpoint, MSI 00
    1.88 +
    1.89 +        ...
    1.90 +
    1.91 +
    1.92 +The Virtual Function only appears after the Physical Function driver
    1.93 +is loaded. Once the Physical Function driver is unloaded. All Virtual
    1.94 +Functions associated with this Physical Function disappear.
    1.95 +
    1.96 +The Virtual Function is essentially same as the normal PCI device when
    1.97 +using it in VT-d environment. You need to hide the Virtual Function,
    1.98 +use the Virtual Function bus, device and function number in the HVM
    1.99 +guest configuration file and then boot the HVM guest. You also need the
   1.100 +Virtual Function driver which is the normal PCI device driver in the
   1.101 +HMV guest to drive the Virtual Function. The PCIe SR-IOV specification
   1.102 +requires that the Virtual Function can only support MSI/MSI-x if it
   1.103 +uses interrupt. This means you also need to enable Xen/MSI support.
   1.104 +Since the Virtual Function is dynamically allocated by Physical Function
   1.105 +driver, you might want to use the dynamic hiding method mentioned above.