Page MenuHomeFreeBSD
Differential D8332

hyperv: enable PCIe pass-through (a.k.a. Discrete Device Assignment)
ClosedPublic
Actions

Authored by decui_microsoft.com on Oct 25 2016, 8:17 AM.
Tags
None
Referenced Files
Unknown Object (File)
Apr 1 2024, 1:13 PM
Unknown Object (File)
Feb 14 2024, 12:33 PM
Unknown Object (File)
Feb 9 2024, 10:22 AM
Unknown Object (File)
Dec 24 2023, 2:19 PM
Unknown Object (File)
Dec 23 2023, 10:08 AM
Unknown Object (File)
Dec 22 2023, 9:44 PM
Unknown Object (File)
Dec 2 2023, 5:34 AM
Unknown Object (File)
Nov 21 2023, 9:13 AM
View All 69 Files
Subscribers
None

Details

Reviewers
howard0su_gmail.com
honzhan_microsoft.com
jhb
sepherosa_gmail.com
imp
Commits
rS308725: hyperv/pcib: enable PCIe pass-through (a.k.a. Discrete Device Assignment)
Summary

The feature enables us to pass through physical PCIe devices to FreeBSD VM
running on Hyper-V (Windows Server 2016) to get near-native performance with
low CPU utilization.

The patch implements a PCI bridge driver to support the feature:

  1. the pcib driver talks to the host to discover device(s) and presents

the device(s) to FreeBSD's pci driver via PCI configuration space (note:
to access the configuration space, we don't use the standard I/O port
0xCF8/CFC method; instead, we use an MMIO-based method supplied by Hyper-V,
which is very similar to the 0xCF8/CFC method).

  1. the pcib driver allocates resources for the device(s) and initialize

the related BARs, when the device driver's attach method is invoked;

  1. the pcib driver talks to the host to create MSI/MSI-X interrupt

remapping between the guest and the host;

  1. the pcib driver supports device hot add/remove.

BTW, I also put the patch at github:
https://github.com/dcui/freebsd/commits/decui/master/1025-all-in-one

Test Plan

I tested the patch on Windows Server 2016 with Intel 82599 NIC and
Mellanox ConnectX-3 NIC. I successfully passed through the NICs
to FreeBSD VM (HEAD) and the NIC worked in the VM.

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

decui_microsoft.com updated this revision to Diff 21664.Oct 25 2016, 8:17 AM
decui_microsoft.com retitled this revision from to hyperv: enable PCIe pass-through (a.k.a. Discrete Device Assignment).
decui_microsoft.com updated this object.
decui_microsoft.com edited the test plan for this revision. (Show Details)
decui_microsoft.com updated this object.Oct 25 2016, 8:20 AM
decui_microsoft.com added reviewers: jhb, imp, sepherosa_gmail.com, honzhan_microsoft.com, howard0su_gmail.com.
decui_microsoft.com updated this revision to Diff 21879.Nov 1 2016, 11:24 AM

The changes to VMBus were moved to 2 separate patches:
https://reviews.freebsd.org/D8410, and
https://reviews.freebsd.org/D8409

decui_microsoft.com added a parent revision: D8409: hyperv/vmbus: add new vmbus methods to support PCIe pass-through.Nov 1 2016, 11:27 AM
jhb edited edge metadata.Nov 2 2016, 4:38 PM

This is an interesting approach for pass through. The only other hypervisor I'm familiar with (bhyve) does the config space emulation in the hypervisor itself so that guests don't need modifications for pass through (they just show up as PCI devices).

sys/dev/hyperv/pcib/pcib.c
1436 ↗(On Diff #21879)

In new drivers I prefer to use 'bus_read_4(hbus->cfg_res, <reg>)' rather than 'bus_space_read_4(tag, handle, <reg>)'. It's shorter and requires less space in the softc.

1539 ↗(On Diff #21879)

This will device_delete_child() any child devices I assume?

decui_microsoft.com added inline comments.Nov 3 2016, 11:33 AM
sys/dev/hyperv/pcib/pcib.c
1436 ↗(On Diff #21879)

Thanks! I'll change to the new style and I can remove some lines. :-)

1539 ↗(On Diff #21879)

Yes.
hv_pci_devices_present() ==> pci_devices_present_work() -> hv_pci_delete_device() -> device_delete_child().

decui_microsoft.com added a comment.EditedNov 3 2016, 11:43 AM
In D8332#175267, @jhb wrote:

This is an interesting approach for pass through. The only other hypervisor I'm familiar with (bhyve) does the config space emulation in the hypervisor itself so that guests don't need modifications for pass through (they just show up as PCI devices).

Yes, Hyper-V's "Para-virtualized emulation of config space" approach is special.
I guess Hyper-V team think that with this approach they need the least changes in the hypervisor, guest firmware, and device model, etc.

E.g., the motherboard chipset and PCI bus 0 emulated by Hyper-V is pretty old so it can't support PCIe, and Hyper-V UEFI VM doesn't have the legacy PCI bus 0. And, to support hot add/remove in standard way, Hyper-V may need to emulate ACPI GPE and SCI interrupts, etc.

Of course, the obvious drawback is: we have to introduce such a pcib driver into the guest...

decui_microsoft.com updated this revision to Diff 21968.Nov 3 2016, 11:54 AM
decui_microsoft.com edited edge metadata.

This new version addressed jhb's comment: "use bus_read_4(hbus->cfg_res, xxx)".

decui_microsoft.com updated this revision to Diff 22124.Nov 10 2016, 11:33 AM

Cleaned up some comments.
Ceaned up vmbus_pcib_detach() according to discussions with Hyper-V team.

No functional change.

sepherosa_gmail.com accepted this revision.Nov 14 2016, 5:40 AM
sepherosa_gmail.com edited edge metadata.
This revision is now accepted and ready to land.Nov 14 2016, 5:40 AM
Closed by commit rS308725: hyperv/pcib: enable PCIe pass-through (a.k.a. Discrete Device Assignment) (authored by dexuan). · Explain WhyNov 16 2016, 9:25 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
head/
sys/
conf/
1 line
1 line
dev/
hyperv/
pcib/
1790 lines
modules/
hyperv/
pcib/
12 lines

Diff 22252

View Options

head/sys/conf/files.amd64

Loading...
View Options

head/sys/conf/files.i386

Loading...
View Options

head/sys/dev/hyperv/pcib/pcib.c

Loading...
View Options

head/sys/modules/hyperv/pcib/Makefile

Loading...