Jump to content
RESET Forums (homeservershow.com)

PCI Passthrough for KVM


chrisaw
 Share

Recommended Posts

Hey there!

 

I am currently trying to setup a HP Microserver Gen8 to passthrough a PCI Express graphics card to a KVM instance.

 

Firstly - the hardware:

 

- HP G1610T Microserver

- Replaced stock CPU with: Intel® Xeon® CPU E31240 @ 3.30GHz (does support VT-d)

- Added a PCI Ex graphics card: Sapphire AMD Radeon HD 6450

 

So far so good - server is working with the CPU, graphics card fires up and the radeon driver detects it.

 

Now when I try to add it to a KVM instance (or even map it to the vfio-pci driver I get the following error:

 

"vfio-pci 0000:07:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor."

 

I did manage to track the issue down to a KB article by HP themselves but I see no resolution for my issue. I have tried to all of the devices 1-16 as seen in the "NIC" section of the article below but none of that helps.

 

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c04781229&sp4ts.oid=5249566

 

I have found several articles online stating that this GFX card works and can be mapped perfectly but sadly not for me!

 

I am currently running a linux kernel 4.2.2.

 

Can anyone help me out here? I'm out of ideas at this point! :(

 

Thanks!

Link to comment
Share on other sites

I seem to recall a regression that manifests this way happening around the kernel 3.17.x versions. Try 3.16.x.

I also seem to recall that some changes related to this made it into EL7.1, relating to qemu-kvm. So you may want to try an older QEMU.

 

Additionally, you should blacklist any drivers that touch the GPU on the host if you are planning to pass the device to the guest, especially with AMD GPUs. It's been a year or so since I tried using AMD GPUs with PCI passthrough on any hypervisor, but there have always been issues that render the card unusable in the guest if the host has tainted it by loading the drivers.

 

Finally, adding:

relaxed_acs_check = 1

to /etc/libvirt/qemu.conf

may help.

Edited by gordan
Link to comment
Share on other sites

Hi - thanks for the response!

 

I did try this previously with 3.16.x (debian jessie default kernel) but this did not resolve the issue.

 

I have not tried older versions of QEMU bot looking at this it looks like the issue is between the system and the kernel rather than with qemu-kvm.

 

I am not currently running libvirt (since I am running Proxmox which has its own frontend for KVM) but have tried this when I was running Foreman (libvirt based) and faced the same issue unfortunately.

 

Thanks for the ideas though - really appreciate it!

Link to comment
Share on other sites

You may want to look into how to pass

relaxed_acs_check = 1

to qemu.

 

Other than that, it ought to be possible to get it working with a kernel/qemu version combo that simply doesn't have the RMRR check implemented.

 

I am running KVM with PCI passthrough of Nvidia GPUs on _massively_ broken hardware (much more so than the G8 is ever likely to be), and have managed to get it working. More details here, if you are interested:

https://www.altechnative.net/2015/04/05/virtually-gaming-part-2-evolution-consolidation-and-move-to-kvm/

 

Other things to try:

kvm_intel kernel module option: unrestricted_guest:bool=1

Edited by gordan
Link to comment
Share on other sites

I haven't been able to figure out how to pass relaxed_acs_check to qemu/kvm unfortunately but I have added the unrestricted_guest module option and sadly that did not help.

 

The output is still the same:

 

vfio-pci 0000:07:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.

 

I have confirmed that both devices are now assigned to the correct driver on boot:

 

Kernel driver in use: vfio-pci
Kernel driver in use: vfio-pci
 
Both the audio and video drivers are blacklisted so are not fighting for control over the devices at this point.
 
I still remain confident that the issue is in the BIOS of the G8 but the "hp-scripting" based solution does not work. I suspect there is some kind of workaround in the ESXi 5.5 kernel which enables that to work since below that version others were having trouble on that too, then that update "fixed" it. Maybe they just patched out the RMRR block code. It's very tempting to just hack the kernel on my box but I'd rather achieve this "cleanly" if possible.
Link to comment
Share on other sites

  • 2 years later...

Sorry to dig up this old thread, but in case someone else find this post and is unable to work through the RMRR issue, I can confirm that you indeed need to patch the Linux kernel, but it works flawlessly afterwards. The RMRR restriction is Linux kernel specific, which explains why this works under ESXi, because of a patch that RedHat explicitly included because this causes major issues under specific conditions. This is explained at https://access.redhat.com/articles/1434873

 

More information and details on how to compile a custom kernel that skips the RMRR check at https://forum.proxmox.com/threads/tutorial-compile-proxmox-ve-5-with-patched-intel-iommu-driver-to-remove-rmrr-check.36374/

  • Like 2
Link to comment
Share on other sites

Sorry to dig up this old thread, but in case someone else find this post and is unable to work through the RMRR issue, I can confirm that you indeed need to patch the Linux kernel, but it works flawlessly afterwards. The RMRR restriction is Linux kernel specific, which explains why this works under ESXi, because of a patch that RedHat explicitly included because this causes major issues under specific conditions. This is explained at https://access.redhat.com/articles/1434873
 
More information and details on how to compile a custom kernel that skips the RMRR check at https://forum.proxmox.com/threads/tutorial-compile-proxmox-ve-5-with-patched-intel-iommu-driver-to-remove-rmrr-check.36374/
Thanks!
I am just starting to play with Proxmox.
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...