PF passthrough: PCI config space returns 0xFF after VFIO reset on MI300X/MI355X #9913

Transferred from amd/MxGPU-Virtualization#25 at the maintainer's request. GIM is not loaded in this reproduction. The failing reset method and root cause have not been isolated.

Summary

After an AMD Instinct GPU physical function (PF) is passed through to a VM using vfio-pci, shutting down the VM leaves the PF's PCI configuration space unreadable on the host. lspci returns 0xFF for every byte, and the device cannot be rebound to amdgpu or recovered by PCI remove and rescan. A host reboot restores it.

This has been reproduced on MI300X and MI355X PFs. GIM SR-IOV VF passthrough is unaffected.

Environment

• GPUs: AMD Instinct MI355X (device ID 0x75a3) and MI300X (device ID 0x740f)
• Host: Fedora 44, kernel 7.0.12-201.fc44.x86_64
• IOMMU: AMD-Vi enabled
• QEMU: 10.0
• Hypervisor: KVM via KubeVirt / libvirt
• GIM: not loaded during reproduction

Reproduction

1. Boot with the GPU PF bound to amdgpu and no GIM loaded.

2. Bind the PF to vfio-pci (example BDF):

   echo "0000:0c:00.0" > /sys/bus/pci/drivers/amdgpu/unbind
   echo "vfio-pci" > /sys/bus/pci/devices/0000:0c:00.0/driver_override
   echo "0000:0c:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

3. Start a QEMU/KubeVirt VM with the PF assigned as a VFIO passthrough device.

4. Shut down the VM, then unbind the PF from vfio-pci and attempt to rebind it to amdgpu.

Expected: The PF rebinds to amdgpu and can be used for compute or another VM session.

Actual: Host lspci -s 0000:0c:00.0 -xxx shows all config-space bytes as 0xFF (vendor/device ID 0xFFFF:0xFFFF). Rebinding to amdgpu fails. Removing the PCI device and rescanning does not restore it; a host reboot does.

Workaround and open question

Writing an empty value to the PF's host PCI reset_method before binding it to vfio-pci prevents the observed config-space failure:

echo "" > /sys/bus/pci/devices/0000:0c:00.0/reset_method

With this workaround, the tested PF binds to vfio-pci, passes through to a VM, and rebinds to amdgpu after shutdown. It disables the device's PCI reset methods, however, so GPU state may not be fully cleaned between VM sessions. Which reset path is failing, and is the fix in the host PCI/VFIO path, GPU firmware/device behavior, or elsewhere?

Context

Discovered during on-demand VFIO passthrough work for the AMD GPU DRA driver (ROCm/k8s-gpu-dra-driver#50). The original issue includes an experimental kernel-module workaround and discussion of a potentially related VF FLR timeout; neither establishes the root cause of this PF failure.