In other words, we are aware and I am working on it.
3 Likes
Is this a unraid only problem? I can not follow under which circumstances it is not working.
no. my issues occur with Debian 10 on Linux 4.19.
i tryed tested in unraid6.8-rc4,pve6 with patch,manjaro,result is this patch is not working 
I have spent the last couple of days on this and I have a refined patch, things still are not perfect though as there are cases where the GPU ends up in an unrecoverable state still. I need some feedback from AMD to resolve this.
From 1a723fd375a530781d344a1315c55467ae9dbb1f Mon Sep 17 00:00:00 2001
From: Geoffrey McRae <[email protected]>
Date: Wed, 27 Nov 2019 00:32:23 +1100
Subject: [PATCH] quirk: AMD Navi 10 series vendor specific reset (v2)
Signed-off-by: Geoffrey McRae <[email protected]>
---
drivers/pci/quirks.c | 132 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 132 insertions(+)
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 44c4ae1abd00..a3325beff5a6 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -3825,6 +3825,131 @@ static int delay_250ms_after_flr(struct pci_dev *dev, int probe)
return 0;
}
+/*
+ * AMD Navi 10 series GPUs require a vendor specific reset procedure.
+ * According to AMD a PSP mode 2 reset should be enough however at this
+ * time the details of how to perform this are not available to us.
+ * Instead we can signal the SMU to enter and exit BACO which has the same
+ * desired effect.
+ */
+static int reset_amd_navi10(struct pci_dev *dev, int probe)
+{
+ const int mmMP0_SMN_C2PMSG_81 = 0x16091;
+ const int mmMP1_SMN_C2PMSG_66 = 0x16282;
+ const int mmMP1_SMN_C2PMSG_82 = 0x16292;
+ const int mmMP1_SMN_C2PMSG_90 = 0x1629a;
+
+ u16 cfg;
+ resource_size_t mmio_base, mmio_size;
+ uint32_t __iomem * mmio;
+ unsigned int sol;
+ unsigned int timeout;
+
+ /*
+ * if the device has FLR return -ENOTTY indicating that we have no
+ * device-specific reset method.
+ */
+ if (pcie_has_flr(dev))
+ return -ENOTTY;
+
+ /* bus resets still cause navi to flake out */
+ dev->dev_flags |= PCI_DEV_FLAGS_NO_BUS_RESET;
+
+ if (probe)
+ return 0;
+
+ /* map BAR5 */
+ mmio_base = pci_resource_start(dev, 5);
+ mmio_size = pci_resource_len(dev, 5);
+ mmio = ioremap_nocache(mmio_base, mmio_size);
+ if (mmio == NULL) {
+ pci_disable_device(dev);
+ pci_err(dev, "Navi10: cannot iomap device\n");
+ return 0;
+ }
+
+ /* save the PCI state and enable memory access */
+ pci_read_config_word(dev, PCI_COMMAND, &cfg);
+ pci_write_config_word(dev, PCI_COMMAND, cfg | PCI_COMMAND_MEMORY);
+
+ #define SMU_WAIT() \
+ for(timeout = 1000; timeout && (readl(mmio + mmMP1_SMN_C2PMSG_90) & 0xFFFFFFFFL) == 0; --timeout) \
+ udelay(1000); \
+ if (readl(mmio + mmMP1_SMN_C2PMSG_90) != 0x1) \
+ pci_info(dev, "Navi10: SMU error 0x%x (line %d)\n", \
+ readl(mmio + mmMP1_SMN_C2PMSG_90), __LINE__);
+
+ pci_set_power_state(dev, PCI_D0);
+
+ /* it's important we wait for the SOC to be ready */
+ for(timeout = 1000; timeout; --timeout) {
+ sol = readl(mmio + mmMP0_SMN_C2PMSG_81);
+ if (sol != 0xFFFFFFFF)
+ break;
+ udelay(1000);
+ }
+
+ if (sol == 0xFFFFFFFF)
+ pci_warn(dev, "Navi10: timeout waiting for wakeup, continuing anyway\n");
+
+ /* check the sign of life indicator */
+ if (sol == 0x0) {
+ goto out;
+ }
+
+ pci_info(dev, "Navi10: performing BACO reset\n");
+
+ /* save the state around the reset */
+ pci_save_state(dev);
+
+ /* the SMU might be busy already, wait for it */
+ SMU_WAIT();
+
+ /* send PPSMC_MSG_ArmD3 with param */
+ writel(0x00, mmio + mmMP1_SMN_C2PMSG_90);
+ writel(0x00, mmio + mmMP1_SMN_C2PMSG_82); // BACO_SEQ_BACO
+ writel(0x46, mmio + mmMP1_SMN_C2PMSG_66);
+ SMU_WAIT();
+
+ /* send PPSMC_MSG_EnterBaco with param */
+ writel(0x00, mmio + mmMP1_SMN_C2PMSG_90);
+ writel(0x00, mmio + mmMP1_SMN_C2PMSG_82); // BACO_SEQ_BACO
+ writel(0x18, mmio + mmMP1_SMN_C2PMSG_66);
+ SMU_WAIT();
+
+ /* wait for the regulators to shutdown */
+ mdelay(1000);
+
+ /* send PPSMC_MSG_ExitBaco */
+ writel(0x00, mmio + mmMP1_SMN_C2PMSG_90);
+ writel(0x19, mmio + mmMP1_SMN_C2PMSG_66);
+ SMU_WAIT();
+
+ #undef SMU_WAIT
+
+ /* wait for the SOC register to become valid */
+ for(timeout = 1000; timeout; --timeout) {
+ sol = readl(mmio + mmMP0_SMN_C2PMSG_81);
+ if (sol != 0xFFFFFFFF)
+ break;
+ udelay(1000);
+ }
+
+ if (sol != 0x0) {
+ pci_err(dev, "Navi10: sol register = 0x%x\n", sol);
+ goto out;
+ }
+
+out:
+ /* unmap BAR5 */
+ iounmap(mmio);
+
+ /* restore the state and command register */
+ pci_restore_state(dev);
+ pci_write_config_word(dev, PCI_COMMAND, cfg);
+ return 0;
+}
+
static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {
{ PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_82599_SFP_VF,
reset_intel_82599_sfp_virtfn },
@@ -3836,6 +3961,13 @@ static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {
{ PCI_VENDOR_ID_INTEL, 0x0953, delay_250ms_after_flr },
{ PCI_VENDOR_ID_CHELSIO, PCI_ANY_ID,
reset_chelsio_generic_dev },
+ { PCI_VENDOR_ID_ATI, 0x7310, reset_amd_navi10 },
+ { PCI_VENDOR_ID_ATI, 0x7312, reset_amd_navi10 },
+ { PCI_VENDOR_ID_ATI, 0x7318, reset_amd_navi10 },
+ { PCI_VENDOR_ID_ATI, 0x7319, reset_amd_navi10 },
+ { PCI_VENDOR_ID_ATI, 0x731a, reset_amd_navi10 },
+ { PCI_VENDOR_ID_ATI, 0x731b, reset_amd_navi10 },
+ { PCI_VENDOR_ID_ATI, 0x731f, reset_amd_navi10 },
{ 0 }
};
--
2.20.1
8 Likes
I’ve heard that Linux 5.4 brought many stability fixes for Navi.
can anybody confirm/disprove whether or not the resetting behavior has improved in this recent kernel? or if gnif’s patches work better/worse compared to other kernel versions?
There is no relation to kernel version for the reset patches. If you’re using VFIO then there is also no difference as you’re not using the amdgpu driver either.
The reset patch uses the BACO reset sequence that is in the AMD fork of the kernel where bleeding edge development is being done, I have been working to port it from the amdgpu driver into a working standalone reset as this is what is needed for VFIO.
If you’re using Navi under linux via amdgpu, I recommend you build the bleeding edge version of the kernel that AMD are working on as Navi support is currently quite poor even in the 5.4 kernel series. (I get GPU hangs after about 34 hours of uptime)
Once this reset patch is working 100% it will be submitted upstream for inclusion in the kernel.
3 Likes
Hi gnif and all vfio’ers, I would like to share my experiences with Navi passthrough.
I consistently run into this “type 127” reset bug whenever I power off and then try powering on the vm guest. This is the specs:
Asus PRIME B450M-A with CSM disabled (to enable booting from GPU on PCH slot), latest Version 2006 BIOS
Ryzen R5 3600
Primary slot: Sapphire RX 5700 XT Nitro+
PCH slot: RX 580 2048sp (RX 570 renewal version in China) through an open-ended pcie x1 extension cable
Debian Buster with backport kernel v5.2
The passthrough of 5700 XT works well with Q35 chips. I can install Radeon drivers and play Dota 2 without major issues except the reset bug. The i440fx chipset would cause blank screen and hard lock during driver installation, so only Q35 is recommended.
I’m not very familiar with kernel development, but can help testing things out and work with some minor tweakings at least. I will try the patch in the coming few days. Gnif, I would like to know if you need more help.
I’m planning to switch to TRX40 platform in the future. It’s not cheap at all, so I have to make sure everything works before buying.
2 Likes
I’ve just tested the v2 patch with the lastest buster-packport v5.3 kernel. VM can now boot after poweroff without “type 127” issue. There are some other issues, though.
- The fresh VM boot worked flawlessly, but the second boot spent very long time on the spinning dots before getting into Windows 10 login screen. The third boot took unreasonably longer.
- Audio did not work ever since the second boot. There was no display audio output device recognized by Windows 10.
Note that on my hw setup, the GPU and the audio of my 5700 XT are on different iommu groups (both were passed through):
IOMMU Group 18:
0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [1002:731f] (rev c1)
IOMMU Group 19:
0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38]
And here is the relevant dmesg:
First power off:
[ 181.070616] vfio-pci 0000:0b:00.0: Navi10: performing BACO reset
Second power on:
[ 227.551286] vfio-pci 0000:0b:00.0: enabling device (0400 -> 0403)
[ 227.645333] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
[ 227.645352] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
[ 227.645358] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x25@0x400
[ 227.645360] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x26@0x410
[ 227.645361] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x27@0x440
[ 227.794880] AMD-Vi: Completion-Wait loop timed out
[ 227.938321] AMD-Vi: Completion-Wait loop timed out
[ 227.939285] virbr0: port 2(vnet0) entered learning state
[ 228.063311] AMD-Vi: Completion-Wait loop timed out
[ 228.188303] AMD-Vi: Completion-Wait loop timed out
[ 228.649562] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0b:00.0 address=0xff8173930]
Also note that a number of AMD-Vi: Completion-Wait loop timed out are also present during host boot up, and the kernel panics unless amd_iommu=on iommu=pt.
Apricot
2 Likes
Unfortunately no (apparent) improvement on my side.
At least those debug messages you added in the v2 patch now trigger for me:
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=29:00.0 address=0x81b590820]
AMD-Vi: Completion-Wait loop timed out
vfio-pci 0000:29:00.0: Navi10: performing BACO reset
vfio-pci 0000:29:00.0: Navi10: SMU error 0xfd (line 3982)
vfio-pci 0000:29:00.0: Navi10: SMU error 0xfd (line 3988)
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=29:00.0 address=0x81b590850]
vfio-pci 0000:29:00.0: Navi10: SMU error 0xfd (line 3996)
vfio-pci 0000:29:00.0: Navi10: sol register = 0x8e1c400
AMD-Vi: Completion-Wait loop timed out
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=29:00.0 address=0x81b5909a0]
the last two lines will repeat for a while. The GPU passed through to the VM will only show a black screen and the CPU usage will be frozen at some point.
EDIT: This is getting really complicated to repoduce but I will give it a shot.
First of all:
(1) After a reboot of the host machine I can start my WIndows 10 VM with and without a monitor attached. Without a monitor attached I can connect via some remote tech and will see a nice desktop. If a monitor is attached everything works as expected.
Now, I if I turn off the VM and turn it back on again there is a difference.
(2) With a monitor attached the VM won’t start and the cpu consumption will get stuck at some point. I have to force the VM off.
(3) Without a monitor attached the VM will start but any remote connection will only show a resolution of 800x600 and the AMD driver won’t recognize that there is a GPU available even though the windows device manager is detecting the correct navi gpu.
(4) If I now shut down the session described in (3) and plug a monitor in with the VM being shut down and start up again. The VM starts. I just successfully “restarted” the VM without rebooting the host.
Important: this does not work reliably as sometimes the VM gets stuck during boot and I get the dmesg output
vfio-pci 0000:29:00.0: vfio_bar_restore: reset recovery - restoring BARs
vfio-pci 0000:29:00.1: vfio_bar_restore: reset recovery - restoring BARs
vfio-pci 0000:29:00.0: Navi10: performing BACO reset
Or I only get 800x600 output and the AMD gpu drive won’t grab the card. Or the VM just get stuck at some point with
vfio-pci 0000:29:00.0: vfio_bar_restore: reset recovery - restoring BARs
But there is also a chance, that the VM runs just fine…
(5) If i got the VM running as described in (4) and now disconnect the monitor I get the following dmesg notification
vfio-pci 0000:29:00.0: vfio_bar_restore: reset recovery - restoring BARs
the VM is then stuck somehow and can’t be shut down. Reconnecting the monitor won’t work. As the VM is still stuck at this point.
Another Sidenote:
Binding the gpu to the amdgpu driver and trying to use after a failed VM start it results in dmesg giving me
[drm] Fence fallback timer expired on ring gfx_0.0.0
[drm] Fence fallback timer expired on ring sdma0
and glxgears does not even get to 1fps.
1 Like
Please be aware that all of this is already known and I am working with AMD to resolve this.
2 Likes
Please also be aware that a monitor is required for the GPU to be used at all, this is not a valid method of testing.
2 Likes
Hey, I just purchased a couple of Radeon VIIs to use for GPU passthrough. I’m not sure if I’m going to use ESXi or Promox yet. If there is anything I can do to help out with the effort, please share! 
XFX just dropped the price of Radeon VIIs to $500 so a gofundme may be worthwhile.
2 Likes
- patch #1 (tested on proxmox 6.0 kernel 5.0): Navi reset not working at all
- patch #2 (tested on proxmox 6.1 kernel 5.3): working on Windows 10 + last updates + AMD Adrenaline(11/12/2019). Shutdown and reboot several times working
Ubuntu 18.04 + AMD Adrenaline 19.30(November 5th 2019) working but the Navi reset fail
Hardware: XEON E5-2696V3,64GB RAM ECC, ASROCK Taichi X99, Saphire Fx5700xt blower
3 Likes
Great idea! I‘m willing to give $50.
1 Like
patch2 (tested on pve6.1 kernel5.3) still not working on win10(amd 5700).may be my patched kernel was wrong？can you share your patched kernel?
1 Like
Sorry, I know this contributes nothing, but @xracer if you could share your build, it would be a lifesaver for me. I’ve never built a kernel before and I don’t think proxmoxs’ is a good starting point.
Thank you.
Thank you for your hard work. 
Patch work for me on : Proxmox 6.0-4 kernel 5.4 !!
I have try several shutdown and reboot with a windows 10 VM and 5700XT.
Hardware: Ryzen 7 3700X, 32GB DDR4, ASUS TUF GAMING X570PLUS, Sapphire Radeon RX 5700 XT Pulse.
2 Likes
Does anyone got the patch to work reliable on Unraid? I’m running 6.8.0 rc7 (kernel 5.3.12.)
If yes, is there anywhere instruction on how to patch kernel with the fix?
If it’s already answered somewhere i’m sorry but i couldnt find it.
Also kinda newb on patching kernels
Thanks