I recently had an adventure getting the NVIDIA A10G to work in an Octominer X12ULTRA, a B250-based cryptomining rig repurposed for LLMs, and I thought I’d share that experience to save the next person 2 weeks of banging their head into this problem. It might work for slotting other enterprise cards that ask for huge BAR space into systems that can’t handle this ask.
The A10G requests 32GB of BAR space for itself and additional space for its virtual functions as it’s intended for virtualization via SR-IOV, all of which the B250 is unprepared to deal with. The allocation thus fails and in turn causing the NVIDIA driver to fail with the below. There seems to be no way to configure the card to ask for less from the start (the similar A40 can when used in display mode instead of in compute mode, but this cannot be configured for A10/A10G). My platform doesn’t natively support ReBAR, so I added ReBarUEFI to the BIOS which didn’t initially seem to help, but probably gave me working ReBAR support which it turned out would be useful later; the box also POSTed way faster after I did that so was useful in any event.
[ 0.295080] pci 0000:02:00.0: [10de:2237] type 00 class 0x030200 PCIe Endpoint
[ 0.295107] pci 0000:02:00.0: BAR 0 [mem 0xdd000000-0xddffffff]
[ 0.295129] pci 0000:02:00.0: BAR 1 [mem 0x2000000000-0x27ffffffff 64bit pref]
[ 0.295152] pci 0000:02:00.0: BAR 3 [mem 0x2800000000-0x2801ffffff 64bit pref]
[ 0.295214] pci 0000:02:00.0: Enabling HDA controller
[ 0.295327] pci 0000:02:00.0: PME# supported from D0 D3hot
[ 0.295399] pci 0000:02:00.0: VF BAR 0 [mem 0x00000000-0x0003ffff]
[ 0.295403] pci 0000:02:00.0: VF BAR 0 [mem 0x00000000-0x007fffff]: contains BAR 0 for 32 VFs
[ 0.295425] pci 0000:02:00.0: VF BAR 1 [mem 0x00000000-0x7fffffff 64bit pref]
[ 0.295429] pci 0000:02:00.0: VF BAR 1 [mem 0x00000000-0xfffffffff 64bit pref]: contains BAR 1 for 32 VFs
[ 0.295445] pci 0000:02:00.0: VF BAR 3 [mem 0x00000000-0x01ffffff 64bit pref]
[ 0.295449] pci 0000:02:00.0: VF BAR 3 [mem 0x00000000-0x3fffffff 64bit pref]: contains BAR 3 for 32 VFs
[ 0.295808] pci 0000:02:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:1b.7 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
...
[ 0.369107] pci 0000:00:1b.0: PCI bridge to [bus 01]
[ 0.369119] pci 0000:00:1b.0: bridge window [mem 0xde000000-0xdeffffff]
[ 0.369128] pci 0000:00:1b.0: bridge window [mem 0x2834000000-0x2839ffffff 64bit pref]
[ 0.369214] pci 0000:02:00.0: VF BAR 1 [mem size 0x1000000000 64bit pref]: can't assign; no space
[ 0.369220] pci 0000:02:00.0: VF BAR 1 [mem size 0x1000000000 64bit pref]: failed to assign
[ 0.369227] pci 0000:02:00.0: VF BAR 3 [mem size 0x40000000 64bit pref]: can't assign; no space
[ 0.369231] pci 0000:02:00.0: VF BAR 3 [mem size 0x40000000 64bit pref]: failed to assign
[ 0.369236] pci 0000:02:00.0: VF BAR 0 [mem size 0x00800000]: can't assign; no space
[ 0.369241] pci 0000:02:00.0: VF BAR 0 [mem size 0x00800000]: failed to assign
...
[ 7.269481] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
NVRM: BAR0 is 0M @ 0x0 (PCI:0000:02:00.0)
Although the initial allocation fails, I discovered configuring ReBAR to a smaller size also triggers allocation of bridge windows even if the initial attempt failed. It made me suspect that if I stage the smaller BAR space first then trigger a rescan, I could readd the card and have it work enough to be picked up by nvidia-smi and CUDA. After confirming this and hammering out some initial issues I wound up with this:
#!/bin/bash
# XXX Determine which device is the A10G first!
DEVICE_A10G=/sys/bus/pci/devices/0000:02:00.0
sudo nvidia-smi -pm 0
sudo modprobe -r nvidia
# On this platform, it seems necessary to trigger TWO rescan operations, with the second one having a smaller BAR size, to get the bridge window correctly defined.
sudo tee $DEVICE_A10G/resource1_resize <<< 10
sudo tee $DEVICE_A10G/remove <<< 1
sudo tee /sys/bus/pci/rescan <<< 1
sudo tee $DEVICE_A10G/resource1_resize <<< 8
sudo tee $DEVICE_A10G/remove <<< 1
sudo tee /sys/bus/pci/rescan <<< 1
sudo modprobe nvidia
After the rescan, I lose network connection and kernel logs suggest a timeout attempting to transmit. I suspect the Realtek NIC kernel driver doesn’t like the rescan, so I have to reload it too when I’m done:
sudo modprobe -r r8169
sudo modprobe r8169
Similar effects could plague other hardware as well; make sure your key devices work after the rescan, especially NICs, NVMe, and storage controllers.
Following this, I was able to detect the card, load a model, and generally use the card as expected for LLMs. I probably don’t have working virtual functions, but I don’t need them for this use case.
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.163.01 Driver Version: 550.163.01 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A10G Off | 00000000:02:00.0 Off | 0 |
| 0% 82C P8 31W / 200W | 21303MiB / 23028MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 Ti Off | 00000000:03:00.0 Off | Off |
| 0% 35C P8 4W / 200W | 21208MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1969 C ./llama-server 21292MiB |
| 1 N/A N/A 1969 C ./llama-server 21200MiB |
+-----------------------------------------------------------------------------------------+
Other lingering issues:
-
The A10G idles poorly, even in a normal server (R7910) that doesn’t need hacks - if anything is loaded on it, the memory automatically clocks to maximum and chews through 70-90W at idle; I have a shell script that manually sets clocks based on load and temperature, but it’d be nice to have a less hacky solution. My detection script also adds 1 second to LLM response times as the card is practically anemic until the script detects the load and opens the throttle.
-
I can’t re-enable NVIDIA persistent mode, only getting unknown error; not really a problem once I have llama-server running. Something has to keep the card open or I lose the ability to control its clocks and power consumption.