TrueNAS Scale PCI Pass through issue

Good afternoon the tl;dr is that I cannot figure out why the PCI pass through in my TrueNAS scale install has stopped working. I’ve tried a lot of different things including reinstalling TrueNAS scale, moving the card to a different PCI slot and about a dozen other things which I have described in detail below.

I have also described all the different things I did that led up to this issue and all the solutions I’ve tried so far to diagnose and remedy. At this point though I have no idea why its now working.

–Back story–

In the beginning I had an old daily driving PC I replaced with some fairly good specs. It has an intel i7-4770 CPU and 32GB of ram. I built it as a gaming PC back in 2013 that I was using up until a few months ago when I replaced it after it had some issues with handling a newer graphics card.

Not wanting to throw it out or have it become a dust collector, I put linux mint on there bought a TV tuner card (Hauppauge 1196 WinTV HVR-1265 PCI Express), attached an antenna to it and hooked it up to my living rooms TV, with Plex and nextPVR. This worked great as a little media server for several months in my living room with no problems.

However, watching level1 and getting inspired to degooglify my life I decided it might be better to turn that machine into something that could handle such things better and I had read and heard about TrueNAS scale. In my research I saw that while it was not quite there to pass hardware through to a docker image, a Virtual Machine should be no problem. So I decided to get some extra HDDs and install TrueNAS scale.

–Discovering the Problem–

I got TrueNAS scale up and running on this machine very quickly and set up my first pool. After tooling around with the provided Plex app I decided a VM is what I really needed. So I set up an ubuntu server vm in TrueNAS. I tried to add hardware but the list was empty. After some research(hxxps://www.truenas.com/community/threads/how-to-pass-through-a-pcie-device-such-as-a-network-card-to-vm.95635/) I discovered there may be some settings not enabled on my motherboard. So into the bios I went, virtualization was enabled but then I dug around and found that vt-d was not enabled so I turned that on and booted back into TrueNAS. I was then able to see all the PCI devices and could add them to the VM.

Using lspci I found the TV tuner card and made sure to add that as a PCI passthrough device in the VM settings then booted up the VM, installed nextPVR and did a channel scan and sure enough I had broadcast TV coming through the VM to my web browser on my other desktop.

From here I went to set up the NFS mounting for this media server VM, but to do that I had to set up a network bridge. As an aside I must say that following these directions from TrueNAS was a complete PITA and required several reboots to get changes to stick (hxxps://www.truenas.com/docs/scale/virtualization/accessingnasfromvm/). Finally the VM could ping the internet and the host machine.

At this point I wanted to add another pair of HDDs to the machine so I shut the machine down and plugged an additional pair of HDDs. Turned it back on and created a second Pool with them.

Happy with my creation, I decided to double check the media servers nextPVR instance to see if the channel guide had updated and when I clicked to watch a channel at the bottom I saw Streaming Failed (transcoder exited). I slowly went through rebooting everything; the VM, the machine TrueNAS was on but kept running into the same issue.

–Diagnosis–

It was previously working. I had seen it with my own eyes, so I wasn’t sure what had changed. I started with the last non hardware thing I did which was the network bridge. So I deleted that and put it back to how it was and that did not solve the problem.

I then suspected it was a power draw problem so I opened the machine back up and unplugged the power from the two additional HDDs I added and the same problem when I booted back up.

Next to make sure it was not an actual hardware problem, I took a USB of Linux Lite and I booted that and installed nextPVR and told it to channel scan and it worked and I was once again able to stream live TV to my web browser. So the hardware was fine.

Next I moved the TV tuner PCI card to a different PCI slot and changed the VM to point to the new PCI address. Same problem.

Next I spun up a new VM and added the TV cards PCI address as a passthrough to that new VM and installed nextPVR there and still the same problem.

Next I started looking into dmesg to see if there were any problems. No errors reported.

[    5.967675] tveeprom: Hauppauge model 161111, rev A1I6, serial# 4036167623
[    5.967676] tveeprom: MAC address is 00:0d:fe:93:07:c7
[    5.967677] tveeprom: tuner model is SiLabs Si2157 (idx 186, type 4)
[    5.967677] tveeprom: TV standards NTSC(M) ATSC/DVB Digital (eeprom 0x88)
[    5.967678] tveeprom: audio processor is CX23888 (idx 40)
[    5.967678] tveeprom: decoder processor is CX23888 (idx 34)
[    5.967679] tveeprom: has no radio, has IR receiver, has no IR transmitter
[    5.967679] cx23885: cx23885[0]: hauppauge eeprom: model=161111
[    5.986754] Console: switching to colour frame buffer device 240x67
[    6.011266] amdgpu 0000:01:00.0: [drm] fb0: amdgpudrmfb frame buffer device
[    6.036847] cx25840 8-0044: cx23888 A/V decoder found @ 0x88 (cx23885[0])
[    6.041001] b43-phy0: Broadcom 4352 WLAN found (core revision 42)
[    6.041634] b43-phy0 ERROR: FOUND UNSUPPORTED PHY (Analog 12, Type 11 (AC), Revision 1)
[    6.041987] b43: probe of bcma0:1 failed with error -95
[    6.042221] Broadcom 43xx driver loaded [ Features: PNLS ]
[    6.049063] [drm] Initialized amdgpu 3.40.0 20150101 for 0000:01:00.0 on minor 0
[    6.668160] cx25840 8-0044: loaded v4l-cx23885-avcore-01.fw firmware (16382 bytes)
[    6.697143] intel_rapl_common: Found RAPL domain package
[    6.697376] intel_rapl_common: Found RAPL domain core
[    6.697590] intel_rapl_common: Found RAPL domain dram
[    6.748200] cx23885: cx23885[0]: registered device video0 [v4l2]
[    6.748484] cx23885: cx23885[0]: registered device vbi0
[    6.748807] cx23885: cx23885[0]: alsa: registered ALSA audio device
[    6.748809] cx23885: cx23885_dvb_register() allocating 1 frontend(s)
[    6.749080] cx23885: cx23885[0]: cx23885 based dvb card
[    6.761028] si2157 7-0060: Silicon Labs Si2147/2148/2157/2158 successfully attached
[    6.761353] dvbdev: DVB: registering new adapter (cx23885[0])
[    6.761595] cx23885 0000:06:00.0: DVB: registering adapter 0 frontend 0 (LG Electronics LGDT3306A VSB/QAM Frontend)...
[    6.762261] cx23885: cx23885_dev_checkrevision() Hardware revision = 0xd0
[    6.762559] cx23885: cx23885[0]/0: found at 0000:06:00.0, rev: 4, irq: 16, latency: 0, mmio: 0xf0800000
d

No errors reported.

The latest thing I’ve done is replace the battery on my machine’s motherboard and after it reset the CMOS, turned virtualization and vt-d back on, and checked it turned on correctly which it looks like it did.


dmesg | grep -i -e DMAR -e IOMMU 
[    0.000000] Command line: BOOT_IMAGE=/ROOT/[email protected]/boot/vmlinuz-5.10.93+truenas root=ZFS=boot-pool/ROOT/22.02.RELEASE ro console=ttyS0,9600 console=tty1 libata.allow_tpm=1 systemd.unified_cgroup_hierarchy=0 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1
[    0.007451] ACPI: DMAR 0x00000000DD955738 000080 (v01 INTEL  HSW      00000001 INTL 00000001)
[    0.007467] ACPI: Reserving DMAR table memory at [mem 0xdd955738-0xdd9557b7]
[    0.018013] Kernel command line: BOOT_IMAGE=/ROOT/[email protected]/boot/vmlinuz-5.10.93+truenas root=ZFS=boot-pool/ROOT/22.02.RELEASE ro console=ttyS0,9600 console=tty1 libata.allow_tpm=1 systemd.unified_cgroup_hierarchy=0 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1
[    0.018097] DMAR: IOMMU enabled
[    0.091269] DMAR: Host address width 39
[    0.091410] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.091604] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.091888] DMAR: RMRR base: 0x000000dd8c8000 end: 0x000000dd8d4fff
[    0.092114] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
[    0.092343] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[    0.092539] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.093041] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.423257] iommu: Default domain type: Passthrough (set via kernel command line)
[    1.176592] DMAR: No ATSR found
[    1.176749] DMAR: dmar0: Using Queued invalidation
[    1.176967] pci 0000:00:00.0: Adding to iommu group 0
[    1.177164] pci 0000:00:01.0: Adding to iommu group 1
[    1.177357] pci 0000:00:14.0: Adding to iommu group 2
[    1.177552] pci 0000:00:16.0: Adding to iommu group 3
[    1.177744] pci 0000:00:19.0: Adding to iommu group 4
[    1.177937] pci 0000:00:1a.0: Adding to iommu group 5
[    1.178129] pci 0000:00:1b.0: Adding to iommu group 6
[    1.178321] pci 0000:00:1c.0: Adding to iommu group 7
[    1.178515] pci 0000:00:1c.1: Adding to iommu group 8
[    1.178707] pci 0000:00:1c.3: Adding to iommu group 9
[    1.178900] pci 0000:00:1d.0: Adding to iommu group 10
[    1.179104] pci 0000:00:1f.0: Adding to iommu group 11
[    1.179299] pci 0000:00:1f.2: Adding to iommu group 11
[    1.179538] pci 0000:00:1f.3: Adding to iommu group 11
[    1.179730] pci 0000:01:00.0: Adding to iommu group 1
[    1.179917] pci 0000:01:00.1: Adding to iommu group 1
[    1.180110] pci 0000:03:00.0: Adding to iommu group 12
[    1.180305] pci 0000:04:00.0: Adding to iommu group 13
[    1.180500] pci 0000:05:01.0: Adding to iommu group 14
[    1.180695] pci 0000:05:04.0: Adding to iommu group 15
[    1.180894] pci 0000:05:05.0: Adding to iommu group 16
[    1.181089] pci 0000:05:06.0: Adding to iommu group 17
[    1.181284] pci 0000:05:07.0: Adding to iommu group 18
[    1.181479] pci 0000:05:08.0: Adding to iommu group 19
[    1.181674] pci 0000:05:09.0: Adding to iommu group 20
[    1.181872] pci 0000:06:00.0: Adding to iommu group 21
[    1.182069] pci 0000:08:00.0: Adding to iommu group 22
[    1.182266] pci 0000:0a:00.0: Adding to iommu group 23
[    1.182463] pci 0000:0c:00.0: Adding to iommu group 24
[    1.182701] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    1.197929] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.
[    1.340976]     intel_iommu=on

And still nothing. Finally I thought I’d pull out a big hammer and reinstall the TrueNAS scale. Which I did but I didn’t nuke my old settings (it did not seem to give me the option) this did not work but I do wonder if I should escalate to that.

–conclusion–

So I am stumped both on what could be wrong and how to diagnose this.

Please let me know your thoughts or any additional information I might be able to provide

Thanks