Moving from FreeNAS to Proxmox, virtualizing FreeNAS and SteamOS/Debian (Now running PopOS!)

[Trooper_ish]

Glad someone likes Vim

[TheCakeIsNaOH]

ProxMox seem’s to be quite good at PCIe passthrough, so that should mean passing a gpu into that VM is drag and drop (knock on wood)

AMD gpu’s tend to be very easy.

Not always, newer AMD cards have the reset bug.

[Camofelix]

Wow, what a strange bug. 100 GB was pulled out of my ass, I have 144GB on the server, so was thinking 32GB for the Steam Box, 100GB for FreeNAS and the balance to Proxmox and maybe PFS sense down the line. Between my 2 pools, one dedup and 1 not deduped I should be around 60 gigs, but don’t see a reason to not give more to the ARC. looks like I’ll just have to “settle” for 96 GiB.

Yeah, as silly as it sounds, I really like the GUI, especially since the 11.1 update where they moved to AngularJS for the platform.

[Camofelix]

Probably on bare metal, but since it’s in a VM, it seems to not like it.

[Camofelix]

Yeah, seem’s to be the place to go to achieve everything I want without completely kiboshing my existing workflows.

[Trooper_ish]

Even odder, as the vm will have “spice” drivers, so technically has a GPU, more than bare metal might as standard.
Oh well Good luck tomorrow night with your tests

[Camofelix]

That’s the one where the VM does’t properly let go of PCIe devices when they reboot correct?

[Camofelix]

Sounds good! will report back when I can, hopefully this thread can eventually be useful to others in a similar situation.

[TheCakeIsNaOH]

I think it is that the PCIe device does not properly reset itself when told to.

There are patches being worked on-

Vega 10 and 12 reset application

Hi All, As some of you may be aware I have been working to find either a workaround or fix to the AMD Vega reset bug. Last week I posted to AMD’s reddit a cry for help to fix this issue in an attempt to show AMD how much demand there is for this. As a result, an AMD Engineer got in touch and has guided me to a possible solution to the problem. Over the weekend I have spent considerable time implementing what seems to be a working reset for Vega 10 and 12, initial testing by a few people confirm that it is working on Vega 10, however it needs further testing. You must apply this patch to your kernel to prevent vfio-pci from attempting to reset the GPU incorrectly. Please note that this application is intended as a interim workaround while I work on implementing this into the kernel for vfio. Download reset-test.tar.gz Usage is simple, obviously you must not be using the GPU at the time and it should be bound to vfio-pci. ./reset-test 0000:24:00.0 The expected output is: ============================================================================ AMD Vega 10/12 Reset Application (Version: 1.0) Copyright (c) 2019 Geoffrey McRae <[email protected]> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. This tool is intended as an interim workaround while I port this into the kernel driver. If you like my work and want to support it you can contribute using the following methods: * Ko-Fi - https://ko-fi.com/lookingglass * Patreon - https://www.patreon.com/gnif * BTC - 14ZFcYjsKPiVreHqcaekvHGL846u3ZuT13 ============================================================================ Attempting Vega 10 reset CMD_READMODIFYWRITE 0x00000e1c CMD_WRITE 0x00000e1f CMD_READMODIFYWRITE 0x00000e2b CMD_READMODIFYWRITE 0x00000e2b CMD_WAITFOR 0x0001667c CMD_READMODIFYWRITE 0x00000e2b CMD_READMODI…

Navi Reset Kernel Patch

Navi Reset What this does is use powerplay tables to turn the card off and back on again. (Insert IT crowd meme here, haha). @gnif has done it again. Good work!! Ok, here is the navi reset. Navi is easier to implement as the SMU has most of the logic in it. Update 27-11-2019: Updated patch down below: Navi Reset Kernel Patch From 69ea42207b544b6e3fa9755022bff09d2ce953d9 Mon Sep 17 00:00:00 2001 From: Geoffrey McRae <[email protected]> Date: Thu, 12 Sep 2019 03:19:28 +1000 Subject: [PATCH] pci quirk: AMD Navi 10 series vendor specific reset Signed-off-by: Geoffrey McRae <[email protected]> --- drivers/pci/quirks.c | 98 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 44c4ae1abd00..d94ddb1c6832 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3825,6 +3825,97 @@ static int delay_250ms_after_flr(struct pci_dev *dev, int probe) return 0; } +/* + * AMD Navi 10 series GPUs require a vendor specific reset procedure. + * According to AMD a PSP mode 2 reset should be enough however at this + * time the details of how to perform this are not available to us. + * Instead we can signal the SMU to enter and exit BACO which has the same + * desired effect. + */ +static int reset_amd_navi10(struct pci_dev *dev, int probe) +{ + const int mmMP0_SMN_C2PMSG_81 = 0x16091; + const int mmMP1_SMN_C2PMSG_66 = 0x16282; + const int mmMP1_SMN_C2PMSG_82 = 0x16292; + const int mmMP1_SMN_C2PMSG_90 = 0x1629a; + + u16 cfg; + resource_size_t mmio_base, mmio_size; + uint32_t __iomem * mmio; + unsigned int sol; + unsigned int timeout; + + /* bus resets still cause navi to flake out */ + dev->dev_flags |= PCI_DEV_FLAGS_NO_BUS_RESET; + + if (probe) + return 0; + + /* save the PCI state and enable memory access */ + pci_save_state(dev); + pci_read_config_word(dev, PCI_COMMAND, &cfg); + pci_write_config_word(dev, PCI_COMMAND, cfg | PCI_COMMAND_MEMORY); + + /* map BAR5 …

[oO.o]

Hell you could even run a FreeNAS VM and leave your pools on FreeNAS. What I don’t know is if Proxmox will mess with the disks, but in theory you can pass through the controller to the VM and it will run as normal.

FWIW, I do this in Proxmox by passing a HBA to the FreeNAS vm it’s been working for many months without issue.

Ubuntu server

I personally recommend this (or Proxmox). You can rest easier knowing the distro supports ZFS natively.

[Airstripone]

Just picking up the thread after a failed attempt at sleep.

@Camofelix I think you have a workable solution but my guidance would be to test build this first if you have the hardware spare.

I’ve never taken the plunge with ‘hyperconvergence’ because I don’t have 100% trust in my geek-fu not to break something and trash all my pools. Restoring from backup is not my idea of a good time.

I’d suggest getting your converged system up and running with hypervisor (thin or thicc) and gaming hosts (you can have windows and Linux on there at the same time!) and test your gaming performance, responsiveness and most importantly, case temps.

If it is all fine, then clone your freenas config, create a new freenas vm, pass through the HBA, load your config and move your disks. Test it all running and see how you feel.

Things that would worry me:

• single point of failure.
• power bills from powerful system running all the time
• temps impacting hardware longevity, especially hard disks
• noise at my desk
• loss of ability to upgrade desktop parts in situ without risking data loss on the Nas
• something I haven’t considered going wrong

But if it works, it will be sweet and we can all enjoy your journey to get there.

Good luck!

[HRSCR]

Linus Torvalds Doesn't Recommend Using ZFS on Linux

[Trooper_ish]

Please check this

Linus Torvalds says “Don’t use ZFS”—but doesn’t seem to understand it

[kdb424]

Several responses that I’m too lazy to quote. If an RX 590 is enough to game on for you, they work amazingly. If you need more power than that in a VM, Code 43 and reset bugs wull be something you have to deal with. I’ve had code43 show up on nvidia cards, and start appearing again even after I fixed it. The only way that I found that truly fixes it was to basically disable all acceleration on the VM basically, as all of the spoofs that were around at the time I ran nvidia cards all prevented me from using any virtio on the VM, and really hurt performance. That may have changed.

As far as VMing FreeNAS, I really recommend trying to convert away from that platform. Proxmox handles ZFS really well and covers all VM needs. You’ll get much better perf on the array natively, and ARC is much easier to manage on the host OS as it will get out of the way if you overprovision and the system needs ram.

[Razor_Blade]

dont_use_zfs_torvalds.jpg800×450 42.4 KB

Don’t tell me what to do.

[sanfordvdev]

Yea he kept referencing Oracle ZFS and never openZFS.

[oO.o]

He’s just worried about licensing and being sued, which are legitimate concerns for him but we don’t need to care about it. Lucky for us, Canonical dgaf.

[Log]

One foot in the mouth wasn’t enough, so Linus stuck the other one in with that goofy follow up. Oracle’s ZFS flat out is not compatible with Linux. It runs on their proprietary version of Solaris for that sweet vendor lock-in.

The only ZFS that matters when it comes to Linux, has been and always will be the openzfs project.

[Camofelix]

I’m thinking this will be my implementation. Proxmox is designed as a hypervisor and should be used as one. FreeNAS is built as a storage appliance and should be used as one.

Also worth noting that, as much as I have backups and so on, it’s preferable for me to not have to mess with any of my data.

[Camofelix]

So I did some testing tonight (oops, just noticed the sun is coming up in my timezone…) and it seems ProxMox on bare metal is the solution for my needs. The outline of the plan would be as follows:

1. Get a SATA SSD to serve as my ProxMox drive.

Install ProxMox onto this SSD. Run Memory test as Sanity check. Create FreeNAS VM

2. FreeNAS would be in a VM. I would Passthrough: my HBA, NVMe and a config backup disk (my Mobo has an SD card slot that I backup my FreeNAS config to using a CronJob.) Give it 4 threads and 96GB of ram as per RazorBlades Comment and experience.
   Upload config from backups to recreate my existing freeNAS install “as is”
3. I Would Create a new Debian VM for Steam. Passthrough a Gpu, 8 threads, 32gb of Ram. Installed on a New/used/whatever I get my hands on SSD. joys of student budgets might delay this VM for a while.

I’ll upload a photo video at some point (away from home for my Provincial Engineering Competition!), but I have access to a sort of attic area attached to a cube door in dorm room that’s partially insulated where I ran Ethernet and Power. Temps in my city where in the minus 25-35 Celsius range, system averaged temps of 10-20 on cpu, 15-25 on HDD’s.

I have a cron job that checks for current temperature via the HP ilo2 shell and has a Plex VM start auto transcoding if temps drop below 5 Celsius, that way the machine is never at risk of complete “Meltdown” (oh my good god that pun was awful, I’m bad and should feel bad…)

The FreeNAS VM seemed to work fine with 2 flash drives set as a mirror for testing purposes, so when I’m back in the city and can get my hands on an SSD for proxmox to run off of to do this project on, I can begin.

luckily the most I’ve ever pulled from the wall at max load is around 240W with everything at 100%. so add in a gpu (depending on what I can get my hands on- Looks like my little brother killed the GPU back home I was going to be using for this project )

I’ve yet to be able to try the steam OS VM for real. What I’m thinking could work temporarily for testing is having steam run the game off of my laptop, and in virtual box on the same laptop run proxmox, and in the proxmox VM run SteamOS. Then from the Steam OS VM stream the game from my laptop bare metal as a proof of concept.