Excelsior! Threadripper KVM Build with GPU Passthrough and ZFS

Greetings True Believers!

Lets go on a journey together. Join me as I plan and execute an amazing computer for myself and my family that will go by the name of “Excelsior!” in honor of the great Stan Lee.

To live up to its name, it must be something truly exceptional. My vision for Excelsior! is that it will have a host such as Proxmox that supports GPU Passthrough as well as ZFS and it will serve as 2 Windows 10 Gaming Workstations, a NAS, and whatever else we can dream and implement!

The Hardware:

  • Processor: AMD Threadripper 2950X
  • CPU Cooler: Cooler Master Wraith Ripper
  • Motherboard: ASRock X399 Taichi
  • Memory: 128GB DDR4 ECC
    • (8x) Crucial 16GB DDR4-2666 ECC UDIMM
      • Part #: CT16G4WFD8266
      • Crucial Part #: CT12146164
  • Storage:
    • (3x) Samsung 970 Evo 500GB M.2 NVME
    • (8x) Seagate 12TB IronWolf ST12000VN0007
  • Video Cards:
    • GPU (Host)
    • GPU (Windows 10 Guest)
    • GPU (Windows 10 Guest)
  • Case: Fractal Design Define R6 Blackout FD-CA-DEF-R6C-BKO
  • Power Supply: Seasonic PRIME 1300 Platimun SSR-1300PD

Excelsior! PC Part Picker

The Software:

  • Host:
    • Proxmox
    • ?Other Linux KVM?
  • Guest:
    • Windows 10 with GPU passthrough
      • Main Workstation
    • Windows 10 with GPU passthrough
      • NVIDIA GameStream VM to NVIDIA Shield TV in living room
      • Parsec Server

Research, Design, Planning:

  • Phase 1: Research and Design
  • Phase 2: Acquire Hardware
  • Phase 3: Windows 10 Bare Metal
    • Hardware Stage 1:
      • Windows 10 Bare Metal Install
        • OEM Cooler Master Wraith Ripper Thermal Interface Material
        • Single GPU
        • Single NVME
        • No 3.5" HDDs
      • Overclocking and Performance Benchmarking
    • Hardware Stage 2:
      • Windows 10 Bare Metal Install
        • Replace OEM Cooler Master Wraith Ripper Thermal Interface Material with Prolimatech Pk-3 Using [H] Method
      • Overclocking and Performance Benchmarking
  • Phase 4: Linux
    • Hardware Stage 3:
      • Dual Boot Windows 10 Bare Metal and Linux
        • Single GPU
        • (2x) Samsung 970 Evo 500GB M.2 NVME
        • (8x) Seagate 12TB IronWolf SATA HDD
      • Learn Linux, Debian, ProxMox, Arch Linux, ZFS
      • Decide on hypervisor and final hardware configuration
  • Phase 5: Virtualized
    • Hardware Stage 4:
      • Single Boot Linux Hypervisor
        • Full Hardware Build
          • Boot from ZFS Mirrorred USB 3.1 Drives
          • Multi-GPU
          • (3x) Samsung 970 Evo 500GB M.2 NVME
        • Windows 10 Guest 1
          • 8 cores, 16 threads
          • 64GB RAM
          • (1x) Samsung 970 Evo 500GB M.2 NVME
          • Single GPU Passthrough
        • Windows 10 Guest 2
          • 4 cores, 8 threads
          • 8GB RAM
          • (1x) Samsung 970 Evo 500GB M.2 NVME
          • Single GPU Passthrough
      • Performance Benchmarking and Comparison vs Bare Metal
  • Phase 6: …

The Build:

It all starts with a blank canvas.

Hardware acquired!

Don’t forget the case that comes in an absolutely massive (but protective!) box.

Lets invite some old friends to the party.

2 SATA cables. Hmm… that’s not gonna cut it!

A beautiful motherboard, but it looks lonely. I think a team up is in order!

Tai Chi, meet Threadripper. This unboxing was very satisfying, not gonna lie. An included torque wrench? Gigity!

Alright, now they need to figure out how to work together.

This is my first go at a Threadripper build, lets see if I can NOT BREAK IT.

Oh lawd… so many fragile pins exposed…

It’s a fascinating install process. It feels very substantial and well engineered.

Threadripper in place. Almost there!

Phew. Threadripper safe and sound in its socket.

Watch out Tai Chi! Ninjas! Oh, wait, you guys wanna team up too?

Lets see what you’ve got Ninjas.

Triple flying ninja kick!

Man, I can’t believe you guys landed that.

Our team is growing in infamy. Mighty champions flock to us to join our righteous cause.

These emerald warriors fit right in. The time before they were part of the team feels like a distant memory.

Watch out! A Wraith Ripper! Be you friend or foe?

Wraith Ripper, with the help of your friend T.I.M., I think you and Threadripper will work well together.

What an overwhelming presence!

The protective films are off. This is getting serious! Wraith Ripper, such a towering behemoth!

I will return soon to further flesh out this first post, but for now I eagerly await your feedback, whether it be on hardware choices, system design, software advice, or otherwise!

Excelsior!

4 Likes

Good luck! Hope to see someone with more knowledge of Proxmox comment, as I’m not sure on any peculiarities with Proxmox and it would be an interesting read :smiley:

1 Like

Things are coming together for the build and I’ve got some time to work on it. I do have a few questions that I’m hoping the community can clear up for me:

  • Which host with GPU passthrough and ZFS support do you recommend?
  • Do I need a GPU for the Host?
  • Where should I install the Host?
  • How much resources does the Host need?

Having only poked proxmox a few times, it seems OK, and is something I would consider if I was building a new KVM/ZFS box.

proxmox with a root filesystem, maybe 64G on a SSD, and then mirror it after build

i would reserve some of the SSDs used for proxmox for ZFS ZIL/L2ARC potential, but test afterwards to see if it was actually useful, depends on your usage cases.

ZFS raidz2 for the HDDs for use as general and VM storage, with ZIL/L2ARC if useful.

I’m currently running some of my KVM windows VMs on a single disk ZFS, and snapshotting it hourly, and syncing the snapshots to the HDD ZFS. This way I get the speed of a SSD, with the security of raidz2, as long as I’m willing to lose some snapshots modulo my snap sync interval. For me, nothing on the windows boxes is important and anytime yesterday is fine, but YMMV.

If you are using KVM & ZFS, you might want consider limiting ZFS’s use of ARC, as I find linux kernel low memory pressure doesn’t get ZFS to release memory from ARC quickly enough, so I have my ARC limited to an acceptable trade off between ARC and system/VM memory.

1 Like

I assume the ! in the name Excelsior! is important :wink:

Seriously though that is some serious hardware, you seem to be writing my dream PC. What GPU’s where you planning on using?

1 Like

I had some time to work on the build. I added some content to the OP.

I’m still looking for feedback, don’t be shy!

After discussing a bit in the L1 Discord, I will be trying ProxMox 5.3 on zfs mirrored USB 3.1 sticks. I do plan on try a lot of different things to learn and see what works best though.

Cheers!

Hahaha! =) I’ve got a 1080ti and a 670 that lost its bracket in a car accident, lol! We’ll see how things shake out as things move forward.

Thank you so much for your valuable input! I composed and submitted a well thought out response, but it must have been lost in the ether. I’m pretty much spent at the moment but I will give a reply a second go when my brain is working again!

Cheers!

Please make this an album!

holy shit that is a big heat-sink.

i knew the TR heatsink was big, but shit…

I’m trying to decide on a HOST os. Originally, it seemed like ProxMox VE would be a good choice as it has tons of the feature I need. However, ProxMox VE 5.3 is based on Debian Stretch 9.61, kernel 4.15.18, QEMU 2.12.1, and ZFS 0.7.12 . I have read that for threadripper, I really want a kernel that is 4.18 or later and a recent version of QEMU (3.0 or later, preferably with the PCI-E link speed negotiation patch). From what I have read, I will not be able to easily update the Kernel in ProxMox VE 5.3 (at least with my skill level) but I’m not sure if I could update QEMU. Going down this rabbit hole, it seems like ArchLinux would be my best bet for a Host OS that is properly updated. What do you think?

I’m considering this exact motherboard cpu combination, can you post which BIOS version you are using and the output from the iommu group script?

I cannot find this information anywhere.

Thanks,
Luke.

If you want really up to date, gentoo testing.

I know this is an old thread, but how has it been running the last year? Any changes? What GPU did you go with? I’m looking to build a combo server to replace my current nas, vm machine, and desktop. Probably a somewhat similar build with just some minor tweaks.