Help choosing new virtualization workstation platform

I am in dire need of a platform upgrade but am on a tight budget which is preventing me from pulling the trigger on something that may not work out in the end. My needs are so specific that not many platforms can not do what I need them to. My motherboard’s clock-gen is becoming unstable and is randomly causing the CPU to jump from 5 GHz to 5.7 GHz (usually upon boot or rapid CPU load fluctuation). While it is stable idle at 5.7 GHz the CPU is at 75-100% load most of the time and usually crashes. I have resorted to running stock CPU clocks with very high voltage to maintain stability when the baseclock fluctuates and then just reboot with the CPU at 5GHz when I want high IPC to play games. I have replaced this board 2 times under RMA already, the first one lost a RAM slot and the second lost all PCIe except the last slot. While it is great virtualization wise, I am done dealing with this board, I want it gone forever, it is time to upgrade. I don’t want to spend more than $500 USD on CPU/Motherboard/RAM upgrade. I have been thinking of going x79 with a 10 core XEON E5-2690V2 and re-using my existing RAM. The only problem with this is single core performance will be about equal or worse. Another option is Threadripper but I would have to save for awhile, by the time I have enough extra to spend on a Threadripper system 1-2 CPU generations will have passed. A dual CPU system with 2 higher clock speed, low core count XEON CPUs might be an option to get high core count and high IPC at the same time on a low budget. Does anyone have experience with a platform that could suit my needs?

What I do with my workstation:
I am a developer of many things and I use a wide range of software and operating systems. I primarily use Debian as a hypervisor and often run Linux, Windows, MacOS, and occasionally more obscure operating systems as guests. I do everything from graphic design, PCB design, 3D modeling, game/operating system/software development, I compile a lot of software that takes upwards of 30 minutes on my current configuration. I also occasionally play and test games in a Windows 7 or 10 VM.

Current system:

  • CPU = 5.057 GHz AMD FX 8350. Actually 100% stable, like uptime of 3+ months and only reboot for kernel patches stable.
  • Motherboard = Gigabyte 990FXA-UD3 (crying for mercy)
  • GPU1 = Nvidia GTX 1070
  • GPU2 = Nvidia Quadro something-or-other
  • GPU3 = AMD RX 560
  • NIC1 = Intel 82576 (Ethernet card 1)
  • NIC2 = Intel 82576 (Ethernet card 1)
  • NIC3 = Intel 82576 (Ethernet card 2)
  • NIC4 = Intel 82576 (Ethernet card 2)
  • NIC5 = Realtek 8169 (onboard)
  • Audio = Realtek ALC1150 (onboard)
  • 1x Marvell 88SE9172 SATA controller (eSATA) 2x ESATA drive docks for formatting drives / installing OSes (onboard)
  • 1x AMD SATA controller 2x Samsumg 840 EVO SSDs (onboard)
  • 6x AMD USB 2.0 controllers (2x ports each) (onboard)
  • 1x VIA USB3.0 controller (4x ports) (onboard)
  • PSU = Seasonic 1000w 80+ platinum

What I require in a platform:

  • Absolute stability, random freezes or reboots due to random hardware / BIOS bugs are unacceptable.

  • Lots of PCIe connectivity, in my current machine I have 2 add in Ethernet cards and 3 graphics cards which I use vfio and a modified version of QEMU to hot plug to/from my various virtual machines.

  • A platform that can handle hot plugging PCIe cards with out power cycling them with out stability issues.

  • 8 minimum physical cores. (preferably 10+) having only 8 cores is a bit of a struggle for me, right now I have to use scripts to allocate CPU cores between the VMs based on their load and activity. I would like to be able to have more cores so I can just dedicate cores to some of my VMs and not worry about having enough left over for basic tasks.

  • Minimum of 32 GB RAM, I currently have 32 GB of 1866MHz DDR3 and its just enough, barely. This is one of the major factors for me not upgrading my platform, 32 or 64 GB or DDR4 is way to expensive.

  • Perfect or near-perfect IOMMU support /grouping, every device must have its own IOMMU group. my current motherboard is amazing for this, even devices like an Ethernet card with multiple NICs on one card have each NIC in their own IOMMU group. My current board also has each of the 7 USB controllers in separate IOMMU groups, the SATA controllers are also isolated.

  • Half decent onboard audio as I will not have any available expansion slots for a sound card and would rather not use a USB DAC.

  • Low latency virtualization, my current setup can manage 250 microseconds (0.25ms) total latency in ultra high FPS games like osu. I am using no virtualized devices, vfio passthrough for everything (NIC, GPU, USB, disk, audio, etc.) to achieve this.

  • Decent IPC, does not have to be the best but I do like to enjoy high FPS gaming from time to time and the FX 8350 is struggling a bit in new titles, it can be hard to stay above my monitor’s 90hz refresh rate. Funnily enough my Windows VM has better IPC than Windows installed on a native FX 8350 due to the staggered coremask I am using. The VM has 4 cores but each core has access to a different cache module instead of 2 cores sharing 1 module. My Windows 10 VM Cinebench r15 single core score is 130cb with this setup.

  • OS agnostic hardware control. Overclocking, fan control, etc. must be preformed by onboard hardware and configured in BIOS utility so it works not matter what OS is running. I don’t want to have to write a Linux driver to turn off the default-on RGB LEDs on a motherboard like I had to do in the past. Or the most egregious one, having to write a fan control driver because my crappy HP laptop requires a Windows only driver to control the CPU fan or it will be stuck at 10% and let the CPU overheat (facepalm)

  • Onboard internal serial header.

  • Dual onboard Intel NICs would be nice, I may be able to eliminate an add-in card then.

1 Like

Haven’t really got a suggestion but a question, Would hotplug of PCI-e work for you over thunderbolt? Like external GPU’s, that’s the best option I can think of since PCI-e isn’t designed for hot plugging generally.

In all honesty though, Coming from my experience, ASUS or ASROCK’s WS lines are probably your best bet. My old Z87-ws has serial, dual gigabit nic. There is probably another board similar though going for X99 might see some stability issues. Would start here
https://www.asus.com/nz/Motherboards/Workstation-Products/
Just a small note, serial isn’t that common anymore so you might end up going to go through serial over USB

@KemoKa73 is selling an ASUS Z9PE-D8 WS with dual E5-2670 processors. I’m no expert in the field, but it sounds at least close to what you’re looking for.

  • Bunch of cores
  • PCIe galore
  • Can use ECC memory (for stability)
  • Other built in doodads and whizbangs
4 Likes

@Yuki

I don’t know what sort of clockspeeds you’re looking for, but there’s a setting on this WS board that allows you to run the CPUs in turbo boost all the time (For the 2670s on all cores I think that’s about 3GHz), so long as you keep the CPUs within thermal limits. A couple of Hyper 212s with good fans on them will do this easily. I also have a bucketload of ECC DDR3-1066 laying around that I could probably just give you if you’re strapped for cash. They’re only 4GB DIMMs so I can only put in 32GB, and it’s slower memory than most, but it should (?) get you started.

2 Likes

Wow dude! I unfortunately cannot contribute because I’m not familiar with tech in the price range of your budget, but I am putting in a request for you to do a build log and post your battlestation contribution.

When you say workstation you aren’t fraking kidding!

Good luck, I hope to see you around the forum more!

I will DM you with some questions if that’s cool.

1 Like

I am actually very interested in this combo! Have you had any issues with IOMMU / VT-D? I was hoping for 4 GHz like a pair of E5 2667 v2 but I would not mind having 2x 2670 and replacing them with 2667 v2 if they don’t quite cut it. After doing a bit more research I will probably be able to make a decision tomorrow if I want to give your combo a shot.

I hear the integrated marvel controller can cause issues unless you disable it in the BIOS utility which is fine as I mostly work and boot from a network server and only need about 4 local SATA drives.

Have you considered shifting your VM workload to a proper server platform with ECC/other fault tolerant hardware, and doing your work via remote session?

Just putting it out there, because your requirement for “absolute stability” and lots of cores and memory tend to lead to that sort of platform.

i.e., offload a lot of your heavy lifting to a headless machine running ESXi on a server platform, and have a less massively specced workstation (maybe your current one in stable clocked form or whatever) for the graphical stuff like gaming and application testing.

If Ryzen 1st gen is stable enough by now you might consider a B350 board with something like a 1700, but you’ll need more memory.

But in any case i think you’ll struggle to get what you want inside of your budget with new hardware. Primarily due to RAM pricing.

as i understand it, PCIe hotplug isn’t actually common (Depends on hardware support AND driver support - good luck with consumer grade) but i’d expect it to be best found on server grade hardware also.

I never really used it for virtualization, but I made sure the 2670s were C2 stepping CPUs so the VT-d extensions should be there. And as you said, if they don’t quite cut it you can replace them down the road. The platform meets all your other requirements as far as I can tell.

Keep me posted! :smiley:

1 Like

Thunderbolt would probably work as well, I thought about going eGPU for my laptop which supports VT-D but have not put in the time yet as that is more of a for-fun project. A a decent thunderbolt dock would push a bit above my budget though and prefer to have everything consolidated in one box, USB serial would be fine and a great suggestion.

I am actually already running ECC memory in my current machine and yes I am looking for more workstation / server grade platforms. I actually am now planning on buying @KemoKa73 's “Asus Z9PE-D8 WS with 2x E5-2670 CPUs” combo and either upgrading to 2x E5 2667 v2 or re-selling it if that does not work out for me.

I already have a dual 6 core XEON headless server with 18 GB of ECC DDR3 which also runs Debian, This does help offload some of the load from my main system but having direct access to hardware is necessary for a few of my VMs. It does support VT-D but only with PCIe cards that have a proprietary HP firmware on them =_=. The “Asus Z9PE-D8 WS with 2x E5-2670 CPUs” combo might just actually end up replacing that machine in the end and use another system like my current FX 8350 one for gaming like you said. Not sure how that will work out yet as the FX 8350 system can not actually boot any version of Windows on bare-metal without pulling PCIe cards out due to various hardware incompatibilities. The Quadro + GTX 1070 breaks the Nvidia Windows driver and my Intel NICs cause Windows not to boot. I would probably end up using just the 1070, buy another GPU for the workstation, and get NICs that plays nicely with Windows.

1 Like

I am curious if you can replace your Linux vms with container. Like using Lxd/lxc or docker. Lxd is more similar to a Solaris zone and a new feature in Lxd you can now cluster them I have not tried it yet. Or you can use a kubernetes cluster. Lxd via Ubuntu juju you can setup a kubernetes cluster I tried it and it is really easy but do not setup ip6 for Lxd I’d you are going to use juju kubernetes that won’t play nice.

Juju and Lxd can you install via snapd and Lxd support other linux distro containers.