[Build Help] SFF Proxmox Workstation w/ VFIO

Hardware Build a PC
, ,
1

TL;DR At the bottom

The Backstory

I sold my last desktop more than 10 years ago because I needed an upgrade and was looking forward to building myself a new PC soon.

But then life happened… The money from the sale slowly dissapeared and for the last 10 years I’ve been mostly using older hand-me-down laptops. Currently rocking a second-hand Thinkpad X270 (i5-7200U w/ 8 GB) running Pop!_OS, beautiful machine.

I moved around a lot and right now I’m at a more stable place. Finally finishing up my studies, and have two decent part time software development jobs. Managing to cover all my expenses and even started some savings. The dream of an building myself an upgraded PC never dissapeared, but recently it has been resurrected by one of my bosses who said he would be willing to co-finance it. I guess he felt sorry after seeing my laptop struggling to run both run Zoom & our demos.

The Idea

I’d like to build a SFF Proxmox “workstation”.

Something where I can run my main distro (currently Pop!_OS, while also running self-hosted stuff in the background.

Self-hosted things that come to mind are a NAS, Gitea, seedbox, media server, nextcloud, some scrapers or whatever else I think off. I’d like to do some occasional gaming, Windows VMs, and virtual machines and experiments. 90% of the time the machine & services would only be used by just me.

Passing through a GPU to the main distro seems like a must. Bonus points if it can be passed to a different machine without having to reboot everything. Extra bonus points if the GPU can be split into virtual GPUs (maybe for some background transcoding, ML, or rare two~three-gamers-one-pc LAN parties).

The smaller form factor is preferred to make moving with the PC easier. Should easily fit in a carry-on with some room to spare. The machine should be relatively quiet, and have a smaller power draw when idle and be quite future-proof.

My Research

My budget is 2750 USD (~18000 DKK) and I’m currently based in Denmark. I’m willing to increase my budget it if that means I’d have a much better machine that is gonna last me longer, but spending less money is always nice.

The foundation of my research is Logical Increments, where according to my budget I can build an “Outstanding” PC. I also did a fair share of browsing around on reddit, other forums and watching reviews. Level1Tech Seemed like the perfect place to ask for help for this kind of build.

CPU

Logical Increments Suggestion: AMD Ryzen 7 5800X

My thoughts: Since I’m gonna be running some virtualized & containerized stuff it might be worthwhile to invest in the R9 5900X due to the additional cores. Otherwise if the cores aren’t that necessary 5700G might be worthwhile to consider due to the onboard graphics.

Motherboard

My Thoughts: Probably a Mini-ITX or Micro-ATX motherboard due to the smaller form factor size wish.

RAM

Logical Increments Suggestion: 16GB

My thoughts: I’d probably prefer 32GB. Now the only thing I’m not sure about is if it should be ECC. For the file system I was thinking of running ZFS (seems cool), and it seems to be always recommended, but not sure how strict of a recommendation that is.

Storage

My Thoughts: I’d probably put a fast 1TB M.2 directly on the motherboard. This would be where Proxmox would be installed and my main distro.

For additional storage I was thinking of an SSD array, starting in RAID-0 upgrading to RAID-5 as I add more drive. Any cheap, low power SSDs would be good. Starting target usable capacity of at least 4 TB, but the hoping to expand two RAID-5 and more drive later.

For backups a large HDD that I’d spin up overnight and perform snapshots of the M.2 and the array.

GPU

Logical Increments Suggestion: RTX 3060

My Thoughts: The AMD alternative would be the 6700XT. On paper I’d like to support AMD with my GPU choice, they have decent open source drivers and they actually fixed the reset bug which is super useful for passthrough. I only found people talking about how it is fixed on the 6800(XT), can somebody confirm about the 6700XT?

On the other hand, both of these cards can’t do virtual GPUs… For Turing cards (RTX 2xxx) we have the vgpu_unlock script. There is an open issue in the repo about getting it to work Ampere cards (RTX 3xxx) but I don’t know how uncertain resolving that issue is. Is it only a matter of time, or is there a chance it will never work? I cant find a similar script being developed for AMD card?

PSU

My Thoughts: I guess to choose a PSU I need to finalize my CPU & GPU choices. I’d just like something that fits my case, is relatively quiet and is efficient.

Case

My Thoughts: The NR200 seems like a budget & beginner friendly SFF case. It might not meet my storage requirements (1 HDD, ~5 SSD) but I’ve seen mods where they fit two so I assume I could force a few more SSDs in there. I anyway plan on starting with only two SSDs for budget reasons.

But as I said previously. I just want a minimal, quiet case that can easily fit in carry-on luggage.

dk.pcpartpicker.com/list/LFcP2V

Final Remarks

Sorry for the long post. Please feel free to share any thoughts, questions, things I should research/read/watch. Correct me if I misunderstood something or if such an build isn’t feasible.

Note that currently I don’t self-host anything (besides a personal site on a tiny VPS), so all of this is gonna be learning/experimenting experience.

Thanks!

TL;DR

Looking to build a (smaller-form-factor) Proxmox workstation. I use Linux on the desktop (Pop!_OS), and would like to run some self-hosted stuff, do some occasional gaming and therefore wish to pass through my GPU.

Budget: 2750 USD (~18000 DKK) can be higher to make more future-proof
Country: Denmark
Preferred retailer: Any in Denmark
Peripherals: Already have everything
Use: Work, fun & self-hosting
Overclock: Not interested right now, but can be convinced otherwise
Custom water-cooling: No
OS: Proxmox + Pop!_OS

Gaming questions: Haven’t played games in a long time but would like to do some occasional gaming again. I’d like to try a wide variety of games at 1080p/1440p decent settings. Nothing crazy.

Render or Editing: No rendering, and rarely some light video/photo editing.

dk.pcpartpicker.com/list/LFcP2V

2

Just a quick update on how my thinking has developed. I’ve decided to start with a partial build with an APU. Later I’ll upgrade with a GPU and all the other things I skip out on now. I also did some research regarding RAM and Ryzen CPUs, and I changed my mind about the case.

CPU: 5700G
CPU Cooler: NH-L12S
Motherboard: MiniITX (VFIO Friendly, WIFI + min 4 SATA ports)
RAM: 2x16GB DDR4 (cheapest 3200MHz dual-rank that fits)
Storage: 1TB M.2 (I’ll add the SSD array later)
PSU: Efficient Modular SFX (according to my math ~600 Watts)
Case: SG13

Can you please confirm my thoughts and help me make a final choice for the Motherboard, RAM & PSU?

PC Part picker throws me an warning about the CPU cooler not fitting in the case, but I’ve seen it work in other people’s builds, especially with SFX PSUs. I’ve also created a small rough sketch of how I assume everything is gonna fit into the build:

sg13-will-it-fit
sg13-will-it-fit644×552 84.3 KB

3

@SgtAwesomesauce has a similar build. Build Log | Monolith

Also going to bump your Trust level to post urls (I just edited your current urls to link now as well)

2 Likes
4

Something to note, big navi does not like bifurcation on my motherboard and will not pass through if you bifurcate the pcie slot. Never did figure out why, but it is what it is.

1 Like
5

So uh, I am doing exactly this. Maybe I should do a write-up about my experience?

You can check out my build here: Ryzen Workstation, Carry-on Edition v2 - Ryzen 9 5950X 3.4 GHz 16-Core, Radeon RX 480 8 GB NITRO+, IM01 MicroATX Mini Tower - PCPartPicker

It’s a little more intense than your current considerations, but I managed to keep my total cost to around 3k USD. Of course, some hardware is old (180 dollar GPU and the 200 dollar enterprise HDDs for example are from several years ago). I think your budget is pretty reasonable for a home workstation.

But, in my opinion, your intended use cases makes Mini-ITX very difficult to work with, mainly because of the lack of PCI-E connectivity that the mITX form factor imposes. All of my research so far has indicated that there isn’t very much support, if any, for multiplexing current consumer GPUs on the market like you would want to as a desktop consumer (while vgpu_unlock works on some recent GPUs, it doesn’t exactly allow you to forward specific ports to specific VMs), so I think that at least in the current status quo that this won’t be possible. That kind of means you have to either get two discrete GPUs, or a CPU with an integrated GPU. The 5700G APU wasn’t out when I finished my build, but I think I might still opt for two discrete GPUs because I’m not sure if there are any catches to passing through APU graphics. Have you already looked into that?

USB connectivity may also be another concern, since I think many motherboards have just one USB controller for all onboard USB, which means that if you needed to, you can only pass its PCI port through to one VM. While you can use qemu’s USB passthrough in Proxmox to pass single devices, I’ve found that the latency is honestly unbearable (especially for audio interfaces), hence why I use PCI passthrough for the USB controllers. This made mATX sort of the minimum I’d go for my setup.

The cases I’ve purchased and used so far that support mATX and are small enough to fit in my carry-on suitcase are, in order: Silverstone’s SG10, Silverstone’s GD05B, and the SAMA IM01. If you have no HDDs, I think the IM01 or SG10 are probably ideal because you have better airflow options (SG10 fits a huge CPU cooler too). I also wrote a few notes about other cases (this is a draft I never finished) that might also be of help.

Hardware choices aside, I’m using Proxmox (with the lae.proxmox ansible role) as my host system and an Arch Linux VM running as my primary desktop on the single slot GPU. I have a Windows VM using the other 2-slot GPU whenever I want to play games, and when I do I’m basically running two desktops simultaneously (Windows on one monitor, and on Linux I disable the monitor being used by Windows) and switch back and forth during downtime using a USB KVM. I don’t run a desktop environment on Proxmox, but in my current setup I do still have VGA access to the console because of the IPMI device on the motherboard, which I use for interacting with boot prompts.

I haven’t yet started to do so, but I’m probably going to use bpg’s Terraform provider for Proxmox to manage some self hosted stuff; I definitely need to cut down on some of my cloud expenses…

Open to questions if you have any.

6

@mutation666 thanks for the trust level bump and linking of the similar build. Was quite excited when I saw it, especially when I started reading about bifurcation.

@SgtAwesomesauce thanks for the feedback. Good to know that the bifurcation does not work on your mobo with bignavi. Do you think this is something that is gonna be fixed with future updates or should I look for a different motherboard?

@lae thanks for sharing your experience and knowledge. Sounds like you figured it all out.

To be honest, all of this starting to feel overwhelming. While I love thinker and researching technology, the part where I’m supposed to make a decision where I’m spending money is causing me anxiety (I guess that’s why I usually stick to working with software).

I’m gonna try to do some more research, but feels like there is no reliable way to do what I have imagined right now, at least not in the form factor I imagined. Debating with myself I should just scratch the whole Proxmox part, and just build a simple Mini-ITX workstation and run some docker-containers in the background.

1 Like
7

I know that feel, it’s kind of why it took me the entire course of a year (or was it longer…?) to actually decide to buy parts.

For me personally, the Sama IM01 is hardly any bigger than the NR200 (some people do call it the mATX NR200 clone, e.g. [1], [2]) so form factor wise I think what you want to do is totally doable. The case itself is like 60 USD.

If you are a bit adamant on using mITX, then I think that yeah, a more comfortable option would be to use your desktop Linux system instead of Proxmox as the host. This is what I previously did for multiple years before moving to using Proxmox for my workstation a few months ago. You can definitely use the integrated GPU in your APU for the host, but this will be limited by your motherboard’s onboard display connectivity (my current motherboard only has one HDMI port that can be used by an APU, for example), and pass through your single dedicated GPU to a Windows VM for games and what not. There’s a ton of documentation for these sorts of setups in the first place so it’ll probably be easier to find help, too. If the APU won’t cut it for your Linux games you can do the same thing with another Linux VM, too.

If a picture might help, here’s one. It’s not super small, but it’s absolutely not super big either.

image
image1753×1315 146 KB

1 Like
8

To be honest I don’t know. I could try it again, but that would require me ripping the whole rig apart and I’m not particularly inclined to try it.

I had meant to ask Wendell about it, but never got around to it. I’d really prefer to have a non-integrated GPU for my host GPU, so I can save on memory bandwidth.

1 Like
9

There are a few things to consider here. If you are building a workstation, especially if you are a developer, I’d recommend you go with fast memory, instead of ECC. In addition, I don’t think you can passthrough the iGPU of an APU. I could be wrong, but I never seen anyone successfully do that on Ryzen APUs (and the ones that I did see, were trying on previous gen bulldozer stuff and always ended in failure).

Mini-ITX might be a good choice for portability, but if you want expansion and especially lots of storage, I don’t think that’s a good option. Grab a micro-ATX. One annoying thing that I observed and Wendell also pointed it out is the micro-ATX motherboards kinda suck compared to ITX and ATX, they are in-between them, making them the easiest to manufacture and cut corners around. There may be some decent options though, but I’m not really familiar with modern motherboards. You can still carry around a micro-ATX system if you get a slightly slimmer tower (or if you find one with a handle).

Regarding Proxmox, for your situation, I think it’s better if you just run your distro of choice as a host and instead run LXC containers for the services you need. If you are comfortable with the terminal, LXD makes it really easy to set up new containers, where you can run things like turnkey-linux, centos, rocky, alma, springdale, ubuntu, debian, alpine and void, in which you can install things like gitea, nginx, nextcloud, seedbox, omv and more.

But, then there’s another issue. I’m not familiar with GPU partitioning, so that might be an issue, but if you get a microATX board, you can always just add a pro-tier GPU to it. If you intend to run Pop!_OS as the host OS instead of Proxmox, I belive you won’t be able to use the dGPU if you are going to pass it through to other VMs, so in that scenario, Proxmox would be a better option, because you can just pass the GPU to your main desktop (Pop) and if you want to then turn to gaming, you can just turn it off and power on another VM.

Regarding storage, I would say you use an m.2 for Proxmox and either 2x m.2 or 2x sata SSDs for the raid you are going to run your VMs from. I recommend you grab 2x 4tb drives (or less if you don’t need that much space) and put them in RAID1. I don’t recommend you run important stuff on just 1 drive, even if you have another one where you backup your data, but that’s just my opinion.

On RAM, if you don’t intend to run a lot of VMs, you can get away with 16GB of RAM, but that is cutting it really tight. 32 GB would be my recommendation.

Then, you can always upgrade later. IMO, anything would be better than the i5 7200U, so you will notice a big difference with even the 5600G. 12 thread will be plenty, but if you are going to run more than 1 VM, I’d go with the 5700G just to be safe. Containers don’t really care about the CPU, they can share just fine. Now, back to upgradability. You can use the iGPU in either the 5600G or 5700G (the difference is almost negligible, but fast RAM makes a big difference) and it should perform well for some 1080p gaming, but again, that would mean you have to run Pop!_OS as the main OS. Once the GPU shortage wears off, you can grab a GPU and convert Pop into a VM. This way, you can save some money on components, like getting just 1 SSD.

10

So I’ve got 64gb (2x32 of 3600cl18 that I’m running at 3733c17). Its totally solid, and I can do looking glass on the apu at 3840x1600@75 just fine. Its definitely pushing the limits and I’ve considered trying to see if I can find a faster 2x32 kit, but at the moment, I’m happy.

1 Like
11

I have also done more or less exactly this! I wanted a super-budget mITX virtualization station to take advantage of the 1070ti that I had after my last system upgrade. I really, really enjoy it; computers are really good these days. I think you’ll be quite satisfied once you have it running.

My system uses the i3 10100 (!) and passes through my old blower 1070ti. My case is exactly the NR200. I have some specific suggestions for your consideration.

RE: Storage
Since you’re going for some pretty nice hardware, I think I’d get rid of the HDD and leave the role of storage/backups to a different, dedicated device like a NAS, and do your backups over the network. Maybe this is not an option for you (I had a NAS already), but getting rid of the HDD has a few benefits:

  • The 5800X, from what I’ve seen on the AMD subreddit, can run hot. You might appreciate using a 240mm/280mm AIO to take advantage of the large open side intake on the NR200 to keep your parts cool. The bottom intake can then be used to keep your GPU cool, especially if you can keep the bottom off your desk or the floor (mine is suspended beneath my desk).
  • I don’t believe putting two SATA SSDs in RAID for performance makes a lot of sense in the era of Gen4 PCIE SSDs. If I were you, I’d buy a single high-capacity 2.5" SATA SSD like a 4TB MX500 instead. That way you can buy another 4TB when you need more space while avoiding the poor performance of the QLC SATA drives.
  • I don’t think you need 1TB of space for Proxmox as your host. You need very little space for ISOs and the OS basically. You can save a little money here and put it towards something else.

RE: other hardware

  • I use virt-manager and not Proxmox for managing my virtual machines. Having an iGPU really made configuration a cinch. You can get away without it, probably, if you’re willing to deal with not having video output. But, I think you’d be better off with the 5700G than the 5800X.
  • “Fit in carry-on luggage” - :grimacing: I’ve never tried traveling with my rig but I wish you luck - it’s quite dense with all the hardware packed in!

I echo @lae’s sentiment that you might find mITX limiting, but you’ll have to make some sacrifices if you want to keep your build small. As long as I understand your comments correctly, I think you could have a great experience using something like your original mITX build (I’d make sure to go for something with an iGPU to make your experience more painless though). I can’t really comment on @lae’s thoughts about USB passthrough; SPICE redirection and QEMU USB passthrough both work well enough for me, at least.

1 Like
12

You can use virt-manager from another PC if you want to run your host OS headless. You install virt-manager on a linux box or even a live environment and connect to your compact virtualization workstation to configure your VM once, then have it auto-start when you power on your host. After that you can install virt-manager on your guest OS and connect to your host OS to add more VMs.

In a pinch, you can just ssh to your workstation from your phone and virsh start vm-pop_os-01 or however you named your VM.

13

I completely missed this in the original post (or actually, I read that you decided on a 5700G so I must have blotted it out of my mind). Yeah, if you do reconsider your earlier CPU choices, I would just suggest to not get the 5800X when working with SFF. I’m totally fine using my 5950X with a low profile cooler but I doubt it would fare so well with the 5800X, which was what I was originally going to get until I found out it ran considerably hotter. So 5900X or 5700G would be my suggestion.

To give an idea, this is how much my Proxmox host uses after I’ve been using it a few months—under 20GiB. I actually do run VMs on the same 1TB disk as Proxmox though (would just be a waste imo to have a small SSD take up an NVME drive slot…). df only shows mounted stuff so the one here is used by an LXC container running an apt cache server.

Filesystem                           Size  Used Avail Use% Mounted on
rpool/ROOT/pve-1                     446G  2.9G  443G   1% /
rpool/data/home                      443G  256K  443G   1% /home
rpool/data/home/root                 443G  512K  443G   1% /root
rpool/var/cache                      444G   86M  443G   1% /var/cache
rpool/var/lib/vz                     454G   11G  443G   3% /var/lib/vz
rpool/var/log                        444G  183M  443G   1% /var/log
rpool/var/spool                      444G  4.2M  443G   1% /var/spool
rpool/var/tmp                        443G  384K  443G   1% /var/tmp
rpool/data/home/lae                  444G  141M  443G   1% /home/lae
rpool/data/images                    444G  528M  443G   1% /var/lib/vz/images
rpool/data/images/subvol-103-disk-0  8.0G  558M  7.5G   7% /var/lib/vz/images/subvol-103-disk-0

So the two main prerequisites, I guess, for USB passthrough to not be a problem is if:

  • you manage audio on your host system, which likely means you’re running a desktop running a pulseaudio server and your qemu just connects as a client to your pulseaudio server for audio.
    • possibly, passing through onboard motherboard audio (i.e. you use audio outputs from your case or rear motherboard I/O) doesn’t have the same latency issues as USB audio but I don’t have any information anyhow for this scenario
  • the games you play don’t require low latency input, e.g. MMORPGs like Guild Wars 2 or FPSes

The main symptom/issue with passing through USB audio interfaces is audio crackling, and it’s not that uncommon. It can get annoying pretty quickly.

I don’t think I ever got SPICE configured correctly with a passthrough GPU so I can’t comment much on that, but there was noticeable mouse delay when playing APEX using QEMU USB passthrough and evdev so…yeah, I had to stick with PCI passthrough for the USB controller. Would be nice if I could just use Looking Glass with SPICE though, but I’m not sure where you would start with a Proxmox host for that; I’m guessing that would work with a virtualized GPU like you’d get with SR-IOV, however. (And if so, those Ampere cards kinda look attractive…)

2 Likes
14

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.