Question: is it possible (or sensible) to setup one’s desktop computer as an xcp-ng machine with other virtualized systems but also including a virtualized desktop with GPU passthrough to have a desktop-like experience that essentially hides the xcp-ng component except when accessing through XOA on the virtualized desktop with GPU passthrough?
In some sense I’m trying to see if this is:
Possible
Wise
Painful (on a scale of 1-10)
I’ve been running xcp-ng on a very underpowered spare machine, but it’s been great. Not in a position to get new hardware at the moment, but converting my existing desktop (better hardware, more storage) would be possible provided I’m not loosing too much otherwise.
Not really, I want my workstation to always work. Always. This won’t be possible with a setup like this. You might have problems with programs misbehaving and games running very slow or not at all. It will work for at least 90%, but what about the last <10%? Is it worth to you that you’ll have to tinker in those cases?
I have ran XenServer 7.2 I believe (around 3 years ago) on a Dell Optiplex 360 with 4gb of RAM (a core 2 duo something). Had 1 Debian VM and 1 Windows 10 VM inside it. No GPU passthrough, but the RDP experience was… well, decent, for what I was using it. So to answer your questions: yes; no; not really, depending on your experience (3-4).
I don’t know, it really depends what your use case it. For this kind of 2-OS-1-PC kinda situations, I rather recommend running any Linux as host and Windows as guest. That way you only need the iGPU for Linux and passthrough only 1 GPU for windows. If the motherboard / CPU has no iGPU, you can go with 2 VMs (which uses a little more resources, but otherwise makes the host Linux more stable, requiring less restarts). But again, what are you trying to do?
I echo the answers above, but the ultimate answer depends on: #a - what you want to run as VMs / services [and how they critical they are] #b - what you plan to do on the virtualized desktop with GPU passthrough #c - what CPU / how much RAM / what storage devices you have…
If #c is less than 6 core, and 16 GB or less, and/or spinning rust, then [IMO] you no chance…
If you have a fairly powerful, machine - e.g. 8+ cores, 32+ GB RAM, multiple SSDs, then you may have a chance, but if you’re running resource intensive background VMs and wanting to play games and/or run graphically intensive software [Photoshop/Lightroom, video editing] in your desktop GPU VM, then you’re risking poor overall performance and / or crashes that bring down everything…
So, can you outline what your vision is?
FWIW, I’ve previously been using Proxmox for GPU pass-through experiments with macOS & Windows VMs, and I’m going to have a go [over the next few weeks] with xcp-ng [B450 motherboard, Ryzen 1700, 32 GB ECC RAM, Radeon Pro W5500 GPU], so I’ll let you know how good/bad the experience is…
[but I don’t plan on having any other VMs running on the machine, as I have Orchestra already installed and running on another headless xcp-ng box]
Thanks to everyone replying with such detailed responses. Mostly this confirms my suspicion that this setup will be less-than-ideal. At this point my main machine is ~7 years old (and wasn’t top-of-the-line when I built it).
@edge-case - the feedback on good/bad experience with xcp-ng on your Ryzen setup would be helpful.
I think I’m basically outgrowing the setup(s) I have but I’m at present not in position to buy any new hardware. I’m using some very old desktops for: PFSense and TrueNAS, and one that just died (again) I’d resurrected to try out xcp-ng and found it very useful. I’m been a Linux-only desktop user for 10+ years now, but setting up useful services locally has only happened in the last 4-5 years.
In some ways this is pushing me to think that my next move should maybe be transitioning to something like a used Supermicro Rack-Mountable 2U and just kind of accept that I like tinkering around with VMs, running local services, and having plentiful network storage.
Open to any and all further thoughts! Appreciate the helpful feedback so far!
I like putting old and capable hardware to good use if I can help it. Sometimes it doesn’t make sense (like in the power consumption department, which is why I am a huge fan of low-powered stuff and why I have 2x AsRock J3455Ms and lots of RPis). I used to have my PC with a Pentium G4560 (a mere dual core with hyperthreading and 8 GB of RAM) run Manjaro as host and Windows with GPU, SSD and PCI-E Card to USB, all passthrough’ed. But I realized that having such low specs and low power consumption, it’s better to split my PCs. That one is running Windows (I have no idea why it is even powered on anymore, I only play DOOM mods on it). My main PC (which I’m typing on) runs on one of the J3455Ms (still in the process to move full time on a Pi 4 8gb), the other one runs pfSense.
What does the above mean? Well, if you have an old system and you want to run both Linux and Windows at the same time… then don’t, just get a newer-ish PC (something like a Dell Optiplex 3020 or 7020) and run Windows on that and keep Linux on your older boxes and downsize your resource requirements as much as possible (needs some relearning to do stuff and get into minimalist Linux setups). Or if you really can afford, just get rid of everything and get an 8 core Threadripper 1900X (they’re pretty cheap these days if you can find them) or Ryzen 1800/X or even a 6 core variant like 2600 or 1600AF and split that into VMs. A rack server only makes sense if you have a rack and a place for it where you can’t hear it. I’m running an Intel Server Board with a Xeon x3440 inside an old Antec desktop case with 24 GB of RAM. It is pretty quiet (compared to my HP 48 port switch), but I’m insane and sleep near jet engine sounds (would not recommend). Or if you really want ECC (which is why I have this old Xeon in the first place, I took it from work, because we wouldn’t keep a desktop case in the data center where we moved).
There are so many things you could do, really, just depends on your preferences. Keep old stuff and “decentralize,” buy one new big bulky server and remove (almost) everything else, buy a decent PC and keep some of the old stuff, make a cluster at home, idk… You still haven’t told us your use case.
If you really like spinning VMs up, if I was you I’d get 2 mediocre spec’ed PCs, one for Windows and one to run Proxmox or Xcp-ng on it and spin VMs and containers on. What I have right now is:
LackRack and 3x 2U chassis with PC components and one empty 2U one (can’t use it at all), out of which are:
pfSense box (always on)
main (Linux) PC (always on)
an old Athlon x2 4450e running Windows 8.1 (haven’t powered it since probably October last year or earlier
(the above 3 are rack mounted)
ProxmoxVE box (tower PC case, but server components, always on)
windows 10 box (SFF PC, always on)
Pi 2 being a jump server (through SSH) from WiFi VLAN to my other LANs (always on)
Pi 3 (haven’t powered it on in a while, currently OS-less)
Pi 4 (trying to move to it, I’m really close, always on)
Windows laptop, only using it rarely, when WFH (always off)
8 inches Windows tablet (used to be always off, updated it and will try to move my work activity to it and gift my laptop to my mom, because her kinda broke).
I got too many devices. I was thinking to make a cluster out of my main racked PC and the SFF one, together with the server and run HA and replication on my most important stuff. Was thinking to make the Pis run micro-services inside LXD, but not sure what I would run on them that my cluster won’t be able to handle (and do it with some level of redundancy).
Just a quick inventory suggests I have way too many computers too [and I was pleasantly surprised by some of the uptime numbers - my UPSs “done good” ;-)…]
You’re philosophy / set-up seems very similar to mine. I also like the approach of small, lower powered machines used for dedicated tasks, and then using older, consumer grade hardware for home built servers… I use TrueNAS core for important data, Proxmox for VMs, and a Proxmox [and soon to be XCP-NG] machine for for experimenting…
To the OP - it all depends on your use cases, what you want to do, and how much money [if any] you can allocate / justify…
Personally, I’m fortunate to have enough room in my house [including large unfinished basement] for physical space not to be a problem, and enough disposable income that multiple small machines, and using old hardware as I upgrade my workstation/gaming computers, are within my hobbyist budget… [I’ve been tempted by current generation Threadripper or Epyc, but always decide to stay away…as that’s not affordable or justified for me, especially when Ryzen 5 1600 [were, at $79] and 3600 [now, at $159 or $179] , are such good value, and ASRock B series motherboards are ~$70-90… [I live near a MicroCenter…]]
TL:DR - my collection consists of Apple stuff, and small bare-bones computers [e.g. Intel NUC] and home built machines using practical ATX tower cases [e.g. Antec], and Ryzen 5 / 7 CPUs with ASRock mboards [and ECC support] is my chosen path…
Gory details [in my defense, this was accumulated over multiple years…and I don’t often check my electricity bill ]:
Real life/work:
Apple laptop and Mac mini w/eGPU for important, real life stuff [e-mail, finances, taxes, personal photo and video library, e-shopping], and iPad for entertainment & reading… [I’ve used OS X as my “primary OS” since the day 10.0 was released…was Amiga OS & BeOS before that… ]
[Company provided Windows laptop for work, supplemented by a personal, 6 year old iPad Pro for multitasking - i.e. video conferencing/meetings]
Workstations / Gaming:
Ryzen 9 / Radeon VII, ASRock x570 Taichi : [by far my most expensive computer ever!] - for my personal workstation & gaming [triple boots Linux and Windows, but mostly Ubuntu with cutting edge kernels and graphics drivers]
Ryzen 5 / 5700 XT for family / living room gaming - Windows & Ubuntu [depending on the games] - connected to a large-screen TV…
Small bare-bones:
HP T640 [embedded Ryzen R1505G APU. I installed Ubuntu. Always on, and used as a desktop to manage all my servers and VMs from my home office [bargain on eBay for $120… NIB, including a 16 GB SSD and 4 GB RAM]
3 various Intel NUCs for services/experiments
Pentium NUC7 - Roon audio server [optically connected to my audio system] [300+ days uptime]
Nuc8 [Crimson Canyon - the 10 nm CPU with AVX512 and the discrete AMD RX 540 GPU - used for experiments with Clear Linux and AVX512]
Nuc8 Core i5 - emulation gaming, random experiments… currently has a test installation of XCP-ng and and an Orchestra VM
Servers:
home built NAS - primary file server [TrueNAS Core, Ryzen 5, ASRock Rack X470 mATX mboard, 32 GB ECC RAM, Antec ATX Tower Case, 8 WD RED HDDs…] [[250+ days uptime]
home built dedicated backup-NAS [Ryzen 3 APU, ASRock ITX Consumer board, Chenbro ITX case, 4 WD RED HDDs…] [400+ days uptime]
Proxmox box [Ryzen 7, ASRock B450, 32 GB ECC RAM, Antec ATX case] - VMs: Plex server, TrueNAS for shared media library, various other VMs - e.g. Pi-Hole] [150+ days uptime]
Experimental VM box - Proxmox - Ryzen 7, ASRock B450 mboard, 32 GB ECC RAM, Thermaltake mATX case] - used for GPU pass through experiments with looking glass, other Linux distros and macOS, etc… will try XCP-ng w/ GPU pass-through on this box…
if you have a little cash to spend check Ebay there are Xeon workstations 3 to 4 years old for like $150.00 or less and with a little research can be upgrade for little or nothing. xcp-ng is great software the only down fall is you really need a seprate NAS to store you ISO/VM’s and external storage on. HP Z640 Workstation 1 X EIGHT CORE 2.40GHz E5-2630V3 8GB 500GB HARD DRIVE TOWER | eBay
While not specific with XCP-NG, I ran this with VMware ESXi for over a year. It’s certainly doable, depending on your hardware config. Mostly because you need to passthrough USB ports as well to get it to work properly, and this is usually overlooked. Make sure if you want to do an auto-start setup with XCP-NG that you can pass through the GPU and at least one USB ports from your motherboard that use CPU lanes to the VM. This will allow you to reboot the VM while using it
]: