So I decided that maybe I’d buy some NVidia GRID from Ebay for my home virtualization server / workstation because basically why the hell not. So I did some research in the field of KVM compatibility, Linux guests, Windows guests, AMD S7150 as alternative etc. and came to following conclusion:
Linux guests do not work on AMD S7150 MxGPU. Aside from that it’s much cheaper solution with no subscription based bs. Totally suitable for home poweruser.
NVidia GRID K1/K2 (first generation of GRID gpus) don’t support Linux guests as well
NVidia supports Linux since Tesla Maxwell which is still quite pricey but kind of affordable
In order to get support for Linux guests you need highest tier workstation license for GRID software
Which is 350$ annually per VM
Tesla M60 supports up to 32 guests.
Which is 32 * 350$ annually for basic Linux VMs (512mb vram, probably 1 cpu core and like 2gb ram)?
So in other words drivers are subscription based so NVidia can just pull the switch and turn your GRID card into paperweight in few years because license for old drivers will expire and new drivers won’t support old card.
Are you f*cking kidding ME???
Am I missing something or it’s such incredibly terrible licensing that it should be illegal? 5 years TCO is like 60 000 $ of such card (just software and hardware without maintenance, thin clients and electricity costs). It’s insane, Jesus! At this point it’s cheaper to just buy 32 workstations.
I have met subscription based appliances in the past for many enterprise products but then typically you get box for free so it’s actual subscription based solution. While this is both pre-paid hardware PLUS additionally subscription based drivers so without subscription your hardware has no intended value and you end up with expensive paperweight. It’s sick, what the hell.
Now compare it to AMD S7150 for 2k$ no strings attached. The only downside is lack of Linux guest support. I think I’ll just go with AMD and stack bunch of crappy consumer gpus for passthrough to linux VMs via pci-e risers because it’ll be like 20x cheaper… But damn it’s unbelievable cost difference. And GRID is unbelievable ripoff. Why nobody is talking about that?
GRID was an enterprise product line that has completely different licensing and cost expectations than consumer stuff, and it’s also a dead program. I’m not sure of the licensing cost, but I’m pretty sure that nobody was actually running 32 vms on one card. When a card has a maximum of 16GB combined Vram, and you’re providing the card because you want it to actually do something computationally (like you would be doing if you were a datacenter buyer…), your not going to be running 32 different VM’s with 0.5GB of VRAM each. With that little VRAM and the cards being stretched so thin by the load of 32 vms, they’re not going to have any headroom left to actually compute something at a reasonable speed.
People would buy something like this so they can run a handful of VMs per card, and also do that in a very small amount of space compared to what you would need to support one vm per card. When space and density are of concern, those GRID cards probably made a lot of sense. The software cost really isn’t even that terrible compared to what windows server or esxi costs. Windows server 2016 datacenter is a cool $6k+ and vCenter server costs almost $8k per production server. When you consider the $350 per VM per year cost that’s really not that bad, especially if it means that you can use as a GRID card to reduce the number of servers you need, reducing the cost of ESXI or windows server licenses plus idk, the cost of more server hardware.
Heck just even compare the price per license against the prices that server operators are charging to use hardware like that. Google Cloud offers a range of Tesla’s for people to use. The cheapest of which being a Tesla K80. At the Tesla K80’s $0.45 cost per hour of use, if used for an entire 30 day month, you’d be paying google $230 just for access to that card, on top of the cost of the rest of the VM. At that kind of rate, it actually makes a ton of sense for google to pay that licensing cost. Not to mention the higher cost that would have been charged for access to hardware as good as GRID cards back when those were relevant…
I wouldn’t say it’s that good deal. Even like you said 6k$ per Windows Server DC and 8k$ for ESXi host. Our company paid 10k$ per Dell PowerEdge R720 after some discounts. 5 hosts are running together 97 VMs. It gives on average around 20 VMs per host. 32 VMs per card do make some sense if you’re counting on high density VDI and unloading CPU.
If you load up server with 4 cards like those you can handle 128 high density VDI desktops per server with some dual Xeon and lets say 256gb ram. So we have 24k $ for 20 VMs per host vs (lets say hypothetically that such R720 could handle 4 GRID cards and 128 desktops with its CPU power and RAM which is not true obviously) 264k$ for 128 VMs in 5 years. It gives 42k $ per 20 VMs. More than twice. It’s still terrible deal!!! If we’re talking about windows realms it’s even worse deal providing that S7150 can also handle 32 VMs for 2k$ / card so with the same hypothetical case it’d cost 32k$ per 128 VMs giving 5k$ per 20 VMs cost which is actually much better deal than our current setup, ignoring fact that such R720 wouldn’t be able to handle such desktops on CPU and RAM level but that wasn’t the point. The point was that 350$ per shitty high density Linux VM is ridiculous price. and when we’re using less VMs then cost of card itself becomes more serious concern and it’s even worse deal.
No matter how you look at it 60k$ for 32 high density VDI desktops in 5 years is quite bad deal. This licensing scheme is useless for high density VDI desktops for casual (non-poweruser) workstations.
And there’s plenty of reasons to consider hardware GPU virtualization for high density desktop starting from unloading CPU thus packing even more VMs per host. Even I do consider such card for standard software developer workstation which would be totally fine with 512mb vram but software VMWare GPU just can’t really handle two fhd displays desktop compositing well and everything stutters noticeably, decreasing comfort quite a bit. My old nvidia 9600 gt was totally fine with such workload and it’s old junk 512mb GPU.
The core of problem is that LInux is supported only by highest workstation GRID tier which just makes no sense at all for high density VDI.
The point of a solution like this isn’t to run 32 VMs per host though… You add a GRID card because you need GPU Compute not just a bunch of virtual desktops. If you wanted more virtual desktops, you would just add cpu to deal with the emulation overhead of not having dedicated gpu resources for every vm. Even if you wanted to use grid instead of adding cpu resources, according to the Nvidia pricing pdf, it is not $350 as you said, but you can do “GRID Virtual Pc’s” for $50 annually, or just pay a one time fee of $100 per Grid Virtual PC License. There’s also a $25 dollar support thing that’s required for the first year only of the perpetual license agreement…
Indeed I must have seen some old licensing document. Thanks for heads up. It’s still terrible but a bit less than it seemed to be. 125$ per VM perpetual is not that terrible.
have you considering rolling back to ESXi 6 U3 and running vSGA with the compatible AMD card? it’s sub $200. it doesn’t have ALL of the API protocols but, enough to do 3D and acceleration.
so if i want 1 or 2 VMs at home to play with i can do the one time fee or $50 annualy and still get 3d acceleration? does it have no linux support all or only when paying?
I know it’s a bit late, but Kepler GRID cards might have it easier on the licensing front. According to a post on Reddit r/homelab, regarding the nVIDIA GRID GPUs,
GRID K-series cards don’t require a license to activate their vGPU featureset. M-series do, and it’s an expensive per-VM license.
Unfortunately GRID K series doesn’t support Linux Guests. So at this point I could just get FirePro S7150 which also doesn’t require license and is cheaper.
I heard AMD slowly starts supporting Linux guests so I might take a look. Unfortunately Instinct cards are still so fresh that I didn’t see anything reasonably priced on Ebay :C
Just a side note. I was looking into getting an AMD FirePro S9300 x2 a few months ago. But supply became limited and the prices skyrocketed to the point that I had to abandon it My SR-IOV dream died on that day. So, I’m stuck with GRID sadly. If you ever find a solution, pls let me know…
simply get a plx card and pass assign any cheap gpu card. It will automatically be faster thant those old grid card, more efficient and all cheaper as well.
When the price is right, and I have more time I am planning to experiment, considering certain cards do not have licensing and others (according to v4.10 documentation [1]) will just remind you. Looking at later grid releases, they are much more harsh with how they behave if you forget to put in your license (15 FPS cap and throttled performance at 20 minutes, and they will basically cripple your card after 24 hours)[2]
[1]
Quotes from the documentation and to provide a source to an earlier users correct claim:
“GRID vGPU is a licensed feature on Tesla M6, Tesla M10, and Tesla M60 GPUs. When booted on these GPUs, a vGPU runs at full capability even without a license. However, until a license is acquired, users are warned each time a vGPU tries and fails to obtain a license. … NVIDIA GRID K1 and K2 GPUs do not require a license to run vGPU.”
Source (v4.10 documentation) :
[2]
linked to the section of more modern documentation with the listed behaviour:
v13:
My SR-IOV dream died on that day. So, I’m stuck with GRID sadly. If you ever find a solution, pls let me know…