This is my first post here, so if I miss something obvious please let me know!
End Goal: Have a lab/workstation with 128gb~ of ram for virtualization, labbing, and general workload stuff with some ML capabilities as well as handle a NAS of up to 50-100TB of storage. (zfs would be nice, I wanna learn more how it works)
Issue #1: Unable to find or source cheaper older lab gear at a reasonable rate that has the memory specs that I’m looking for. Most servers I see on ebay,amazon, etc only come with 8 or 16gb of ram out of the box and sourcing additional is super expensive still for some reason…
Issue #2: If I go the threadripper route, it seems ESX isn’t support as of this writing. Perhaps I would just make this a linux build to match the rest of my ecosystem at home but I’m not sure if the threadripper is cost effective for the end goal listed above.
Issue #3: If I go the workstation route, I would like the ability to have a ton of storage attached to the host to use as a NAS for my home network.
Option 1: Go the Lab route for cheaper equipment up front with the ability to use ESXi.
Option 2: Go the workstation route for a higher cost upfront, and sacrifice some of my ‘end goals’ like storage and memory.
DDR4, especially ECC Reg DDR4 is pretty expensive everywhere. I mean $10 usd per gig is not a bad price in the current market.
DDR3 can actually be pretty cheap comparatively, at like $2-3 per gig used.
Unless you can get a server for almost free and have cheap power, don’t bother with AMD DDR3 based servers, but Intel LGA 1366, 1356, and 2011(-v1) are still pretty good. You should be able to get 8x16gb of DDR3 for under $300.
Well, when you hunt / wait for cheaper used ddr4 Reg Ecc, you can get a 16GB stick reasonably reliable for about 60 to 80€ here in germany.
Sometimes even lower, won a 2x16Gb kit for 90€ once.
And one thing not many talk about is that X99* supports Reg Ecc if the CPU supports it too. So you could get yourself some used X99 with a nice Xeon E5 1650V3 6c12T for maybe 200$ or 1660V3 8c16t for maybe 400$?
Or you go for a a nice high corecount ES which are sometimes around 400$
And there is a Turbo Glitch in the 00 Microcode, so overclocking is possible, but not easy.
A good 2696V3 ES gets sometimes as low as around 400$ if you again, hunt for it. Warning, no QEY8s!
X99 Boards should be relatively cheap on the used market.
On the Other hand, SandyBridge-E / EP and IvyBrdige-E / EP are Cheaper in terms of CPUs and Ram, Boards depend on what you want.
i have no clue about 1366, though i think you’d need to dual socket for the amount of ram if you use 16GB dimms, since only 6 slots per cpu ?
Workstations don’t sound like jet engines, and that consideration more than any technical reason is why I’ve gone the workstation route for my homelab.
I’ve also shifted away from big, beefy monolithic servers in favor of more smaller boxes. The compute and memory resources available in server gear is insane. Even my professional workloads don’t benefit much from scaling vertically these days, and I’ve found that being able to scale horizontially has greater benefits.
I’d say spend your money on a few older, cheaper workstations and build a cluster. This gives you more PCI slots as well, and more potential for GPUs for the ML workloads you’re targeting.
Supermicro X8 gen hardware is very cheap, takes DDR3 and supports 5600 series Xeons. I have several of these and they work well. They will run ESXi 6.5, but not 6.7. However, electricity consumption and noise are very real concerns. If electricity is expensive where you live and/or you do not have a sound-isolated place to put it, you should go the workstation route.
A a workstation could be built with similar spec’d used hardware without the fan noise of server hardware so you kind of get best of both worlds there…