It’s gonna be used pretty much as “i don’t wanna run it on the main PC constantly” box, stuff like Stable Diffusion, Mathlab/Vivado/Quartus servers, Navidrome, just general “secure” network storage, etc., running (probably) Windows Server.
Right now the only thing out of the list that is bought - is the case, so anything i’m going to get will have to fit in there. Also, since it’s gonna be used for different FPGA stuff too - a lot of pci-e slots are very much welcome, especially faster stuff.
Any suggestions on motherboard/CPU/GPU combinations? I’ve heard that Tesla GPUs are a pain in the buttocks to set up in “normal” PCs, and i didn’t plan on how i’d route cooling to it yet.
Maximum budget i’m willing to spend is around 100-110k ₽, used stuff is fine except for storage and PSU. Shops like X-COM, DNS, Onlinetrade, Aliexpress and OZON are preferred due to avaliability. For the used stuff - Avito.
UPD: Most links are “page does not exist” for me now.
That server is good performance but it is also drawing a ton of power.
When you run a home server that is always on you will use ~20% of whatever your from the wall idle power draw is. That is, if the idle is 100W your server will consume 120W on average. If it is 200W it will consume 240W. This is a general rule of thumb though.
So why does this matter, well, every average watt on a 24/7 server will use .72 kWh per month, or 8,76 kWh per year. Those server parts? Around 250W-300W average, which means your yearly power draw will equate to 180 - 216 kWh per month or 2 190 - 2 628 kWh per year. That usually equates to hundreds of dollars that gets paid per year.
I would invest ~$1k - $1.5k in a low power server that handles most always-on tasks and then use IPMI to start the big iron. If the cost of running two servers saves you $1000 over a five year period, it is so worth it.
Right now the power draw price won’t really matter, since currently i pay a fixed rate regardless of kWh’s spent (in a reasonable range, but they have no issues of me running 1-2kW heater semi-constantly).
As of later - it’s gonna be, at worst, 0.079$ for 1kWh.
So right now i plan on getting the more powerful one first, while the prices on used EPYC parts are still reasonable (motherboard got +25% increase right after the new year’s eve), and maybe the less powerfull and smaller box for just the Navidrome later down the road (as this one will have to be 24/7).
To be honest, I’m more concerned about fitting all that stuff in the chosen case, espceially since i do want to install at least 3-4 RAID drives in it as well.
Very similar to what I’m in the process of building out right now, minus the GPU and storage. I’ve no need for a GPU (save those watts) but have a bunch more storage.
I would suggest looking at motherboard and going with the H12SSL-i. When I was pricing, I was able to find them for a very similar price on the second hand market as your MB (which was my first plan) but after seeing the newer revision of the same board, but with differences like Gen4 PCIe, one more PCIe slot, onboard SlimSAS, it was an easy choice to got with the H12SSL-i
Not sure what the secondhand market looks like where your at or what sites your looking at, but it was not much more $$$ for me, and well worth the benefits.
And FWIW… $0.08 - %0.11 per KW/h here, depends on the time of day
At $0.079 that is still $173 a year for a server that draws 250W on average. A 50W server would cost $35 - You would therefore save $138 a year by going with a low power server.
The advice on getting a secondary low-power server is definitely something i’ll look into, but since i can only get one at the moment - i’d go for the more powerful one first.
The comparison for the price goes something like this: H11SSL goes for around 29-32k, H12SSL goes for like 68-72k, and that TYAN motherboard is around 40-42k, while still having PCI-e 4.0. I was also looking at EPYCD8 mb instead of H11SSL, but it cost nore due to lower availability.
As i said, i was mostly looking for the used stuff on Avito and Aliexpress.
0.079$ is the price during the day, nightly one in that tariff is only 0.030$.
I’m so jealous at those power prices. Here i am paying 0,48 euro per kwh.
I think those parts are all second hand? Looks fine and probably not really possible to get more performance for less money. For a server i would have a bit more of a better storage solution. Like running zfs in proxmox for data integrity, but that does mean multiple drives/ssd’s.
The current solution with just one ssd is a “minimum viable solution”. I can add like 6-8TB HDDs later on for bulk storage. Those are around 16-20k rn, so they aren’t a priority.
Since the motherboard prices seem to fluctuate quite often, and not in a nice “price dropped” direction - i wanna choose one as soon as possible.
Just as a reminder - the total price for that basic setup shouldn’t go above like 1200$ or so for now.
Me personally I would invest in an Asustor Flashstor + 4TB or 8TB drives (depending on what gives the best bang for buck). At $449 for the base unit plus $180 for 4TB m.2 SSDs, it is amazingly good value. $1500 for an all SSD NAS 20TB redundant storage is amazing and the price per TB is only going to go down from here. It draws ~17W from the wall with six drives. That is ~150 kWh per year, or a cost of $12 a year with your cheap electricity.
The Flashstor is, however, also in a completely different league and is not really suited for heavy workloads. A SOHO is not a heavy workload though
The only two competitors to that hardware I could see is a W680 build with a 24 core / 16+16 threads Core i9 14900k, or a B650 server build with a 16 core / 32 thread 7950X. Better IPC, better idle values, worse bang for buck in both cases. You could build these systems for ~$1000-$1500 excluding drives though.
Stuff like Asustor would definitely look better as an always-on 24/7 server for stuff like music and torrents, but i do in fact need something to offload heavy stuff to from my main rig which has to do everything both light and heavy servers will. Since these are, to my surprise, readily awailable locally - i’ll look into them.
The W680 and B650 stuff unfortunately lacks in PCI-e connectivity and maximum memory. Besides, i could just keep using my current setup with pretty much the same effect (Z690 + 13600K).
Maybe LGA3647 or some older LGA2011v3 platforms could be better bang for the buck than 1st/2nd gen EPYC? Especially if i’ll be buying PCI-e 3.0 motherboard like H11SSL?
Yep, I agree that the Flashstor isn’t going to fit that use case. For modest server needs like mine it fits the bill and then you can use it for the 24/7 stuff and then boot your other machine for offloading when required.
Of course if you need a render farm 24/7 or something the yeah, Flashstor is not covering that.
Well… Yes and no, 192 GB of memory is a freaking huge amount of memory already, and in a world where almost all add-ons are perfectly fine with USB C speeds and latency (yes, even Pro Audio stuff), more than one full x16 slot and three m.2 slots is going to require some pretty exotic use cases.
As stated before though, if you really need those lanes then cough up for an EPYC or Xeon server. These are not cheap with $700 Motherboards and $1000 CPUs, new. But they do exist.
I don’t really care about M.2s in such a server, since i can just convert regular slots into them. But i at least need one full connection to a GPU, and another - to the FPGA card i’ll be tinkering on (which uses x8 1.1 slot). And that’s not counting any interface cards and so on. You also are surely aware of the prices on dense memory (to get up to 192GBs), while i can get all 8 channels with 16GB sticks of used memory for just 16k total (my 32GB kit of run-of-the-mill DDR5 currently costs 14k).
Welp, to the H11/TYAN i go, i guess. I kinda hoped there’s some cheaper alternative out there.
Nope an x8 PCIe 5.0 connector is not able to be saturated by current CPU tech. For real. Even an x8 PCIe 4.0 is damn difficult to saturate. Anyone telling you otherwise is peddling snake oil.
So what you need is a dual x8 / x8 for that setup, technically speaking. But an EPYC or Xeon is certainly not wrong if you feel it is worth the peace of mind. Not every decision has to be taken with bang or the buck mentality and overspeccing is certainly not a sin if you can afford it
Since i’ll probably be using that P40 - it’s gonna be 3.0 x8. And as far as i’m aware - that bifurcation will drop to the lowest denominator. And some older server-grade ssds also use 3.0 x8. So it’s better for me to have more real lanes, than to have less, but faster lanes.
It’s not all that about overspeccing. It’s like EPYC 7282 and 7302P cost almost the same, but the latter has double the cache and more clock speed. Or with TYAN board vs. the H12SSL. I assumed there could be more “hidden gems” out there or something.
UPD: Rechecked how the bifurcation works. Apparently each device would run at it’s maximum PCI-e gen, not the highest common.
Okay, i decided on that TYAN S8030 motherboard. if the 24-pin and other power cables won’t fit properly - i guess i’ll have to solder vertical sockets instead of horisontal ones.
Now i just wait for the reply from Aliexpress seller on what do they ship with it.
I also checked on the GPU guide for neural inference, and apparently P40 is a really bad choice due to poor FP16 perfomance (my RTX2070 is orders of magnitude faster in it). P100 would be interesting, since it has the same FP16 as my current gpu, but has double the memory (16GB) for double the price of p40.
By the way - does anyone know if this CPU will run with no cooler bolted on? I will need it to run for just a little bit, to check if all RAM sticks are detected.
