That entirely depends on what you mean by “supplying GPUs with data”. if you have all feature engineering done (i.e. you just load a dataset from disk) you just need a nvm-e ssd and any kind of semi-modern CPU. If you need to do more feature engineering, massaging data, prepping data on the CPU you need something beefier.
I’d say experiment on the hardware you have/can easily afford and take it from there. You can make DL models as large/small as you want, just by choosing the number of parameters. Take the LLama models for instance which come in versions from 7B to 130B. It does not make sense to scale to the largest models unless you are making money. If you want to learn, experiment with the smaller models; build some experience; get a job or a gig where your employer provides hardware or cloud expenses.
Yes, that is exactly what I mean. Every little bit more advanced project requires large amount of data, and with large amount of data, there is a lot of cleaning etc. All stuff done on processor. I did a small project earlier and my pre processing took literarly days to complete. I want to be able to parallelize that across the multiple threads so to make my work faster.
I am curious to see a use case where you are NOT IO limited, and you manage to starve an RTX 3090 by using a 7800x. Seriously, eighter you have an IO problem or a Software problem.
Yes PCi-E 3.0 is fine, the SSD is the only issue that might become a topic, but in the end, often the latency is a problem on the IO side, not the throughput. You load single files from the disc.
In the end, get the CPU that is the best value; it doesn’t matter as much as we think it does. How much data are you trying to process? And is the encoding correct, or are you artificially inflating the whole thing before loading it to the GPU?
I am not concerned about that, I am mostly concerned about the questions which I asked above. I can get ‘tr 3955 pro + asus mobo’ or ‘epyc milan 7313 + supermicro mobo’ for 1.5k atm but I am wondering, is it too much for start. I definitely feel that I need more slots for gpu so the consumer hardware is out of question. Used epyc rome+supermicro mobo(7302) is like 400€ cheaper atleast which I could use for extra ram and storage. There is also option with intel xeon but with pcie3 and asus mobo, same at 1.5k. All of them with 16/32 c/t. So the older gen should not pose some real issues?
The CPU is $1200
The Motherboard is $800
32GB sticks of ram (parted out from 25 stick kits) are around $80 each.
You need a boot drive. You may have parts or you may need to buy something. Don’t make it USB, sata is fine. I got a pair of optane, and recently a larger data store SSD after using spinning rust for a few weeks. I was able to see that yes, it is the bottle neck.
Get a good Power supply that can power your CPU, and motherboard. The motherboard requires 2 8 pin connections in addition to atx+4.
If you need more wattage than your power supply can accommodate, you can use a power supply combiner cable. It uses a few pins on the atx cable to establish ground and one pin to determine if the computer is on or off. After that add a power supply every other GPU.
1Terabyte at 1 gigabit is 8000 seconds, or a bit over 2 hours. at 10 gigabit it is 800 seconds, or under 15 minutes. Many models are 1 TB. How patient are you.
the NT model with dual 10GB ethernet does not take pcie lanes away from anything on the N model. 108 pcie lanes are used on the NT model, 104 are used on the N motherboard model. But be aware that there are only drivers for that 10G chipset available for linux and windows server. There is no windows 10/11 drivers.
You can get x8 pcie riser cards, or just get cables from the oculink ports to pcie slots, like:
I know that there are others that don’t require an ATX cable. I used a similar one for about 10 years that brought the pcie m.2 out of an intel NUC so that I could use an nvidia GPU. The genoa supermicro epyc motherboard has 3 pcie5 8x oculink ports.
In total you could theoretically connect 8 GPUs at 8x or faster.
I just got an SSD because the hdd was taking 6 minutes of disk io per 1 minute of gpu compute just to load the models for stable diffusion. I got:
$369 6.4TB intel D7-P5600. This is the pcie4x4 high write endurance model for 5 years at 3dwpd. And it was equivalent price of a samsung 980 for the amount of capacity.
I also got the u.2 cable to oculink on the motherboard. This drive should load the model in 2 seconds instead of 6 minutes.
BTW the bandwidth to main memory on the Genoa is 460GB/sec. On the previous models it was 80GB/sec max.
Hello. Thank you for your reply. However, I already made a decision to go with milan build.
CPU) epyc 7313 800€
MOBO) H12ssl-i-o 600€
RAM) 2*32 non ecc 3200Mhz 160€
PSU) Be quiet pure power 12m 1000W 150€
CASE) Thermaltake view 51 200€
CPU_FAN) Noctua nh-d15 90€
I aim toward 7373x from milan-x in future or similar.
Also, I am not quite sure how to set 2 psu in my case, but will have to come up with some idea soon. The thing is that I do not believe the cpu is that important at this point, in other words, 5% on gpu performance improvement outweights 15% on cpu, atleast for my use case. Do you see any possible bottlenecks over there?
Well, I also considered water cooling since my main idea is to try get those clock speeds as high as they can go and haven’t found any viable water cooling that fits sp3, but this with Noctua seems to be a second best option.
okay, well thanks for notice. I will buy then 8x16gb I guess. Can you suggest me some air coolers for my desktop case that would do well on cooling those server components, or is the water cooling the best option over there?