GPU for LLM

[gc71]

I had posted in this thread some thoughts and experiences but I dont think they are at quite the scale you are looking at

Epyc Build

This is my first time being able to actually build a workstation class pc, I am not sure how to figure out what workstation gpu is best bang for buck, im not really trying to play games on this pc it will be specifically for rendering, and I am interested in Ai/deep learning tasks, but that would be more of a side case, it will be mainly for 3d ren…

If you are putting these GPU’s into servers, then you really need to make sure you are getting GPU’s with blower style fans, not “gamer cards”. This is really important especially if its gonna be running for a long time.

You are generally better off getting the most recent model cards possible with the largest VRAM per-card. AFAICT, fewer beefier cards with larger VRAM are better than more numerous smaller cards with less VRAM.

@hotfur

Tim Dettmers recommended used 3090/3090Ti for transformer task and I think that one is also very good. However, they will cost me ~700$ each. With this money, I could theoretically stack up 3x 3060 and obtain more aggregated speed/memory as each 3090 is only twice as fast and memory as a 3060. I would not mind connecting more servers to the Infiniband switch.

You do not want RTX 3090 Ti, only the 3090 (blowers if possible, as I described here).

I seriously doubt that you are going to be able to do anything with RTX 3060 12GB besides just running small LLM models, or running Stable Diffusion, etc… I would not expect them to be useful for training.

Honestly I think you need to more seriously consider cloud for this. I do not think its gonna be economical to go bare-metal for this purpose.