progressEdd's llms, ai, and dev corner

[progressEdd]

I probably should turn this into a blog post. On picking local llms

Help installing local LLMs

Heuristic for model size, parameter count, and GPU VRAM Agreed. To add a heuristic, I usually look the file size on the downloads is approximately the amount of VRAM you need. [image] Another is that for every 1 Billion parameters ~ 1GB of Vram. Though the model size is a much better indicator of if the model will fit in your gpu. There’s also a…