progressEdd's llms, ai, and dev corner

[progressEdd]

This is nice, need to bookmark this for when I switch from cloud coding plans

Optimal local AI on 12 GB VRAM?

After using my 55W 16gb VRAM GPU in the laptop for 7 months, I can tell you that while it is quite easy to get smaller parameter models running (2B, 4B, even decent 8B), you have to focus on larger Q5-Q8 quants as the quality really takes a hit - especially with toolcalls. As a matter of fact, with only 12GB, I’d say to not plan on any agentic mu…