+1 for ollama
I’ll raise you a lmstudio. It is great for finding out which models are compatible for your hardware
For inferencing (using a pretrained model), the unified memory is great. I don’t know if model training is better as pytorch doesn’t have a native version for apple silicon. IIRC Wendell talked about it on a link with friends show I can’t remember. The macbook is great if you need to be portable and offline, but much more expensive to scale out the memory
I have a m2 pro with 32gb of shared ram and a desktop with a 8gb RTX 2070, Gemma 2 9b q8 runs very well for following instructions and doing text classification.
If you are confused by the naming, the structure of the name for gemma-2-9b-instruct-q_8_0 is as follows:
gemma-2: model name9b: number of parameters for model settings,B=billions of parametersinstruct: the task the model was fine tuned on, instructions following instructions vs code generation modelq_8_0: Quantization level, the datatype of the model weights and how compressed the model weights are.
A quick heuristic I use is for every 1B of parameters, it’s about 1 GB of ram/vram. Each drop in quantization level is a lower quality output q8 > q4 > q0. I try to find a model with the highest quantization possible with the the highest parameters my hardware can handle
For coding, you might want to try out qwen, deepseek or any coding models in the lm studio discord
