Glm 5.2 reap

There is someone, 0xSero, who is championing open source LLMs, and right now is working on using Cerebras REAP frame work to prune experts from MOE models to reduce the memory footprint with minimal impact to performance. The experts that get pruned depends on how you set up the prune.

0xSero has pruned DeepSeek v4 flash for coding in the past, and I was going to do the same. However I ran into his Huggingface last night and saw he has also pruned GLM 5.2 for coding!

The claims are that GLM 5.2 are Opus 4.8 level intelligence, but I have never run it because it is enormous, and won’t fit on my system. However, now that it has been pruned and quantized all the way down to 4, 3, and 2 bit, it is possible.

Now, it’s still big, the 4 bit (both GGUF and NVFP4) is still large, over 300 gb of VRAM, but I can now fit it on my hardware and I am going to try it out this weekend. Frankly, this is amazing.

Anyway I just wanted to come here and share that for anyone else who is running locally. If you don’t have the hardware to run it, I would still be excited because what I think this is starting to show is that we are going to get some crazy good local options in the future for much less VRAM. The future is looking very interesting.

By the way, REAP is on GitHub and you can prune models yourself, as long as they are MOE, so you could take something that doesn’t fit on your hardware today and make it work for your use case.

7 Likes

This is legit bananas, glm 5.2 on a 4x rtx 6000 machine

7 Likes

Interesting, what is your experience with reap models in regards to coding. I did hear rather disappointing feedback about reap models before. They were described as lobotomized, and producing bad output.

Though those were general usage models and the reap was done to reduce size as far as I understand, but I never heard using that to make a coding only model.

Indeed, I rather run a smaller model than a larger one that is indeed lobotomised.

I am gonna be honest that it is only what I heard but it does make logical sense to me. On the other hand if all the coding related experts are still in there it would make sense that it still performs on coding tasks. My question is more how accurate this process is and if the results are what you would hope.

REAP50 and the model is pretty much useless, it can’t hold a chain of thought. If you do 5-15% you may get away with it.

1 Like

Yeah my experience with REAP models has always been lack-luster. Deliberately removing experts only hurts model capability. There’s a chance that the removed experts aren’t as pertinent to your own workloads, and you might not see it as much, but I guarantee there’s latent activations that cross expert boundaries. And it’s not like quantization either, where the info is there but with less fidelity. REAP legitimately just removes the information altogether.

My experience is limited, but there are two things I believe are true.

  1. You must activate the correct experts during pruning or it won’t fit your use case, and you can drop some percentage of experts off the model with almost no loss.
  2. There are other techniques that drop the model size that are not as performant.

There was someone who did lobotomize a model with REAP but it comes down to how you do it and how much you prune.

I’ll report back once I get GLM running on my server. According to 0xSero’s posts, it held it’s coding and reasoning intelligence.

2 Likes

Wild

I wonder how long before this can fit in a 24gb vram gpu

How does the reap version compare to Unsloth? Looks like reap version is more optimized for NVFP4 whereas Unsloth is for macs and other gpus
I saw sentdex cover this recently

I have the GLM coding plan, so this is getting closer to reducing my subscriptions

Running tests now. I haven’t gotten MTP working yet but so far so good.. Only 35 ish tokens/sec so far though..

2 Likes

2 Likes

Ok with MTP on I am getting the following. I think this is as good as it gets.

2 Likes

What the actual… GitHub - kacper-daftcode/vLLM-Moet: Run DeepSeek‑V4‑Flash (159B sparse‑MoE) on a single 96 GB GPU. A vLLM patch + hand‑written SM120 SASS kernels: 2‑bit MoE experts + an FP4 "delta" cache that recovers precision — matching the official FP4 checkpoint's quality on one RTX PRO 6000 · GitHub

I wish this were just a touch faster with the full context

1 Like

@wendell Things are accelerating so fast.

That GitHub just posted above your comment shows GLM 5.2 running unpruned off the Nvidia NVFP4 checkpoint on 4x RTX 6000 pro without compression. Edit: correction: no quantizing.

It also shows Deepseek V4 running on a single 5090!

There have been 2 major breakthroughs since that one from 0xSero and it was only 4 days ago!

surprisingly deepseek v4 flash was the fastest for him while still near the performance of a compressed glm 5.2 model

video just covers the article + his rambling about how large models seem to degrade when trying to compress them.

Also relevant for thread, he says that reap lobotomizes the model for tasks other than coding

3 Likes

Perhaps tangentially related, just came across this:

Looks like it’s not reap. Runs on 6x 5090’s.


All of this is much too heavy for my hardware so I wont bother, but it’s interesting and exciting to see people trying to get frontier class running at home.