progressEdd's llms, ai, and dev corner

[progressEdd]

The new deepseek models are pretty interesting. I haven’t gotten a chance to test it out yet.

I like Yannic’s coverage of the r1 math research paper.

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language...

I watched half of it and got confused by the GRPO. My takeaway is that they had a better pipeline for curating their training dataset.

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Paper Explained)

I wonder how long until the NEAT algorithm gets applied

Neuroevolution of augmenting topologies

https://neat-python.readthedocs.io/en/latest/neat_overview.html

[progressEdd]

I just finished reading the deepseek v3 paper. If the abstract isn’t detailed enough the conclusion is a much better summary without getting into the weeds of the tech.

In this paper, we introduce DeepSeek-V3, a large MoE language model with 671B total parameters and 37B activated parameters, trained on 14.8T tokens. In addition to the MLA and DeepSeekMoE architectures, it also pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. The training of DeepSeek-V3 is cost-effective due to the support of FP8 training and meticulous engineering optimizations. The post-training also makes a success in distilling the reasoning capability from the DeepSeek-R1 series of models. Comprehensive evaluations demonstrate that DeepSeek-V3 has emerged as the strongest open-source model currently available, and achieves performance comparable to leading closed-source models like GPT-4o and Claude-3.5-Sonnet. Despite its strong performance, it also maintains economical training costs. It requires only 2.788M H800 GPU hours for its full training, including pre-training, context length extension, and post-training.

While acknowledging its strong performance and cost-effectiveness, we also recognize that DeepSeek-V3 has some limitations, especially on the deployment. Firstly, to ensure efficient inference, the recommended deployment unit for DeepSeek-V3 is relatively large, which might pose a burden for small-sized teams. Secondly, although our deployment strategy for DeepSeek-V3 has achieved an end-to-end generation speed of more than two times that of DeepSeek-V2, there still remains potential for further enhancement. Fortunately, these limitations are expected to be naturally addressed with the development of more advanced hardware.

DeepSeek consistently adheres to the route of open-source models with longtermism, aiming to steadily approach the ultimate goal of AGI (Artificial General Intelligence). In the future, we plan to strategically invest in research across the following directions.

• We will consistently study and refine our model architectures, aiming to further improve both the training and inference efficiency, striving to approach efficient support for infinite context length. Additionally, we will try to break through the architectural limitations of Transformer, thereby pushing the boundaries of its modeling capabilities.
• We will continuously iterate on the quantity and quality of our training data, and explore the incorporation of additional training signal sources, aiming to drive data scaling across a more comprehensive range of dimensions.
• We will consistently explore and iterate on the deep thinking capabilities of our models, aiming to enhance their intelligence and problem-solving abilities by expanding their reasoning length and depth.
• We will explore more comprehensive and multi-dimensional model evaluation methods to prevent the tendency towards optimizing a fixed set of benchmarks during research, which may create a misleading impression of the model capabilities and affect our foundational assessment.

DeepSeek-V3 Technical Report

My takeaway is that the team worked smarter than harder for example they did the following:

• developed a data pipeline that produced higher quality data in a higher volume

  • Compared with DeepSeek-V2, we optimize the pre-training corpus by enhancing the ratio of mathematical and programming samples, while expanding multilingual coverage beyond English and Chinese. Also, our data processing pipeline is refined to minimize redundancy while maintaining corpus diversity. Inspired by Ding et al. (2024), we implement the document packing method for data integrity but do not incorporate cross-sample attention masking during training. Finally, the training corpus for DeepSeek-V3 consists of 14.8T high-quality and diverse tokens in our tokenizer.

  • We curate our instruction-tuning datasets to include 1.5M instances spanning multiple domains, with each domain employing distinct data creation methods tailored to its specific requirements.

• applied the Mixture of Experts architecture with their own spin

  • Beyond the basic architecture, we implement two additional strategies to further enhance the model capabilities. Firstly, DeepSeek-V3 pioneers an auxiliary-loss-free strategy (Wang et al., 2024a) for load balancing, with the aim of minimizing the adverse impact on model performance that arises from the effort to encourage load balancing. Secondly, DeepSeek-V3 employs a multi-token prediction training objective, which we have observed to enhance the overall performance on evaluation benchmarks.

• optimized the training process with their model architecture to fully utilize their available hardware and optimize cost, as well as, suggesting hardware level optimizations

  • Architecture: Innovative Load Balancing Strategy and Training Objective
    - On top of the efficient architecture of DeepSeek-V2, we pioneer an auxiliary-loss-free strategy for load balancing, which minimizes the performance degradation that arises from encouraging load balancing.
    - We investigate a Multi-Token Prediction (MTP) objective and prove it beneficial to model performance. It can also be used for speculative decoding for inference acceleration.
    Pre-Training: Towards Ultimate Training Efficiency
    - We design an FP8 mixed precision training framework and, for the first time, validate the feasibility and effectiveness of FP8 training on an extremely large-scale model.
    - Through the co-design of algorithms, frameworks, and hardware, we overcome the communication bottleneck in cross-node MoE training, achieving near-full computation-communication overlap. This significantly enhances our training efficiency and reduces the training costs, enabling us to further scale up the model size without additional overhead.
    - At an economical cost of only 2.664M H800 GPU hours, we complete the pre-training of DeepSeek-V3 on 14.8T tokens, producing the currently strongest open-source base model. The subsequent training stages after pre-training require only 0.1M GPU hours.

I’ll come back and add quotes/citations later.

I am going to virtual paper presentation to hear smarter people talk about it. Here’s a discord event link if you wanna also listen in

Join the Yannic Kilcher Discord Server!

[Iron_Bound]

@progressEdd

I watched half of it and got confused by the GRPO

The success is effectively applying reinforcement learning (the bitter lesson) to the problem rather than previous method of an evaluation (critic) model for scoring,

They use a more simple algorithm Group Relative Policy Optimization (GRPO) which takes a group of outputs, scores them against the average and can use this as training data, this math can train the model to do more positive outputs not the negative ones.

This covered the two algorithm differences in simple enough language:

The Math Behind DeepSeek: A Deep Dive into Group Relative Policy Optimization...

[progressEdd]

Last weekend, I decided to work on porting my fastpages blog since the project was archived.
I encountered a bug and submitted a issue

posts titles unicode characters cause exceptions when running nbdev_migrate --path

I only realized it after I submitted it in the wrong repo

`nbdev_migrate --path posts` throws exception for unicode characters

If you want to see the migrated version, you can view the blog here!

BlogEdd posts – What’s The Progress

[progressEdd]

Gemma 3 released recently. People on hacker news and local llama had mixed reviews

Google calls Gemma 3 the most powerful AI model you can run on one GPU

https://old.reddit.com/r/LocalLLaMA/comments/1je1cus/gemma3_disappointment_post/mif61vd/
someone recommended the unsloth post, so I’m downloading it right now

Tutorial: How to Run & Fine-tune Gemma 3 | Unsloth Documentation

ollama run hf.co/unsloth/gemma-3-27b-it-GGUF:Q4_K_M

I was pretty happy with gemma2’s performance, so I’m going to try it out just to see.

while browsing the unsloth post I also found the QwQ post

Tutorial: How to Run QwQ-32B effectively | Unsloth Documentation

ollama run hf.co/unsloth/QwQ-32B-GGUF:Q4_K_M

[progressEdd]

I probably should turn this into a blog post. On picking local llms

Help installing local LLMs

Heuristic for model size, parameter count, and GPU VRAM Agreed. To add a heuristic, I usually look the file size on the downloads is approximately the amount of VRAM you need. [image] Another is that for every 1 Billion parameters ~ 1GB of Vram. Though the model size is a much better indicator of if the model will fit in your gpu. There’s also a…

[SgtAwesomesauce]

Just saw your response in the other thread. Awesome work, thanks!

Somewhat related question for ya: do you have any resources at a “for dummies” level that explain the quants? I understand the bit reductions, but I don’t fully follow the other flags associated. So, for example, with qwq on ollama, it’s Q4_K_M; from my understanding, that’s a 4 bit quantization, but I don’t understand what the K and M indicate.

[progressEdd]

I usually just look it up on the internet. The LocalLLaMA reddit and HackerNews has replies with resources.

For your question, I had to look it up because no one has asked me before.
https://old.reddit.com/r/LocalLLaMA/comments/1d32q63/what_do_the_numbersletters_in_the_quantized_model/

These helped me better understand the voodoo magic of quantz

llama.cpp/examples/make-ggml.py at 5f6e0c0dff1e7a89331e6b25eca9a9fd71324069 · ggml-org/llama.cpp · GitHub

GitHub - ggml-org/llama.cpp: LLM inference in C/C++

phi3:14b and phi3:medium is actually (1e67dff39209) 14b-medium-4k-instruct-q4_0

Old quant types (some base model types require these):
- Q4_0: small, very high quality loss - legacy, prefer using Q3_K_M
- Q4_1: small, substantial quality loss - legacy, prefer using Q3_K_L
- Q5_0: medium, balanced quality - legacy, prefer using Q4_K_M
- Q5_1: medium, low quality loss - legacy, prefer using Q5_K_M
New quant types (recommended):
- Q2_K: smallest, extreme quality loss - not recommended
- Q3_K: alias for Q3_K_M
- Q3_K_S: very small, very high quality loss
- Q3_K_M: very small, very high quality loss
- Q3_K_L: small, substantial quality loss
- Q4_K: alias for Q4_K_M
- Q4_K_S: small, significant quality loss
- Q4_K_M: medium, balanced quality - recommended
- Q5_K: alias for Q5_K_M
- Q5_K_S: large, low quality loss - recommended
- Q5_K_M: large, very low quality loss - recommended
- Q6_K: very large, extremely low quality loss
- Q8_0: very large, extremely low quality loss - not recommended
- F16: extremely large, virtually no quality loss - not recommended
- F32: absolutely huge, lossless - not recommended

I now try to use Q5_K_M and larger models. General idea is greater accuracy at the cost of more Vram and storage space. From what I can tell most base models are Q4_0.

ggml-org/llama.cpp/blob/5f6e0c0dff1e7a89331e6b25eca9a9fd71324069/examples/make-ggml.py#L16C1-L37C51

The llama.cpp wiki is also good

Tensor Encoding Schemes

This guy used deepseek-r1 to answer the same question lol
https://old.reddit.com/r/ollama/comments/1ifv422/can_someone_clarify_the_subtypes_of_models/mak8at8/

Here’s a clear breakdown to help you navigate model types and quantization:

1. “Text” vs. “Instruct” Models

   Text (Base) Models:
   These are general-purpose models trained to predict the next token in a sequence (e.g., GPT-3). They’re good for text completion (e.g., writing stories, code, or emails) but may struggle with direct instructions.
   Example: llama-2-7b, mistral-7b-v0.1

   Instruct Models:
   Fine-tuned on instruction-response pairs to follow prompts accurately (e.g., “Summarize this article”). They’re better for tasks requiring structured outputs, Q&A, or step-by-step reasoning.
   Example: llama-2-7b-instruct, mistral-7b-instruct-v0.2

When to use which:

• Use text for creative generation.
• Use instruct for task-oriented prompts.

2. Quantization Explained

Quantization compresses models by reducing numerical precision (e.g., 32-bit → 4-bit). Lower precision = smaller files and faster inference but potential quality loss.
Common Quantization Types (GGUF/GGML format):

Naming Scheme: qX_Y or qX_K_Z
    X: Bits per weight (e.g., q4 = 4 bits).
    K: Indicates "k-quant" (advanced method balancing accuracy/size).
    Z: Block size (S=small, M=medium, L=large). Smaller blocks retain more precision.

Type	Bits	Use Case	Quality-Size Tradeoff
q2_K	2	Extreme compression (low RAM devices)	Lowest quality
q3_K_S	3	Lightweight tasks on weak hardware	Low quality
q4_0	4	Default balance for most users	Good quality, smaller size
q4_K_M	4	Better than q4_0 (recommended)	Balanced
q5_K_M	5	Near-fp16 quality	High quality, moderate size
q6_K	6	Minimal quality loss vs. fp16	Largest quantized size
q8_0	8	Almost identical to full precision (fp16)	No practical gain over fp16
fp16	16	Unquantized (original model)	Best quality, largest size

Key Takeaways:

Start with q4_K_M or q5_K_M for a good speed/quality balance.
Use q2_K/q3_K only if hardware constraints demand it.
Avoid q8_0/fp16 unless you have high-end GPUs.

3. How to Choose

   Hardware:
   <8GB RAM: Use q4_0 or lower.
   8–16GB RAM: q5_K_M or q6_K.
   >16GB RAM: Consider fp16 if quality is critical.
   Task:
   Simple chatbots: q4_K_M.
   Complex reasoning: q5_K_M or higher.

Quantized models are available for both text and instruct variants. Always match the quantization to your hardware and task needs!

Let me know if you need specific examples!

Edit: Credit to my buddy DeepSeek-R1

[progressEdd]

The UI could look better, but the core idea/demo looks good enough

bedHedd's AI invasion

looks interesting, I had some time during the holidays and used it as a excuse to learn marimo for the front end [image] I think you should use structured outputs as you won’t need to write regex to parse out the premise, opening paragraph, etc. I have a example here from __future__ import annotations from typing import Optional, List from py…

Local Novel AI demo

[progressEdd]

I updated the readme to include some demo videos the recording has a model selector drop down menu

progressEdd/ai-invasion/blob/master/README.md

[progressEdd]

Updates to my project template repo / vibecoding workflows

After getting to experience claude/open code, vibecoding frameworks (spec kit, openspec, gsd), and git worktrees at work, I decided to update my project templates repo.

GitHub - progressEdd/project-template at vibe-coding-gsd

This is the current config that I used to vibecode a webapp version of the marimo mockup I made a while back

GitHub - progressEdd/LAND.E at webapp-ui

Why Get Shit Done?

I picked Get Shit Done as my vibecoding framework

GitHub - gsd-build/get-shit-done: A light-weight and powerful meta-prompting,...

I found it while browsing vibecoding frameworks

Reddit - The heart of the internet

BMad if you are an Engineer. GSD if you wanna prototype or get shit done!

There’s a couple of things I like about gsd over spec-kit and openspec (which is a derivation of speckit)

GitHub - github/spec-kit: 💫 Toolkit to help you get started with Spec-Driven...

GitHub - Fission-AI/OpenSpec: Spec-driven development (SDD) for AI coding...

1. Using .planning for the folder name and better folder organization
   • It makes it easy for me to find at the top of my repo.
   • Openspec doesn’t yet support a custom name/path GitHub · Where software is built
2. There’s less review cycles/steps to create a plan
   • The chat context captures all my requirements and feedback, so I don’t have to jump back and forth between the interface and markdown files.
   • GSD has 3 steps plan, execute, verify. Spec kit has specify, plan, tasks, and implement this is great in theory, but has too much back and forth.
3. Better feedback management
   • if you have any feedback, gsd will create a new file for it. Although it makes the planning folder a bit cluttered, it’s nice because the llm context just needs to load a file rather than the full file

[progressEdd]

I wish I got to see the talk How We Scaled Kimi K2.5 [S81695] live and in person at GTC

The session was interesting because Zhilin Yang explained the concepts in a manner that engaged technical and non technical audiences. Furthermore, the presentation had a little of everything for everyone:

• Low level Neural Net Optimizations
• Training optimizations
• Agentic swarms as a concept

Some screenshots I took

The slide deck too if you want to look at it
S81695_1773825130217001pmt7.pdf (10.4 MB)

[progressEdd]

I gave a talk about my experience vibecoding to a developer network at my company. I used marp for the slides

slides.md

@progressEdd

Updates to my project template repo / vibecoding workflows

After getting to experience claude/open code, vibecoding frameworks (spec kit, openspec, gsd), and git worktrees at work, I decided to update my project templates repo.

GitHub - progressEdd/project-template at vibe-coding-gsd · GitHub
This is the current config that I used to vibecode a webapp version of the marimo mockup I made a while back

GitHub - progressEdd/LAND.E at webapp-ui · GitHub

Why Get Shit Done?

I picked Get Shit Done as my vibecoding framework
GitHub - gsd-build/get-shit-done: A light-weight and powerful meta-prompting, context engineering and spec-driven development system for Claude Code by TÂCHES. · GitHub

I found it while browsing vibecoding frameworks
laksgandikota comments on Spec Driven Development (SDD): SpecKit, Openspec, BMAD method, or NONE!

BMad if you are an Engineer. GSD if you wanna prototype or get shit done!

There’s a couple of things I like about gsd over spec-kit and openspec (which is a derivation of speckit)
GitHub - github/spec-kit: 💫 Toolkit to help you get started with Spec-Driven Development · GitHub
GitHub - Fission-AI/OpenSpec: Spec-driven development (SDD) for AI coding assistants. · GitHub

1. Using .planning for the folder name and better folder organization
   • It makes it easy for me to find at the top of my repo.
   • Openspec doesn’t yet support a custom name/path [Feature Request] Allow configurable OpenSpec directory name · Issue #581 · Fission-AI/OpenSpec · GitHub
2. There’s less review cycles/steps to create a plan
   • The chat context captures all my requirements and feedback, so I don’t have to jump back and forth between the interface and markdown files.
   • GSD has 3 steps plan, execute, verify. Spec kit has specify, plan, tasks, and implement this is great in theory, but has too much back and forth.
3. Better feedback management
   • if you have any feedback, gsd will create a new file for it. Although it makes the planning folder a bit cluttered, it’s nice because the llm context just needs to load a file rather than the full file

Based on LAND.E the project. For most of my recent developments I switched to the pi harness and GLM 5.1 with the pro coding plan (~$60-80). The performance is as good as opus 4.5.

I recently had GLM + pi + gsd add a delete story button. Using /gsd-discuss-phase I was prompt with questions, that then helped create a context file

.planning/phases/04-story-deletion/04-CONTEXT.md

The context file got used for generate a ui spec (gsd-ui-phase) and plan with /gsd-plan-phase

.planning/phases/04-story-deletion/04-UI-SPEC.md

.planning/phases/04-story-deletion/04-01-PLAN.md

There’s more in the 04-story-deletion folder

LAND.E/.planning/phases/04-story-deletion at master · progressEdd/LAND.E

If you want to see the app code the branch is here

GitHub - progressEdd/LAND.E at webapp

I use worktrees, so if you want to inspect the code + plans, clone from the master branch then add the webapp branch as a worktrees

LAND.E/02-worktrees at master · progressEdd/LAND.E

[progressEdd]

This is nice, need to bookmark this for when I switch from cloud coding plans

Optimal local AI on 12 GB VRAM?

After using my 55W 16gb VRAM GPU in the laptop for 7 months, I can tell you that while it is quite easy to get smaller parameter models running (2B, 4B, even decent 8B), you have to focus on larger Q5-Q8 quants as the quality really takes a hit - especially with toolcalls. As a matter of fact, with only 12GB, I’d say to not plan on any agentic mu…

[progressEdd]

We are getting closer a small model that matches frontier performance. GLM 5.2 Q4 on ~512GB of VRAM

GLM-5.2 - How to Run Locally | Unsloth Documentation

There has been a situation in AI

Now for hardware to get cheaper or for newer models that match this performance!

[progressEdd]

Could Open Source AI be Banned?

We should have a t-shirt don’t outlaw math

[progressEdd]

I’ve been using pi for my harness, GLM 5.2 on the GLM coding plan for model and endpoint and get-shit-done for my prompting framework. For my development environment, I’ve been using vscodium to browse/edit files with pi in the terminal

Pi Coding Agent

I got influenced to try pi out from this talk

Building pi in a World of Slop — Mario Zechner

Prior to that I was using opencode with the Alibaba cloud coding plan, but when I couldn’t resubscribe I decided to switch to GLM because I used 4.7 more on the Alibaba plan than kimi and mini maz

GLM Coding Plan — AI Coding Powered by GLM-5.2 & GLM-5-Turbo for Agents & IDEs

I went with the pro (middle) plan because it was the same price at the time of my Alibaba cloud coding plan $20. I recently helped out my friend and we found that the lite tier was good enough and didn’t hit the quota.

For the prompting framework, I found gsd to follow my preferences. I want to discuss what I want to build and get some suggestions for how it can be accomplished

Pi Coding Agent

GitHub - open-gsd/gsd-core: Git. Ship. Done - Core

I used my setup recently to implement story deletion + llm endpoint connection management

progressEdd's llms, ai, and dev corner

I gave a talk about my experience vibecoding to a developer network at my company. I used marp for the slides Based on LAND.E the project. For most of my recent developments I switched to the pi harness and GLM 5.1 with the pro coding plan (~$60-80). The performance is as good as opus 4.5. I recently had GLM + pi + gsd add a delete story button…

[progressEdd]

Update, GLM altered the deal, if you sign up now, the coding plan isn’t as good anymore especially If you are a heavy consumer of tokens. I am unhappy by this decision and do not plan on upgrading. I will continue to hold my Legacy V2 plan because they more open in publishing their weights and training techniques than Anthropic.

At this point, I’m saving up and waiting until hardware is cheap enough to run Deepseek V4 Flash locally like sentdex

You Can Just Download More Tokens/Sec — Terminal-Bench

Also until hardware gets cheaper, I think I’ll try to build out the datasets based on my pi sessions for my own data flywheel like wendell

Stop Wasting Cloud Tokens - Local AI FTW

Your company is probably spending too much on AI: Switchyard, local models, routing, and the data flywheel [You Don't need to use Cloud AI! Switchyard and Nemotron 3.5 Lightning] So I have been working on an enterprise consulting project where the plan is “adopt AI” but my go-to IT toolbox is monitoring and management of “how” and “wha…

If my plan does expire, I will probably follow the guidance from the opencode reddit which recommends open code go + minimax

ollama cloud vs opencode go - r/opencodeCLI

I ended the OC GO limits in 2 full 5 hours sessions of GLM5.1+Qwen3.6Plus usage to implement one complex task. Speed was pretty good. If you go mainly for Qwen3.6 Plus and MiniMax M2.7, you’d have much more usage. Use the GLM5.1 as planner and reviewer only. Consider buy MiniMax Coding Plan + OC Go and use M2.7 as a workhorse, then you’d never hit the limits and M2.7 is fast even on 10$ plan.