2/184 to 170/184: one epoch on two R9700s took a 14B coder from useless to 5 points behind gpt-5.6-sol

High-Performance Computing Machine Learning, LLMs, & AI
, ,
1

Been a while. Last time I posted the dataset had passed 300k
entries and I had one 14B fine tune with 81% structural
correctness. Since then I did the thing I should have done from
the start, I built a proper benchmark and published it so people
can check my numbers instead of taking my word for it.

What got published:
184 test prompts for MQL5 (the language for algo trading bots),
the scoring rules, per item results for every model, and a
verifier script that checks the release hashes and recomputes the
headline numbers from the rows. A pass is zero compiler errors
and a built binary. Thats it. Compiling is a low bar and the card
says so, it does not mean the bot trades well.

The numbers:
Base Qwen2.5-Coder-14B-Instruct, untouched: 2 of 184 compiled.
Same model after one epoch on my data: 170 of 184.
gpt-5.6-sol on the same 184 prompts: 179 of 184.

So the local 14B went from useless to about 5 points behind a
frontier API model on my niche. One epoch. I am not going to
pretend thats the finish line but it is the first number I have
that somebody else can reproduce, and that matters more to me
than the number itself.

Hardware, since this is L1T:
Everything is still AMD and still local, but the layout changed
since April. The main rig is now a 7950X3D with 64GB and one
Radeon AI PRO R9700 in it, the 9800X3D and the 9070XT are out.
That box is where both local arms of the benchmark were run, base
model and tuned model, same card, same settings. The
Threadripper 2970WX has two R9700s now and does the training
under ROCm. So three R9700s total across two machines, 96GB of
VRAM, no CUDA anywhere. I fought ROCm plenty (some of you saw the
Windows thread) but it does the job and the numbers came out of
it.

Links:
Hugging Face:

GitHub mirror:

What is not in the release is the training data and the tuned
weights. What is in the release is every prompt and every result
row, so you can run your own model against the same 184 and
compare.

Caveat thats also on the card: the test prompts come from the
same generator family as the training data, so this measures how
good the model got at my kind of spec, not how it does on random
human written requests. I have a bigger private holdout in the
works and a version 1.1 of the card coming with those results.

Still grinding. If you are on AMD and want to compare notes on
ROCm training or eval, ask away.

MQL5 is a trademark of its owner. Not affiliated, independent
project.

1 Like
2

I have questions!

First what framework are you training with ROCm?

Are you using open license datasets?

What’s your backend? Training vocab? Do you preset LoRa for after base teraining or skip finetuning altogether?

Actively training a ~1B LLM in an experimental framework on a 16GB RVII no less. I’m intterested in the training you use and where, if you have the weights avail?

EDIT: I see you are using Base Qwen2.5-Coder-14B-Instruct

1 Like
3

Yep, Qwen2.5-Coder-14B-Instruct is the base I used for the published eval.

I’m training on AMD through ROCm on Linux. The 14B is really just an evaluation tool for me though, not the end product. I’m building the MQL5 dataset first and using models to measure whether each version actually transfers more capability.

The training corpus itself is synthetic and privately held. The public Hugging Face repo is the benchmark/evidence side: prompts, scoring, results, etc. I’m not publishing the training corpus or model weights right now.

The published run was QLoRA SFT for one epoch on the existing Qwen tokenizer/vocab. No continued pretraining and no tokenizer changes.

Hardware is 2× Radeon AI PRO R9700 on the Threadripper box for training, ROCm/Linux. I’ve got a third R9700 elsewhere in the lab for generation/eval work.

I’m happy to share the exact training config/framework too. I’ve been documenting this pretty heavily because I’m trying to learn how far a properly built domain dataset can push different base models.

Benchmark/evidence:
https://huggingface.co/datasets/CompilingThings/compile-benchmark

Your 1B-on-Radeon-VII experiment sounds interesting too. What framework are you using on that card?

4

Claude and I handrolled a Rust based GPT style RoPE SATI framework design from the ground up to work over a mesh, on consumer cards(GPU’s typically 16GB and under)

It’s unlike any other framework, and completely experimental, but designed on RoCm 6.4 and now udated to 7.1. We aren’t to advanced methods like attention and quants or recursive efforts. full front back pass training past 100k steps. 150M->500M->1B step up models with inference response gguf exports.

It’s a mismash, the training side, then the dataset harvest side, with lots of stages and sync steps. Very pre-alpha framework. It does work though.

I’m mostly interested is 1. Public domain dataset generation 2. ROCm experience, hiccups, blas/kernel mixups in training models, and 3. Do you want to test my framework once it’s share-able?

With training 14B, do you need 64GB VRAM to fit the weights?

Is quant training destroying other experts?

1 Like
5

Yep, I’d definitely be interested in testing it once you’re comfortable sharing it. A Rust/ROCm training stack built specifically around consumer cards is right up my alley.

My current dataset factory is synthetic and privately held. The public part is the eval/benchmark side. I’m also very interested in public-domain dataset generation, especially where the outputs can be verified mechanically instead of just judged by another model.

ROCm has been a project in itself. I’m on ROCm 7.1 now. Most of my pain has been around getting the whole stack stable across multi-GPU training/inference, kernels, container/runtime versions, and making sure a performance problem is actually compute and not the software around it.

For the 14B QLoRA run I used 2×32GB R9700s. I wouldn’t say 64GB is strictly required though. QLoRA keeps the base quantized and trains adapters, so the VRAM requirement is much lower than full fine-tuning. Context length, batch size, checkpointing, etc. change the answer a lot. I used the two cards because that’s the hardware I had and it gave me headroom.

And no, QLoRA isn’t “destroying other experts” in my case. Qwen2.5-Coder-14B is dense, not MoE, and the base weights stay frozen during QLoRA. The adapter can absolutely bias the model toward the domain, which is intentional here, but the original base is still there underneath and the adapter can be removed.

My actual goal is pretty narrow: spec in, working MQL5 out. The 14B is mostly a test for measuring whether the dataset is getting better.

1 Like