Training requires orders of magnitude more hardware and time than quantizing or distilling something though, not to mention you need a really strong dataset to do the training. But everybody has huge frontier reinforcement learning traces and logit captures sitting in their basements 50tb flash storage arrays right?
Tuning an MoE model sized to run inference on a 6k (lets say 65B+ neighborhood) requires ~a couple B200 nodes to run the fine tune for days, maybe ~1 epoch per day? and moe is really tricky to get right so assume 3 months in the training loop to get close.
Let me know if u want to borrow my data curation pipeline though, human hand labeling and scoring took me months of listening to audio books while clicking my touchpad.
Edit: Here, I have a simple 1 dimension gradient descent simulator posted. I was working on to help explain the concept of tuning hyper parameters to someone:
Now multiply the complexity by 5000 dimensions to spot the challenges 