havent asked it about tianenmen square or the uyghurs yet though 
honestly my company pays for deepseek and glm 5-3 flash models through API and this feels right up there, even the token rate is similar
Hate to say it, but the speed and performance difference between llamaserver and vLLM is kind of nuts, its way more intelligent with prompt caching etc, i’ve barely had to wait a second even on large prompts
Ok you know what? stretch goal:
Lets see if i can make an mxfp4 quant from the orcarouter version and mash it into the clavviger engine: https://www.youtube.com/watch?v=hFDcoX7s6rE
OK well its doing ….something, i have a feeling this will take a while
[AA] first layer took 9m55s; estimating 7h56m for 48 layers → finishes ~00:38:46
[AA] layer 0: 1539 modules (1516 with stats), 2 stage(s), capture 100.6s recapture 85.8s quant 234.8s fwd#2 74.9s io-wait 0.0s, E-moved 0, micro-batch 4
[AA] PLE table: 128 mmap’d shards x 2500012 rows (torch.bfloat16), scale no, gathered on CPU
See you gents tommorow
Stretchgoal accomplished XD, requanting isnt as hard as i though performance is near identical but now i can ask it about the uyghurs
I’m going to make a version of the qwen 3.8 27B next on mxfp4 using this engine too, i feel like i can finally use these cards to their true potential all of a sudden XD
If you install the ROCm Validation Suite, you can use the TransferBench tool to test the p2p speed GPU to GPU and GPU to CPU.
I didn’t have any issues getting p2p to work with Proxmox, but the speed was a lot lower than when running the same tests bare-metal.
Ah i know transferbench, thats how i know its still root-complex routing rather than the PCIE switch, now that VLLM and MXFP4 are sorted its my next target to fix 
1 Like
Chatterbox+WhisperLarge+qwen 3.8 flash next configured at C8 with an 850k shared context pool, I’m maxed to the hilt, Atleast i’m not leaving anything idle
AM4 ‘s bottlenecks really don’t bite as hard as i had initially thought if you plan around them properly
The AI Pro branding isn’t all bullshit it seems