Hello there community. First time poster. Long time enthusiast. Occasional lurker. Engineer (mechanical). Over the past many months and about 3 months too late, I began Frankenstein-ing a machine to journey down the LLM rabbit hole that resides on a rocket ship. I’m no expert in regards to software, far from it, I may word things or explain things slightly off as a result.
Frankenstein is real:
Background (if you care):
Ever have that problem when you had way more VRAM than RAM when you actually need RAM? Probably not, but I did. And I did it for cheap too!
I had a budget. A very, small, budget. I don’t need to elaborate. We all know, hence the late part.
Its been a journey and continues to be - but in case anyone would be interested who is also on a real budget – I have found an interesting way. Its still a WIP. Every day. On and off I have a stable build of chaos; the off part is only when I go for MORE, which is often, like building custom Llama, vLLM, and or ROCM drivers. Yay stupid ideas! (I have made significant progress on the ROCm stuff for the mi25’s!)
Anyway, I think this community would genuinely appreciate this feat - albeit , mostly for its ‘wtf were you thinking’ or ‘how in the hell…’, or ‘im actually impressed’ sort of reactions.
Setup (if you care):
B650M (stress on the M) + 7700K + 32GB DDR5 6K CL30 (ahh yes the good stuff, bought years ago) + 9070XT + 9100PRO, 980PRO, 950PRO yada yada…That was the base. But then the LLM bug hit and when my wallet was its thinnest and was laid off. So, I did what I do. Make it better. Unfortunately, Broadcom AND AliExpress had already joined in on the overcharging game and Lane expansion was out.
The trial was then readjusted after a LOT of research (because this level of hardware things is new to me, WTF is an MCIO? kind of place), and going to be 1x Mi25 → WX9100 conversion, bifurcation x8/x8 risersTLDR - 9070XT x8, 2x MI25 converted to WX9100 PCIE 3.0x4, (THIS IS NOT MINE NOR DOES IT DO HEAVY LIFTING, it will be someone else when I build their system) Intel B70 x4 via chipset, 9100PRO 5.0x4 and 980PRO 4.0x4 via chipset.
It took a long time. Especially the 37 versions of 3d printed adapters it took to convert an older case to house up to 4 cards Via dual 4.0x16@8 MCIOs → dual, dual pcie4.0x16@4 risers + one pcie 5.0x16@8 (optionally plugged in between one dual riser or itself, which its hosting the 9070XT)
Where Strata comes in (I assume you care):
Then I saw Strata come out, but I only had 32GB RAM. However, but I have plenty of VRAM, not a common problem, i get it, BUT considering the cost of RAM, you can literally buy several MI25’s for less than another 32GB hence this post. I’d imagine most of us have at least one primary decent GPU which is an important piece to this.
I installed. I got about 2-3 Tok/s. Hey it worked.
Then I started understanding how it worked, what it does, and what I can do so it doesn’t suck.
The numbers (what you probably care more about):
I have some more tricks up my sleeve but I’m hitting average with a 205K context window, 32K prompt tests. There are two test groups in most cases each with 3 different prompts. NOTE MANY OF THESE WERE RUN WITH DIAGNOSTICS WHICH COSTS TOK/s, estimate 3-5% increase
This group uses <=4096K context windows and low token count first prompts, you can see the progress:
| Configuration | Decode tok/s | Prefill SECONDS |
|---|---|---|
| Earliest staged GPU runs; short prompts | 3.59–3.62 | 9.16–9.57 |
| Subsequent native/adaptive staging; short prompts | 6.40–6.47 | 8.11–8.31 |
| Persistent P2P; short prompts | 4.75 / 8.49 / 11.65 / 11.51 | 4.72 / 3.22 / 2.23 / 2.27 |
| P2P storage, 1K-token prompts | 8.89 / 11.98 / 11.81 / 11.45 | 55.24 / 43.37 / 42.99 / 43.82 |
| 4K storage topology; later failed memory audit | 9.18 / 12.06 / 11.87 / 11.75 | 40.46 / 41.88 / 41.30 / 45.21 |
| Early active-WX workers, eight prompts | 57.69–65.35 | 40.76–46.61 |
| October 10, prefill chunk 1,024 | 64.16 / 65.47 / 64.03 / 63.63 | 41.56 / 40.73 / 40.78 / 41.61 |
| Same suite, prefill chunk 2,048 | 64.53 / 61.94 / 63.16 / 64.79 | 24.51 / 24.47 / 25.36 / 23.20 |
| R15 completed group; suite failed later | 62.35 / 61.75 / 64.58 | 24.59 / 26.67 / 24.35 |
| R15R1 completed group; suite failed later | 44.21 / 64.39 / 67.73 | 25.82 / 22.79 / 22.80 |
| R16R2 discrete-GPU control; suite failed later | 59.70 / 58.11 / 60.29 | 17.01 / 16.55 / 16.45 |
| R17, profiling on | 60.49 / 57.75 / 60.59 | 16.96 / 16.54 / 16.48 |
| R17, profiling off | 68.33 / 66.72 / 69.10 | 17.26 / 16.56 / 16.47 |
| R18 adaptive, group 1 | 40.62 / 39.08 / 41.22 | 23.94 / 23.24 / 23.23 |
| R18 adaptive, group 2 | 40.54 / 38.70 / 41.19 | 23.85 / 23.12 / 23.13 |
| R19R1 fixed; duplicate group failed later | 73.45 / 71.47 / 74.82 | 24.03 / 23.16 / 23.11 |
| R19R2 fixed | 74.19 / 71.76 / 74.55 | 23.86 / 23.16 / 23.10 |
| R19R2 duplicate cache | 73.08 / 70.98 / 73.94 | 23.79 / 23.24 / 23.11 |
THE FOLLOWING USES 205K CONTEXT WINDOWS AND 32K PROMPTS, AGAIN YOU CAN SEE THE PROGRESS AND NEXT TO NO SLOWDOWNS:
| Prompt | Original prefill | Summed prefill | Original decode | Summed decode |
|---|---|---|---|---|
| 1 | 80.81 s / 405 tok/s | 79.36 s / 413 tok/s | 70.39 tok/s | 69.25 tok/s |
| 2 | 81.05 s / 404 tok/s | 78.33 s / 418 tok/s | 69.88 tok/s | 68.62 tok/s |
| 3 | 80.20 s / 409 tok/s | 78.63 s / 417 tok/s | 69.68 tok/s | 67.97 tok/s |
| Configuration | Prefill SECONDS | Decode tok/s |
|---|---|---|
| R26 idle-balanced | 86.65 / 58.68 / 57.95 | 70.26 / 68.12 / 69.46 |
| R28 idle-balanced | 84.49 / 52.23 / 51.65 | 69.68 / 67.95 / 69.69 |
| R30 idle-balanced | 83.89 / 52.38 / 51.71 | 70.60 / 69.86 / 67.58 |
| R31 Q8 + idle | 80.58 / 49.08 / 48.26 | 69.94 / 69.56 / 67.67 |
| R32 fixed + summation | 79.36 / 78.33 / 78.63 | 69.25 / 68.62 / 67.97 |
| R33 learned startup + idle | 48.96 / 47.62 / 47.27 | 65.62 / 68.19 / 68.79 |
Yes many of these are in seconds, but keep in mind 32K/time….hitting over 700 tok/s prefill. The newest numbers are NOT included. I think I can press 80tok/s decode and 750+ prefill
What’s doing what:
I know what some of you may be thinking…a B70 and a 9070XT…big deal. But no. The only thing the B70 does in this setup is contain a ledger and map in its memory. Its effectively repurposed as really fast RAM and at a PCIE 4x16@4, it’s possibly a bottleneck. Could i repurpose it to do expert execution? Yes, but that is not the point of this, nor would I know the cost.
The 9070XT is the primary. It handles the routing and coordination while also handling the toughest jobs.
BOTH MI25’s ARE RUNNING AND ACTIVE. 100%.
RAM is temporary fast access expert offload if needed.
There are MANY optimizations, now whether I invented them versus strata, I don’t know, I just piggy-backed Strata because I was waiting for someone to actually do it.
The custom optimizations, some of them anyway:
- First and foremost, im running Debian, so I had to compile custom kernels for P2P transactions to work
- Smart Expert Placement: the 9070XT is the bread winner, and we need to take advantage.
- Idle load balance: Fact is, when we’re actually building shit, we get the response and go do something. This uses that to your advantage - in between every prompt
- First startup load balance: At each startup, the highest used and most process intensive experts go to the Primary card, in this case, the 9070XT, the remaining experts are split between the two slow cards and balanced by both usage and expected loads of the expert.
- A complete expert ledger is maintained:
- W/Diagnostics: the ledge counts how many times experts are used as well as processing demand - you only need to run this so many times before it learns what’s what
- W/O Diagnostics: The expert usage counts are logged each transaction. It maintains the previous 30 sessions as we typically use the same experts over many sessions.
- A complete expert ledger is maintained:
- Active expert exchanges:
- Dynamic: This utilizes the ledger maintenance functions but over a shorter duration, is particular experts are causing slow downs on the slower cards or particular experts are being used more frequently than expected based on the ledger, that expert is then loaded onto the faster card via either RAM OR SSD depending on if its already in RAM or not, and will kick off its unused experts into RAM to be able to fast swap back if/where needed. If that expert in RAM is needed, it routes to either the 9070XT to replace another unused or low use Expert OR gets sent to the slower cards, replacing the expert that the faster card already loaded in.
- Fixed: All experts have definitive homes on each card. This is used in conjunction with the ledger tracking to determine which cards get which expert. If for some reason not all experts fit on the GPUs, the GPUs will fetch that expert from SSD and send the expert its replacing to RAM and will be available to go back to the same location if its called, where that expert then drops. I don’t remember if i designed that to go back to RAM or just replaced I don’t remember
- Summation Vectors - this is a big one for network optimization/transactions. PCIE 3.0x4 remember. Sometime the same tokens are used for multiple experts. Instead of sending the same token three times, it sends on vectorized summation that the primary GPU will decode and feed to the relevant experts.
- Quantized Transfers - this is also a major time saver. The MI25’s don’t like heavy quantization so for awhile I was just working with FP32. When I realized that, I said that’s dumb, and quantized each transfer between GPUs - this cut traffic substantially and outweighs the costs of encode and decode
- Traffic management - this is still a WIP but generally i’ve tried to avoid large chunks and keep everything managed but I think I have some room to improve here
These are just some things and I’ll add more as I think of them. I DO NOT KNOW IF THESE THINGS EXIST ALREADY. I’m just doing things as I understand the picture better and what’s happening and giving back to maybe someone else who can use it.
Why SHOULD YOU CARE (probably): if you’re on a budget, you should.
- MI25’s can be had for less than $70 if you know where to look. You can get riser kits for less than $100. You can expand your system with 3x GPU’s (assuming you have bifurcation available) for around $300. Plus some cooling and power. Depending on your primary card, thats an extra 50GB of HBM loaded VRAM to play with…USEFULLY. And yes…I think i have some room to grow performance wise. I see decodes hitting over 100 tok/s and prefills i’ve made climb by over 1000% in just the past few days, though it took MANY more days to get there.
Currently a big hold back i suspect is driver support. I’m been on a multi week journey building custom The ROCK (10+) drivers, for ALL FAMILIES OF PRODUCTS FOR THE FULL STACK for the 9070XT, the iGPU, and both MI25’s/WX9100s. I’m making progress. Good progress. I’m hopeful this will unlock decode in the MI25’s which is a hurdle. Will it work? I have no fuggin’ clue. I’m not that smart in the arena.
IF ANYONE IS INTERESTED: when I get this thing tuned up, I can put it on GITHUB. I’ll do my best to clean up the slop, and make it installable for whatever AMD card but that’s WELL OUTSIDE OF MY WHEELHOUSE. THERE ARE PLENTY OF PEOPLE ON HERE MUCH SMARTER WITH THIS STUFF THAN ME…maybe you want to help?
Please note: i’ve been doing ALL of this with a $20 GPT PLUS plan. I CANNOT STRESS HOW DIFFICULT GETTING TO EVEN HERE HAS BEEN with that model. Especially the level of complex other tasks I have been doing. I cannot stress the amount of frustration I have endured. I’m sure some of you understand. All I ask is, deal with the slop and if you think anything in here is useful for others and have the skills to help, let me know.
PS - any employers out there need an accomplished director level preferred, engineer in manufacturing or business dev or product dev? …Beuler??? 
Only half joking. Asking for a friend
