ok so i am trying to increase performance of llms i have running on my 7900 xtx via hipfire, i notice i get significantly higher total tokens per second when i run mutiple slots at once when running qwen 3.8 27b.
i am running into 2 road blocks tho, 1 context each slot gets a progressively smaller context as increase the number of slots. is there a good way to do more with the same given total amount of vram, i assume there is going to be a large amount of overlap for stuff like system prompts, tho i use a relatively minimal system prompt that links to files so if a model only actually reads system prompts realvent to itâs own task which makes it harder to just say this text gets put into a shared context, i also image if i have them working on similar tasks there will be a large amount of overlap.
2 alot of tasks arenât easy to split up into mutiple smaller tasks that can be done in parrel without proper knowledge of previous and future steps, maybe something like a shared context or a messaging board would work well?
also a more experimental idea to further speed up general prompts is to instead have all my prompts processed by a âorchestrationâ slot which splits up a given prompt into X number of tasks then gives these to new prompts to different slots and then the âorchestrationâ slot dies to free up more vram for context, i could also route something like this threw a decision model like kev to determine if a given task would work well being split up.
Can you clarify what you mean by slots? Most of us are probably interpreting slots to mean physical pcie slots, but it sounds like you mean something else.
each slot is kind of a instance of a âchatâ running off the same weights, so i can have qwen 27b loaded on 1 card and 2 âchatsâ running at the same time off those same weights, they can either have 1 shared context or each within own. i saw roughly a a 30% increase in tokens per second when i ran 2 slots simultaneously and another 30 when i ran 4.
Gotcha. You can easily do that. If youâre using something like Open WebUI you can just open a second instance of it and interact at the same time. Just rember that tokens/sec doesnât always equal more productive. The card can crank out lots of individual chats at once, but the more you add the slower each one will feel after a certain point.
yeah thats why i posted this as i am trying to figure out how to best leverage this, i find if i give each slot a similar task and shared context i can easily run 4 at once, and also i am kinda making this around my own custom harnes built around pi so the software stuff i can deal with
Good observation, what you discover is similar to hyper threading : leveraging tiny down time of the CPU to juice out every bit of power creating a continuous flow of tasks.
But on top of that, LLM workload is split in two really different task : one is compute bound and the other is famously memory bound one.
This mean that when you are generating your answer you could have use compute to decode the next question.
In the end what you are asking is what agentic AI is doing, this what is agent in essence : parallelize a given generic problem.
Switching to that would prevent you to reinvente the wheel.
Yep, this holds true on the Splash engine on macOS (and probably other engines too). For my MacBook Pro (M5 Pro 15C/16C 48GB RAM), highest total throughput is 4, then 2, then 1 concurrent session(s). Though the tok/s for each individual session gets slower the more there are.
As to how useful that observation is⌠if you were running sub-agents with the goal of completing the whole chunk of work as fast as possible, I guess it would technically be fastest that way? I tend to use 1-2 sub-agents at most (and usually 2 different models on different machines so theyâre each as fast as they can be for a single stream) so making each lane slower in order to produce a higher total throughput just isnât worth it for me.
Figure out a good bench set based on work you actually do/want to do (LLMs will be good at writing scripts to run these, and larger models can produce examples of tricky bugs that small models will struggle with). For testing this youâd need to make it a single goal with multiple steps that can be done concurrently, then you can see if itâs actually faster to run them sequentially (one session/fix at a time) or concurrently. Thatâd decide if this juice is worth the squeeze at all.
Though it also opens you up to other variables to think about/test, which can get annoying. A higher-bit quant may run slower, but produce a better result, so it burns less time fixing its own mistakes, making it practically faster. Or it could be faster, actually (I had this on my Mac, a flat 4-bit runs faster than 3-bit, itâs something with the way it does the conversion math that makes 3-bit slower to compute). And then youâve got to consider whether your temp settings are affecting the modelâs reasoning capabilities (making it too dull, or too verbose and leading to loops that burn a ton of tokens), as thatâll effect time to completion.
any suggestions on how i could automatically have a llm parllise a given task? ie convert my linear steps of intuitions that are nice and easy to give and split it up into X amount of tasks that can happen in parcel. i could either use a temporary instance of the chosen model for this or i could have my always running qwen 9b (base) run it. (it;s awlays running as i have a kev lora for model routing on a secondary gpu)
yeah good suggestion i have already been kinda doing those kinds of benchmarks, iâm running Gentoo linux and iâve been trying to debolt my system by remove X11 and 32 bit libraries where possible which makes a pretty good antigenic benchmark. the issue with higher bit quants on a 24 gib card that serious cuts context and amount of slots i can run at once, right now i can get 4 to run pretty easily and find that a single merged context for them gives the best results. r
right now my goto model is 4 bit hipfire quant of qwen 3.8 27b. also experimenting with larger models like nemotron bolt, glm 4.6 air and running most of the weights on the cpu (96 gib of ram) and they keep preforming dramatically worsen then qwen 27b.
Whatâs your make.conf look like? Path is /etc/portage/make.conf. And do you remember what profile you chose when installing? IIRC moving to a no-multilib profile is a nightmare, they say in the installation guide to pick multilib or no-multilib and stick with that, as itâs nearly impossible to switch from multilib â no-multilib.
also here are the results of a benchmark that gave me the confidence to do this. i set the global pythion target to 3_15 and mark set to install every package with the highest possible python version, which gets complicated sense everything that depends on a python pacakge needs the same python version.
Looks like the USE flags are set up right. Some stuff just demands X, so youâre best off adding a file in /etc/portage/package.use/[whatever your file is named] and flipping X on for specific apps, or removing the â-Xâ from your make.conf and flipping -X per-app. Some examples from my manual-use file:
I run a 12600KF and a 9060 XT, I donât know if youâd need some other libs if you have different hardware, Iâve just gone with the list from the Gentoo guide and then removed a couple trouble causers I didnât need, and then I think added a few that were missing? I kinda just fix stuff if it breaks, I donât have a master plan for my config files.
You can also reinstall Gentoo with no-multilib and install Steam as a flatpak (and anything else that needs 32-bit), but I like being on the latest mesa drivers so it was worth dealing with multilib for me. And the Steam flatpak can be picky if you try to install to secondary drives.
Not sure what these are required by but I mustâve added them for a reason.
Since youâre using LLMs and theyâre essentially massive pattern recognition machines, you can also point them at your configs and theyâll find things youâre missing or should redo. Tis what I did to get my USE file all cleaned up for example, and to help with stripping my kernel of any unneeded modules. Any sort of âread a bunch of information and cross-reference it to each otherâ task LLMs are wicked at.
