I’m pretty new to linux, and saw a video, by @wendell using a framework Desktop with an external GPU, running an LLM considerably faster (and with more RAM)
I have a GMKtec Evo-X2 and have my external gpu working through oculink on my m.2 drive port.
I’m likely using the wrong tool (LM Studio), but when I activate my 2080ti (as a proof of concept) it seems like the 8060S cannot be used, and even with a LLM like a30-3B, it runs terribly (2-3 t/s) and cannot seem to see a way to get them working in parallel? On the 8060s, alone, it goes 10x that speed…
Any help or thought would be greatly appreciated, for a Linux novice!!
ik_llama.cpp is what you want to use here. you can use a vulkan backend but for certain mixture of expert models you can run the dense layers on gpu and everything else is running from CPU.
you can kinda use ik_llama.cpp under wall if you’re using lm studio on windows.
Thats really helpful, thanks @wendell for the quick reply!! Are there any guides or other threads for doing this (sorry, I’m really new to all this and to “ik_llama.cpp”)
thats what some of our videos are about? not with strix halo but with ik_llama.cpp and this general approach. So yes, the general approach works fine.
with strix halo the problem becomes gpu bar space. I’ve had okay luck with 16/24gb vram gpus but rtx pro 6000 doesnt always work because resize bar, strix halo igpu, etc.
the minisforum pc not wanting to do the A1000 via pcie slot was also disappointing but it “can” work with fiddling. just obvious no one on the bios team (amd?) tested that or through strix halo would be used with a gpu.
the 20gb SFF 4000 plus these big-mem PCs are great tho
I’m just now seting up my Framework Desktop (128GB) and the plan is to attach the 5080 that’s on my dock right now (USB4 and then Oculink when the Redriver PCIe adapter arrives), and hopefulyl soon replace it with a 5090.
Is 32GB too much for this to address?
I, too, will be trying to make a local LLM application work with both the 8060S and the Nvidia card, depending on the model. But anything more involved than LMStudio will be new and a learning experience for me so I’m hoping to figure all this out
Did you have any luck in the end for this? I am going to try and get this working in Windows / LM Studio over this weekend as a mini-project, so if I have success I will try and write up a full guide.
I managed to get only one or the other showing up. annoyingly, when I run through Vulcan, my RTX 2080 ti is the only GPU that is picked up… however, the RAM that is allocated to the GPU for the Strix halo is “invisible” all of a sudden… However, when setting the GPU ram to 512mb via bios, I can use the (paltry) 11gb ram from the e-GPU along with the “full” 128gb of available vram… just that the models run predictably about 35% to 40% of the speed they normally do…
It sure would be amazing if someone as skilled as @wendell could make a vid / point us in the right direction, as I think there are quite a few of us who’d be very interested in matching an eGPU with their machine
Both GPU’s are visible, and “seem” to work in parallel (both gpu’s in Task Manager show utilisation) but still breaks with models over the GPU ram limit size i.e. if I load to the 2080ti first, then it goes bang… but fine if I load to the 8060S iGPU first…
I just got the MS-02 Ultra but it doesn’t recognize my DEG2. How did you make it work? I am using an RTX 3090 and the DEG2 provided TB5 cable. I had tested with all 3 USB-C ports on the MS-02 Ultra just to verify but I get the same result.
It is not recognized using the provided cable of the deg2. I tested with another TB5 cable and it gets recognized but once the GPU appears in the device manager Windows 11 freezes and I have to force shutdown everything.
The deg2 works well with the provided TB5 cable with a Minisforun UM780 XTX (the link speed is just 20Gbps but at least it works and is stable). So at least I know that the hardware is good.
I had no problem at all with the DEG02 on my Ultra. My first thought is what USB4 port on the dock. One is up and one is down. Not sure if the other works I have not switched the port on the dock. ( Wendell’s oculink thought seems like something to check off your list right away)
Got some setup and testing done with the new Max.
Success with the DEG02 on the MS02Max with 5060 tested. I had to move right back to the Ultra for some work.
I will post some results as some time frees up this week.
Mostly having fun with the Comfy on the Strix in Windows so far.
