I have been running inference using Llama.cpp with Vulkan on a 9070 XT. Its 16GB ceiling has been limiting, especially where I’ve been exploring multimedia tools.
I know it’s a terrible time to buy hardware, but I’ve been keeping tabs on Framework’s Ryzen AI 400 desktop. Not just for memory capacity, but also for its portability.
I hesitate only because t/s is supposedly very low when actually utilizing that memory for larger >=hundred billiion parameter models. Are Strix/Gorgon users really settling for <20 t/s? And am I really ready to drop thousands on a device that apparently has a design-level mismatch between compute resource (iGPU) and total memory capacity?
I really wish they hadn’t wasted package space on an NPU and instead just included more GPU.
I built up a Threadripper with 4x (at first 3x) AMD R9700 GPUs just before prices on everything got goofy. I was debating a build like that or a 128GB AMD 395+ mini PC which would have been a LOT cheaper and a lot easier.
Main reason I didn’t go the mini PC route is I had a Minisforum HX99G that was great for a good while, but developed a hardware fault and became completely unusable. Aside from the RAM & SSDs I installed previously, there was nothing I could replace or salvage from it. I didn’t want another goofy little kinda cheaply made device I couldn’t actually service or replace major parts on.
Secondary reason is I would have almost all 128GB of VRAM available, plus most of 128GB of system RAM available, then the fact that a single R9700 is a good bit better than the 8060S iGPU, and I could upgrade or test GPUs in the future assuming the CPU & motherboard remains functional years later.
plus I am tinkering with a little ITX & 5060Ti 16GB build for smaller models, Comfy UI and other CUDA happy local AI, big bonus with the lower power draw.
I’m a big believer in common replaceable parts, especially for a primary rig, easy to reconfigure or upgrade. Can you get another 9070XT and a beefier PSU into your build, and would the 32GB of combined VRAM be sufficient enough for your uses?
I regularly spend time in different places, so a portable desktop that I can take with me would be a lifestyle improvement. Maybe I should reevaluate what counts as ‘portable’.
As it is, my primary system has already grown to accommodate a full size GPU. But that is exactly one of the pain points in making a transportable system. The weight of GPUs in their brackets and long car rides are a hazard.
It is difficult for me to gauge how much VRAM really is needed. For LLM agent work, perhaps 24GB. Maybe I should just get a R9700.
Do you actually need the hardware to move with you? Can you make remote access work for your use case? A lot of AI workloads aren’t exactly latency sensitive to interact with so the input delay isn’t as critical. I wouldn’t want to travel with a $4,000-12,000+ setup if I could avoid it.
There are software only options built into windows 11 Pro (pretty sure), and hardware plus software options like GLiNet remote KVMs that don’t require any sort of subscription. You get a delay and other downsides, but you can have a 1500w desktop weighing 70lbs powered by a UPS and connected to your local only NAS, all of that at safely at home, with remote access from a laptop or basic mini PC and portable monitor using a 5g hot spot.
The R9700 is very capable for a 32GB VRAM single GPU option that performs well at a reasonable price. You don’t get the benefits of CUDA, but software improvements and support is actively happening for the R9700 (search reddit for recent R9700 posts). It performs just like 9070XT that has double the VRAM, with probably a bit worse gaming performance than a 9070XT.
I second that idea. I have three sites over tailscale sharing resources for AI and a chromebook to connect to whatever I need on the road.
NAS/UPS/Local AI is hard to replicate on a small machine. Hardware you can service is another good argument against a ‘‘portable’’ device.
Having said that, I am on the verge of a strix halo purchase. Having a 120W device always on beats a workstation burning 24/7 just to serve models. I still would not travel with it – it would connect to it to a ups and NAS resources and have it serve over tailscale.
If speed / training matters, go with the workstation build. But there are pros and cons on both options. Just don’t carry it with you. Travel with access to your resources, don’t travel with your resources is the way I would put it.
I was going to start a GH thread but then saw this one. GH has 2 advantages:
Lower entry price(s) than Spark
Larger max UMA RAM (192 GB)
Unfortunately it lacks CUDA, so the tooling is a less mature and a bit more difficult to setup, plus inference performance is slower. Also, if you use Windows, CUDA has an all-GUI workflow while non-CUDA doesn’t.
Nice. So what speeds were you getting?
Strix Halo can do that already.
128 GB is the entry point for meaningful functionality. Below that you’re tinkering at best unless you have 96 GB GDDR7 dGPU(s).
WireGuard + RDP. Have ChatGPT or Claude step you through the server setup of the former, that’s how I migrated from OpenVPN.