In short:
- Don’t. Sell the Razor Core X.
- Buy another 4070 Ti, whatever it takes, to have a total of 32GB of VRAM.
- Buy a motherboard with dual x8 slots to the CPU, like ASRock X570 or X670 or X870, etc.
- 32GB of DDR4 is fine.
/TL;DR
NOTE: Razor Core X is Gen3 and TB3 x4. Super super slow.
There’s a few things here to define… LLM and Generation. LLMs will generally run great, regardless of the PCIe/TB bus.*. Image/video/re-training will suffer greatly over slow PCIe/TB links.
*With LLMs, there are two measurements:
- Prompt Processing (PP), or how fast it can read your system and user messages, as well as prompts, and uploads (PDF, images, videos - per frame analysis!, Doc or even 500,000 lines of code, etc).
- Token Generation (TG), or how fast inferencing is measured. This is the streaming text you see for your answer.
PP will vastly suffer over slow PCIe or any TB connection (they are only x2 and x4 lanes). Have a 500,000 line codebase to reference? Yeah, all that context had to be loaded every time you submit a prompt. A slow connection means very slow movement from CPU to GPU VRAM (e.g. xferinf a 200MB PDF). On a slow system, we are talking several minutes to compute everything - before it starts to think and use tools.
TG is about the only think that’s not really affected here. Inference is nice and fast - if you can fit the entire model into VRAM. It’s just the initial PP will be much slower.
Also model changing will be super slow (mine over Gen5 x16 is about 2-4 seconds. On my PCIe4 system, it’s about 6 seconds to switch.)
Also, from my Razor Core X on Linux over TB3 years ago, it was a bit unstable. It would either show on boot… or, not. And it required many complete power off cycles to get it to show again. Firmware updated and all.
I personally would not waste my time with eGPU enclosures and Ai.
With all that said, you want to stay far away from ANY type of CPU offloading. You will have to use models that 100% fit in 16GB VRAM.
However, right now - in the last 4 months - new models have gotten so good to actually rival frontier models! The problem is, that threshold is usually 24GB right now - and you really need 32GB of VRAM to have enough room for models and context and cached processing to make it viable.
I speak from my 16GB VRAM laptop experience. Doh.
So if all you got is the 4070’s 16GB of VRAM, IMO I would buy a second 4070 Ti to match. Next, I’d find an ASRock X570 motherboard (Gen4, AM4 socket) as their X570 splits the Gen4 x16 into dual x8 Gen4. Most mobos don’t do that any more. That’s the lowest cost to entry, while gaining a full 32GB of Gen4 x8 goodness that will smoke!
But maybe, in another year, we’ll have higher performing and small models? The trend sure does seem that moving that way. 