I have a RX 7900 XT in my machine and might be able to get one or two lower end RTXs.
I have been experimenting with running models locally and thought it would be interesting to try and load multiple models on different gpus and route between them.
However, here and there I read that heterogeneous hardware can be a real headache (eg, using ROCm and CUDA side-by-side) and that it should be avoided. (I don’t actually have a link to where I read this, it has been just in passing and that was the impression I got.)
I get having all the various tools installed and needing to set things up to work with both are extra steps, but is there a material limitation or disadvantage to doing this? Something beyond annoyance or inconvenience?
I’m interested this for the sake of knowing, but for added context my current plan is to put a mid-tier model on my 7900 and then lower tier models on the RTXs.
I use Ollama with a 7900 XTX. I also have a 3070 that I use for image generation with something from Pinokio that doesn’t support ROCM. I haven’t had any problems, but I really haven’t done a whole lot with it.
Short answer: works like a charm under Linux. Zero issues with both Nvidia and AMD in the same box, even using different runners. It all comes down to hardware assignments when you run the docker image (and yes, setup and run Docker for this!).
You can even combine them using llama and Vulkan if you really wanted the VRAM combination.
