I have a question. Is there any ChatGPT type AI that runs on AMD’s 7900XT or AMD’s enterprise-based GPUs. Everything I see seems to either runs on CUDA or is CPU inference based.
You could probably use AMD’s HIP/ROCm to compile the CUDA stuff to OpenCL and Native AMD GPU code.
KoboldCPP is your best bet.
I would suggest running airoboros 33b, split between CPU and GPU, until Facebook officially releases LlamaV2 34B.
You can try LlamaV2 70B if you have a 7900 XTX + 32GB of RAM, or any system with 64GB of RAM.
OpenCL is OK right now, but rocm and vulkan are both being worked on.
1 Like
Runs on OpenCL (AMD), just wont get the high speeds.
3 Likes
This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.