ChatGPT type AI on AMD GPUs?

Hardware GPU
1

I have a question. Is there any ChatGPT type AI that runs on AMD’s 7900XT or AMD’s enterprise-based GPUs. Everything I see seems to either runs on CUDA or is CPU inference based.

2

You could probably use AMD’s HIP/ROCm to compile the CUDA stuff to OpenCL and Native AMD GPU code.

3

KoboldCPP is your best bet.

I would suggest running airoboros 33b, split between CPU and GPU, until Facebook officially releases LlamaV2 34B.

You can try LlamaV2 70B if you have a 7900 XTX + 32GB of RAM, or any system with 64GB of RAM.

OpenCL is OK right now, but rocm and vulkan are both being worked on.

1 Like
4

Runs on OpenCL (AMD), just wont get the high speeds.

3 Likes