Does anybody know if the arc a770 is good for inference on AI. I know there are probably better gpus out there, but if you could get a couple Arc a770s for about $300 or less each with its 16 GB of vram good AV1 encode and decode, would it not be able to run some larger models relatively easily? I don’t know if there is a way for ollama to see them all as a big gpu or something with like 32 or 48 GB or if there is an easy way to have Ollama use them as a mixture of experts with each 16GB gpu as holding its own specialized model that is hidden away from the user? I figured even though its not as polished as the enterprise stuff its relatively cheap and newer. Anyway, any feedback would be great I’m new to this and have a lot to learn.
I think I have read somewhere that Ollama can glue GPUs together for you but of course this is still not CUDA so you’ll get subpar vs Nvidia’s offering, but yes it can run.
I have seen one post in the Indian subreddits playing with three A770 for ML& AL workloads. but it was in a college research or after hours repurposing of a setup.
think of it as Craft computings 2 gamers one cpu. type of thing.
I’m no expert on this topic but from what I’ve seen there’s a trend where people buy mac mini w M series chip in configuration with some ridiculous amount of RAM for AI stuff.
You might want to look into that I think.
Lol I think I answered this yesterday, But yeah Intel arc 770s with Intel AI playground2 is probably the cheapest and easiest way to run R1 to stable diffusion and A lot more. Have a intel card I’ve used the arc 770 16gb get normal drivers for the card download and install AI playground 2, that’s it. When you try to run a model it will automatically download and install it for you.
Side note if you have a intel gpu and Nvidia gpu in the same computer you may have some problems. Intel used some hijack code I helped make to get models to use intel cards when they are making cuda calls. You dont need it to run ai playground (its in there) but if your interested in how, here is this.
The Intel AI Playground has ComfyUI built in now, so running 3 A770 is not a problem you can load different models in them and run them in a pipeline no problem.
1 Like
This is all really interesting. Does the AI playground work with ollama on linux? I already have a setup using a CPU but want to add some acceleration since it can be slow.
I heard how the Mac minis can be really good its just the lack of hardware flexibility and the price of RAM upgrades being prohibitively expensive.
Thanks for all the help