Local Ai, Any recommendations for local run Ai for coding

answered some of my own questions. it appears ollama does now support AMD!


so I started poking. at first ollama service wouldn’t start
ollama[18396]: Error: could not create directory mkdir /var/lib/ollama: permission denied

so we make the dir and give permissions (after finding the user in the service)
$ sudo mkdir /var/lib/ollama
$ sudo chown -R ollama:ollama /var/lib/ollama/

success! service starts but falls back to cpu. At first glance my 6750XT [gfx1031] looks not supported:

ollama[30982]: time=2024-03-15T17:45:38.102-06:00 level=INFO source=amd_linux.go:88 msg="detected amdgpu versions [gfx1031]"
ollama[30982]: time=2024-03-15T17:45:38.104-06:00 level=WARN source=amd_linux.go:114 msg="amdgpu [0] gfx1031 is not supported by /tmp/ollama1209781978/rocm [gfx1030 gfx1100 gfx1101 gfx1102 gfx900 gfx906 gfx908 gfx90a gfx940 gfx941 gfx942]"

As a test I ran basic ollama run llama2 and it worked but was a little slow…

However we can workaround it by adding this line under [Service] Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"

to the ollama service: /etc/systemd/system/ollama.service

Restart the service and:
systemctl restart ollama.service

Warning: The unit file, source configuration file or drop-ins of ollama.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.

okay… we’ll reload the reload the daemons then try again:
$ systemctl daemon-reload
$ systemctl restart ollama.service

and… VIOLA! :violin: It works:

ollama[44620]: time=2024-03-15T19:43:25.001-06:00 level=INFO source=amd_linux.go:88 msg="detected amdgpu versions [gfx1031]"
ollama[44620]: time=2024-03-15T19:43:25.001-06:00 level=INFO source=amd_linux.go:246 msg="[0] amdgpu totalMemory 12272M"
ollama[44620]: time=2024-03-15T19:43:25.001-06:00 level=INFO source=amd_linux.go:247 msg="[0] amdgpu freeMemory  12272M"

llama2 test is is now significantly faster. didn’t run benchmarks but the difference is night and day.

Great success!

So it looks like the 6750XT is not officially supported but does totally work after applying the HSA_OVERRIDE_GFX_VERSION=10.3.0 work around.

now to try to set up for coding…

3 Likes