Looking for tips for local LLM

High-Performance Computing Machine Learning, LLMs, & AI
1

I’ve got deepseek-r1:14b running on my desktop machine, hooked up to Open WebUI that’s hosted on my NAS.

Looking for tips/advice on how to get the most out of my setup and/or what upgrades could be worthwhile.

Currently running the LLMs with ollama-rocm.

I know there was the video and blog post a few months ago with the setup that used both GPU and CPU, but it wasn’t working yet with AMD GPUs. Any developments on that front I should know about? Or other tips for config?

Is there a better model that I could be running with this hardware?

Here are my machine’s specs:

CPU: AMD Ryzen 9 7950X3D 4.2 GHz 16-Core (Zen 4 (Raphael))
GPU: Sapphire PULSE Radeon RX 7900 GRE 16 GB
MB: Gigabyte X670 AORUS ELITE AX (rev. 1.0) ATX AM5
Memory: Corsair Vengeance 2 x 16 GB DDR5-6000 CL30
Storage: Samsung 980 Pro 1 TB M.2-2280 PCIe 4.0 X4
Storage: Crucial P3 Plus 2TB PCIe Gen4 3D

Running NixOS if that matters.

My overall goal is to get a useful AI that I have full control over. Not because it’s handling particularly sensitive info, but just for the same reason I self-host anything or run Linux: I want to have control over my tools. I don’t want a company to be able to say “oops sorry, the price just went up.” or “that feature is not supported anymore.”

As for my use-cases, I use AI to help troubleshoot a lot of misc tech stuff. Looking at server logs to help me find problems, looking at code to help me learn how to improve it. Helping me to learn coding languages.

I’ll be continuing to research on my own to look for improvements, but I’d greatly appreciate any tips from the fine folks on here. Thanks!

2

Hello,

Deepseek-r1:14b is pretty old at this point, you should consider newer releases which are better and faster.

Consider the Qwen 3.5 35B A3B which is a MoE model instead of the dense type like your current one. It also supports image input.

You should be using llama.cpp, the Vulkan backend is fine although you’re free to experiment with the rocm backend (get Vulkan up and running first).

You can also use LM Studio in a pinch - https://lmstudio.ai/ instead of Ollama and Llama.

The above model will utilise both your CPU and the GPU since it doesn’t fit in your VRAM completely.

If you need something super fast then consider the smaller gpt-oss:20B which should fit entirely in your VRAM.

3

Your setup is actually pretty solid for local LLM work. A few thoughts:

Models to try with 16GB VRAM

DeepSeek-R1:14b is a good starting point, but you can push further. With 16GB you can comfortably run models up to ~14B at Q4 quantization, or squeeze in larger ones with more aggressive quants:

  • Qwen 2.5 Coder 14B (Q4_K_M) — significantly better for code-related tasks than DeepSeek-R1:14b. Since you mentioned troubleshooting and learning code, this one is worth trying first.
  • Mistral Small 24B (Q3_K_M or IQ3_XS) — you can just barely fit this with partial GPU offload. Noticeably smarter than 14B models for general reasoning.
  • Gemma 3 27B (Q3_K_S) — another option with partial offload. Your 7950X3D’s massive L3 cache actually helps a lot with CPU-offloaded layers.
  • Qwen 2.5 14B Instruct — strong general-purpose model if you want something less code-focused.

Optimization tips

Since you’re on ROCm with the 7900 GRE, make sure you’re running the latest ollama — AMD support has improved significantly in recent builds. You can tune num_gpu in your modelfile to control how many layers get offloaded to GPU vs CPU. With the 7950X3D, CPU inference on the remaining layers is noticeably faster than on most chips thanks to that 3D V-Cache.

Also worth trying: set OLLAMA_FLASH_ATTENTION=1 as an env var if you haven’t already — it can improve throughput on ROCm.

For your use case (log analysis, code review, learning), I’d say Qwen 2.5 Coder 14B is the single biggest upgrade you can make without changing hardware.

If you’re considering a GPU upgrade later — the jump from 16GB to 24GB VRAM is massive for local LLM. A used RTX 3090 or RX 7900 XTX would let you run 27-34B models fully in VRAM, which is a huge quality jump. When comparing cards for LLM workloads, focus on VRAM capacity and memory bandwidth — those two specs matter far more than raw TFLOPS.

4

Those models are pretty old at this point. For coding, Devstral Small 24B is better. Gemma3:27B is quite horrible at coding.

Even the smaller Qwen 3.5 releases might be better at ‘coding’ - Qwen3.5 - How to Run Locally Guide | Unsloth Documentation.

For OP’s use case these should be far better -

  1. Dense - bartowski/Qwen_Qwen3.5-27B-GGUF · Hugging Face
  2. MoE - unsloth/Qwen3.5-35B-A3B-GGUF · Hugging Face