Using old parts to run local AI - new to this, advice appreciated

Mandas · October 15, 2026, 12:07am

The point of this thread is to ask for advice on what I can do with my available hardware.

I can’t believe I didn’t think of this before.

My old system:

CPU: Ryzen 5800X3D
GPU: Sapphire Radeon RX Vega 64, 8 GB HBM2
- GPU driver: amdgpu, using Mesa RADV/Vulkan
RAM: 64GB DDR4 at 2133 MT/s (I seem to recall it being 2666 at some point…)
Motherboard: x470 aorus gaming wifi 7
- bios version F65
Storage: 500 GB WD Black NVMe SSD

Running ubuntu server 24.04

And the GPU is nicely working at x16, Pcie3:

$ sudo lspci -vv -s 0c:00.0 | grep -E 'LnkCap:|LnkSta:'
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 8GT/s, Width x16

After asking some instructions, I got llama.cpp set up and getting results like this:

~/llama.cpp$ ~/llama.cpp/build/bin/llama-cli   -hf bartowski/Qwen2.5-Coder-3B-Instruct-GGUF:Q4_K_M   -ngl 99   -c 4096   -p "Write a Bash script that prints the 10 largest files in my home directory."   -n 256


Loading model...

▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b11639-a0e6dc738
model      : bartowski/Qwen2.5-Coder-3B-Instruct-GGUF:Q4_K_M
ftype      : Q4_K - Medium
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern



> Write a Bash script that prints the 10 largest files in my home directory.
Certainly! Below is a Bash script that prints the 10 largest files in your home directory. This script uses the `ls`, `sort`, and `head` commands to achieve the desired output.

```bash
#!/bin/bash

# Get the home directory of the current user
home_dir="$HOME"

# Find all files in the home directory and sort them by size in descending order
# Use 'head' to get the top 10 files
find "$home_dir" -type f -print0 | xargs -0 du -h | sort -rh | head -n 10
```

### How the Script Works:
1. **`find` Command**:
   - `find "$home_dir" -type f -print0` searches for all files (`-type f`) in the specified home directory (`"$HOME"`). The `-print0` option ensures that filenames with spaces or special characters are handled correctly.

2. **`xargs` Command**:
   - `xargs -0 du -h` calculates the size of each file found by `find`. The `-0` option tells `xargs` to expect input files separated by null characters, which is the default output format for `find -print

[ Prompt: 415.0 t/s | Generation: 109.1 t/s ]

And for a larger model:

~/llama.cpp$ ~/llama.cpp/build/bin/llama-cli \
  -hf Jackrong/Qwopus3.6-27B-Coder-GGUF:Q4_K_M \
  -ngl 16 \
  -c 4096 \
  -p "Write a Bash script that finds the 10 largest files in my home directory. Explain each part of the script." \
  -n 512
Downloading mmproj-F32.gguf ──────────────────────────────────────── 100%
Downloading Qwopus3.6-27B-Coder-Q4_K_M.gguf ──────────────────────── 100%


Loading model...

▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b11639-a0e6dc738
model      : Jackrong/Qwopus3.6-27B-Coder-GGUF:Q4_K_M
ftype      : Q4_K - Medium
modalities : text, vision, video

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern
  /image <file>       add an image file
  /video <file>       add a video file



> Write a Bash script that finds the 10 largest files in my home directory. Explain each part of the script.

[Start thinking]

1.  **Understand the Goal:** The user wants a Bash script to find the 10 largest files within their home directory and an explanation of how the script works.

2.  **Identify the Core Command:** The `find` command is the standard Unix tool for locating files. To sort by size and take the top 10, we need to combine it with `sort` and `head`.
    *   *Find files:* `find ~`
    *   *Only files, not directories:* `-type f`
    *   *Print with size in a readable format (or bytes):* `-printf '%s %p\n'` (where `%s` is size in bytes, `%p` is the file path). *Alternative:* `ls -lhS` is often used but has issues with filenames containing spaces and doesn't recurse as cleanly as `find`. Let's stick to `find` for robustness, or `find ... -exec ls -lh {} + | sort ...`. Actually, the most robust and standard way is `find ~ -type f -exec ls -lh {} + | sort -k5 -rh | head -n 10`.
    *   *Wait, let's evaluate the `find -printf` vs `find + ls` approach:*
        *   `find ~ -type f -printf '%s %p\n' | sort -rn | head -n 10` -> This is very efficient. It prints the size in bytes, then the path. Then sort numerically in reverse, then take the top 10.
        *   However, showing the size in *bytes* might not be user-friendly (e.g., 1073741824 bytes).
        *   Let's use `find ~ -type f -exec ls -lhS {} + | sort -k5 -rh | head -n 10`. Wait, `ls -lhS` sorts by size already, but sorting across multiple files from different directories can be tricky with `ls` because `ls` sorts within its arguments, but `find` passes batches. Actually, `find ~ -type f -exec ls -lh {} + | sort -k5 -rh | head -n 10` works perfectly.
        *   Let's check the sorting column for `ls -lh`: The 5th column is size. `-rh`

[ Prompt: 14.0 t/s | Generation: 2.2 t/s ]

This seems awesome. Self-hosted LLM and it’s working pretty decent. Who would have thought this old Vega64 would still be useful for something.

Cloned llama.cpp and compiled locally.

Compile commands used:

cmake -B build -DGGML_VULKAN=1 -DLLAMA_OPENSSL=ON
cmake --build build --config Release -j2

Although it seems I compiled against the latest commit on the master branch, a0e6dc7 …

But, I’m pretty new to this. I’m wondering what the recommended tools there are to use. Things to enhance the experience, maybe have a local gui, quality of life things, etc…

Or maybe realize how limited I am?

Any advice is appreciated.

I’m told this is important information:

~$ env -u DISPLAY -u WAYLAND_DISPLAY vulkaninfo --summary
'DISPLAY' environment variable not set... skipping surface info
==========
VULKANINFO
==========

Vulkan Instance Version: 1.3.275


Instance Extensions: count = 24
-------------------------------
VK_EXT_acquire_drm_display             : extension revision 1
VK_EXT_acquire_xlib_display            : extension revision 1
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_direct_mode_display             : extension revision 1
VK_EXT_display_surface_counter         : extension revision 1
VK_EXT_headless_surface                : extension revision 1
VK_EXT_surface_maintenance1            : extension revision 1
VK_EXT_swapchain_colorspace            : extension revision 5
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_display                         : extension revision 23
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2         : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_portability_enumeration         : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_surface_protected_capabilities  : extension revision 1
VK_KHR_wayland_surface                 : extension revision 6
VK_KHR_xcb_surface                     : extension revision 6
VK_KHR_xlib_surface                    : extension revision 6
VK_LUNARG_direct_driver_loading        : extension revision 1

Instance Layers: count = 3
--------------------------
VK_LAYER_INTEL_nullhw       INTEL NULL HW                1.1.73   version 1
VK_LAYER_MESA_device_select Linux device selection layer 1.4.303  version 1
VK_LAYER_MESA_overlay       Mesa Overlay layer           1.4.303  version 1

Devices:
========
GPU0:
        apiVersion         = 1.4.318
        driverVersion      = 25.2.8
        vendorID           = 0x1002
        deviceID           = 0x687f
        deviceType         = PHYSICAL_DEVICE_TYPE_DISCRETE_GPU
        deviceName         = AMD Radeon RX Vega (RADV VEGA10)
        driverID           = DRIVER_ID_MESA_RADV
        driverName         = radv
        driverInfo         = Mesa 25.2.8-0ubuntu0.24.04.4
        conformanceVersion = 1.4.0.0
        deviceUUID         = 00000000-0c00-0000-0000-000000000000
        driverUUID         = 414d442d-4d45-5341-2d44-525600000000
GPU1:
        apiVersion         = 1.4.318
        driverVersion      = 25.2.8
        vendorID           = 0x10005
        deviceID           = 0x0000
        deviceType         = PHYSICAL_DEVICE_TYPE_CPU
        deviceName         = llvmpipe (LLVM 20.1.2, 256 bits)
        driverID           = DRIVER_ID_MESA_LLVMPIPE
        driverName         = llvmpipe
        driverInfo         = Mesa 25.2.8-0ubuntu0.24.04.4 (LLVM 20.1.2)
        conformanceVersion = 1.3.1.1
        deviceUUID         = 6d657361-3235-2e32-2e38-2d3075627500
        driverUUID         = 6c6c766d-7069-7065-5555-494400000000

And this:

~/llama.cpp$ ./build/bin/llama-cli --list-devices
Available devices:
  Vulkan0: AMD Radeon RX Vega (RADV VEGA10) (8192 MiB, 8180 MiB free)

Tannerdoestech · October 15, 2026, 1:03am

As fair as UI, the first thing that comes to mind is Open Web UI, which gives a “chatgpt like” web UI for your models.

Here is the Ollama Docs on open web UI

I am also running some older stuff, and planning to update later on if the cost of time is worth the reward

edit: see below for a much more thoughtful and thorough answer.

eduncan911 · October 15, 2026, 1:25am

You’re starting off on the right foot with llama.cpp, as it’s the only tool that uses Vulkan - and Vulkan is the only engine that supports the Vega series. I stuffed quad Vega64s into my ThreadRipper 2950X rig earlier in the year and ran llama.cpp across them (32GB). It wasn’t too optimized, but I quickly found that MoE models are far more efficient on this platform with the limited VRAM.

Qwen2.5 is a very fast set of models - a bit too fast for their own good. It’s only really useful for small things like intellisense and small tasks. I used to run qwen2.5 Coder 1.5B on my CPU+DRAM only, leaving the GPUs free for full models and context.

You should load up Qwen3.8 35B A3B @ Q4. Read up on how Mixture of Experts (MoE) works for this type of model, and the exact llama.cpp parameters to offload MoE. In short, only 3B parameters are active which easily fits within the 8GB VRAM - the rest of the layers will be offloaded to system DRAM. I think you must have offloaded layers in order to get 27B to work anyways, right? The Qwen 27B model is what we call a “Dense” model which is the worse model to use with very limited VRAM - dense models must fit completely into VRAM in order to get speed out of it. However, MoE can be offloaded.

You have a rare motherboard there - it supports native x8 + x8 direct to the CPU. You could pickup another Vega64 for $100 to have 16GB VRAM with layers split.

After playing with Open WebUI, I recommend setting up llama-swap in a docker image. And then configure multiple models to “swap out” when requested. Like Gemma E4B @ Q4 for writing prompts, while switching to Qwen3.8 35B A3B for coding tasks.

Oh, and try to use Docker from now on. You don’t have to compile anything, just run the long commends (by sticking in a bash file).

That 64 GB system ram is what is saving you here to overcome the small 8GB VRAM. Anything you can do to increase that to 16GB, or really 24GB is what you want to target, will blow your mind in what will fit with full context windows. Then it’s a matter of how much you want to pay to increase VRAM even more.

You may also consider upgrading that ram to 2933 or 3200, when you can afford to, as it’s basically 50% faster. Which directly translates to 50% faster tokens, regardless of your setup.

Just don’t spend too much on this setup. No matter how to slice it, you’ll be stuck on Q4 quants and while those work pretty damn well, especially with an agent harness that has a closed loop to analyze its own results and make corrections, will get you pretty far. You can fit Qwen3.8 35B A3B @ Q8 in that DRAM + VRAM for context and any other layers you can fit.

However, if you want more quality and/or less divergence, you really want to plan on larger GPUs with 24-32GB VRAM - two of them - in order to get to the level of running FP8 and FP16 variants for some solid coding tasks. But that’s $2500, minimal, to get started. $1000 for Intel B65 32GB, $1500 AMD R9700 32GB, etc. You can then reserve this Vega64 strictly for image generation using image models as most of those fit. You can also run Qwen2.5 Coder solely on this GPU as well for really fast intellisense, class refactoring, and similar small dedicated tasks. Just remember to turn down Temperature to 0.2 or even 0.0.