!!! 〜(￣▽￣〜) hi (〜￣▽￣)〜 !!!
So… Huge problem. I don’t understand free vs paid models very well. So I want to learn to benchmark in an attempt to understand.
So I’ve been using AI casually for a year now. Talking nonsense to all the online platforms, putting mustaches on your mom, ex. Copilot, ChatGPT, Grok, Gemini, Brave AI search/Leo, Meta, etc.
Over the summer I started playing with small local models using my desktop’s Radeon 6800 and system memory. I really think the tech is cool, but I don’t fully understand it.
My Server:
- CPU: 5950x
- RAM: 128GB DDR4
- GPU1: V100 32GB PCIE
- GPU2: V100 32GB PCIE
- NVME1: 2tb gen3
- NVME adaptor to pcie 16 slot - HBA 8 card
- Pool: 5x 22TB RAIDZ2, about 59T usable
I installed proxmox for the first time 5 months ago. I used copilot to help me set up proxmox, pterodactyl, home assistant, and my storage pool.
We didn’t know zfs allowed you to add drives later, so I went raid6 and mdadm; didn’t know you had to manually save the config or proxmox would break loop till the end of time. gave up for a few months.
Around september I got super disheartened at the prices of a mac studio m5u, researched for 2 weeks around early october, decided my broke booty could afford (at the time) 2 v100s. Installed them 10-11 days ago.
Then re-installed proxmox over the previous install, set up proxmox SMB share. Back up configs, and my proxmox install to an old 2tb usb hdd from college 10 years ago.
Spent a few days trying to get drivers working. Failed. Asked Muse how to let it connect and help me get drivers working. Installed SSH, installed tailscale, let muse ssh in, and set up my drivers.
- Driver: 580.159.03
- CUDA: 12.9.1
- nvcc 12.9
Llama-swap, LiteLLM (using all the free model providers listed from freellm net and subdivided by if model trains or not), and Open Web UI.
I do also have Hermes installed on the server… but I don’t really have a full grasp of how I should use such software.
Anyways back to the main problem. I don’t know how to benchmark :C My friend asked to me test out the fork of strata for v100s but that’s been a headache 
Models I’m testing:
- Qwen3.8-Flash-Next-GSQ-RCO-Q2_0, on Strata, 36G + 27G shards, max-context 262144, KV int8
- Qwen3.8-Flash-Next-UD-Q4_K_XL, on Strata, Q4, setup not started yet, needs GPUs free
- Qwen3.8-27B, on llama.cpp / llama-swap, label qwen2.5-7b-bigga, that label actually serves 27B
Engine:
- Strata-V100, CUDA 12.9.1, spec 4, spec-min-p 0.5, MTP on, prefill auto, ple-io ram, KV-resident 32768, parallel 2, GPUs [0,1], port 8081
- Llama-swap on 8080, LiteLLM, Open Web UI
Strata keeps giving us errors:
“prefill: the PLE rows’ upload failed: an illegal memory access was encountered”
“the model’s output became non-finite (NaN) twice in a row (#879); the engine has dropped its cached state - send the request again”
“strata generate: /last-survivor/applications/strata-data/packs/q2_0 is a native (IQ) pack: it needs --native SHARD1, --spec T (T >= 2) and --prefill CHUNK”