Hi all, new to level1. I am not sure if this is where I should post but I wanted to get some tips on my setup and advice on how to build out an AI council with Hermes agent. I have an unraid server that I use for a lot of my home labing (Bought a lifetime license before the first price hike) and thought it would be interesting to learn a little about it. This has lead me from an RTX a4000 to 2 intel pro b70s. Intel certainly has some limitations, but 64GB for under 2,000 pre-tax was a no brainer for me. My current build is as follows.
Hardware:
Motherboard: ROG CROSSHAIR VIII HERO
CPU: Ryzen 9 5900XT
RAM: Micron ECC unregistered 2x 32GB DDR4 3200hz
Storage:
GPU: 2x Intel Arc Pro B70 @ PCIE 4.0x8
SSD: 1 TB SK hynix Platinum P41
HDD: 3x 12TB Exos 14 HDD
Networking:
Mellanox ConnectX-3
PSU:
PowerSpec 850w Gold
The build has been hit by the current Memory Crisis sadly, I had planned to upgrade over to Epyc or threadripper to get off of DDR4, be on PCIE Gen 5, and have a more server grade experience.
Software:
OS: Unraid
Inference Engine: llama.cpp:server-intel Dockers by ggml-org
Current Active MCP Servers:
Tavily
AI Performance:
Single GPU on Qwen3.6-35B-A3B-Q4_K_M
| model | size | params | backend | ngl | test | t/s |
|---|---|---|---|---|---|---|
| qwen35moe 35B.A3B Q4_K - Medium | 19.77 GiB | 34.66 B | SYCL | 999 | pp512 | 1067.06 ± 15.29 |
| qwen35moe 35B.A3B Q4_K - Medium | 19.77 GiB | 34.66 B | SYCL | 999 | tg128 | 75.49 ± 0.27 |
Dual GPU Qwen 35B. A3B
| model | size | params | backend | ngl | test | t/s |
|---|---|---|---|---|---|---|
| qwen35moe 35B.A3B Q4_K - Medium | 19.77 GiB | 34.66 B | SYCL | 999 | pp512 | 938.29 ± 13.61 |
| qwen35moe 35B.A3B Q4_K - Medium | 19.77 GiB | 34.66 B | SYCL | 999 | tg128 | 72.69 ± 0.30 |
Dual GPU Qwen3-Coder-Next-UD-Q4_K_M
| model | size | params | backend | ngl | test | t/s |
|---|---|---|---|---|---|---|
| qwen3next 80B.A3B Q4_K - Medium | 45.91 GiB | 79.67 B | SYCL | 999 | pp512 | 582.65 ± 27.57 |
| qwen3next 80B.A3B Q4_K - Medium | 45.91 GiB | 79.67 B | SYCL | 999 | tg128 | 60.00 ± 0.14 |
I currently use the Qwen 3 coder as my coding model and qwen 3.6. However, I am wanting to move to a router based agentic workflow so that I can auto swap models via Hermes or something of the like.I currently manually disable the Docker that is running dual GPUs for qwen 3 coder and start up the single GPU instance for qwen 3.6. I would like to automate this as well as have more of a council like experience. I am wanting to have a few models “discuss” the prompt and activate whichever larger model is needed. I would like to have the coder and a image generation prompt waiting to be loaded onto the GPU memory until it is called by the router. I am also curious if I can use an on prem ai for the workstation that has Hermes installed and have that be activated as an additional ai in the council. I am very green behind the ears on this and only know of some things from brief internet searches and youtube videos.