Look, I know Docker is an easy “push to start” for most things but for my environment where I have access to VMs and I’m running a hypervisor as-is rather than a docker-based server, it’s nice to actually get the bare metal setup.
For this I am using Fedora Server 44, and this comes with a few of its own quirks when building.
I basically spent the day because I really wanted that “double performance” for running a Radiance specific model and Quen 3.8 27B is a pretty decent model. While it’s not “amazing” I am curious now how far I can push it with RAG tools and all kinds of other things.
Recommended: Configure a static IP for your server
This guide assumes the following hardware configuration:
- 32GB of system RAM minimum (assigned to VM in my case)
- At least one Radeon R9700 GPU accessible to the system
The reason is that the MXFP4 container fails on this build with either KV cache type:
fp8→ “the ‘qwen35_fp8’ plugin does not serve an E4M3 cache for this model; i8 is the 8-bit cache”i8→ “blk.0.ssm_inz.weight is stored as MXFP4, which this build does not serve”
The IQR-V4 family (iq2r … iq8r ) is the int8-recipe line that the error itself points to. A single sentence — “Use an IQR-V4 container (iq2r–iq8r), not the MXFP4 one; the MXFP4 export’s GDN weights are not served by this build”
Without further ado, here we go.
Install git, and development packages to build from source.
sudo dnf install gcc gcc-c++ make cmake ninja-build git rocm-hip-devel rocm-smi hipcc
sudo dnf group install -y c-development
We want to do the following steps with this next shell segment
- Create
/opt/radiancefolder to contain the application - Clone the git repo (codeberg)
- Configure and build the application.
cd /opt
# cloning makes the radiance folder
git clone https://codeberg.org/StillDeadcode/radiance.git
cd /opt/radiance
# Special fedora flag is DROCM_PATH as /usr
# Most of this is default options, but to make absolute certain
cmake -S . -B build -G Ninja \
-DCMAKE_INSTALL_PREFIX=/opt/radiance \
-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
-DRAD_WITH_HIP=ON \
-DRAD_GPU_TARGETS='gfx1201' \
-DROCM_PATH=/usr
cmake --build build -j "$(nproc)"
sudo cmake --install build
We need to poke a hole in the system firewall to allow the new service to communicate on port 8080.
sudo firewall-cmd --permanent --new-service=radiance
sudo firewall-cmd --permanent --service=radiance --add-port=8080/tcp
sudo firewall-cmd --permanent --service=radiance --set-description="Radiance Local LLM Server"
sudo firewall-cmd --reload
Next we download the model we want (in my example here, it’s the IPQ6 for a 6-bit quant). I made an entry in the /srv directory
sudo mkdir -p /srv/models/qwen3.8-27b-iq6r
cd /srv/models/qwen3.8-27b-iq6r
# Download from huggingface (radiance adjacent repo)
curl -L -O https://huggingface.co/StillDeadcode/qwen3.8-27b-iqr-v4/resolve/main/qwen3.8-27b-iq6r-v4.rad?download=true
And now we can build the configuration yaml if you would like to test before we go ahead and make it a service.
sudo vi /opt/radiance.yaml
Here’s the file contents I used:
# radiance.yaml
server:
host: 192.168.x.x # This is the listening IP, keep it local
port: 8080
# Optional: point to your build tree's plugins if you are running
# the binary directly from the build directory.
# radiance-home: /opt/radiance/build/radiance_home
memory:
# Set a VRAM budget slightly below your card's 32 GB total.
# The example from the provider uses 30000 MiB for a 32 GB card.
gpu-memory-mib: 30000
models:
qwen3.8-27b-iq6r:
# Path to your downloaded .rad container, or the Hugging Face repo ID.
model: /srv/models/qwen3.8-27b-iq6r/qwen3.8-27b-iq6r-v4.rad
# Your single R9700 means tensor parallelism is 1.
tp: 1
# Use an 8-bit KV cache to save VRAM on a single card.
kv-cache-dtype: i8
# Starting context length; you can try increasing this later.
max-model-len: 32768
# Conservative number of concurrent sequences for a single GPU.
max-num-seqs: 8
# Optional but recommended: keep the token embedding on the card.
# embedding-placement: vram
Validation Steps
If you like, we can go ahead and run some checks against running the radiance service and hopefully catch any errors.
We check the yaml then confirm it can start up with everything.
/opt/radiance/bin/radiance --config /opt/radiance.yaml --check
/opt/radiance/bin/radiance --config /opt/radiance.yaml
As long as it starts without any errors, we are good to continue.
Now we can configure the service.
Create and edit the file /etc/systemd/system/radiance.service
[Unit]
Description=Radiance Local LLM Server
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStartPre=/opt/radiance/bin/radiance --config /opt/radiance.yaml --check
ExecStart=/opt/radiance/bin/radiance --config /opt/radiance.yaml
Restart=on-failure
RestartSec=10
TimeoutStopSec=60
# Raise file descriptor limit for many concurrent connections
LimitNOFILE=65536
# Locked memory for GPU pinned buffers (optional, see note below)
LimitMEMLOCK=infinity
# Logging: everything goes to the journal
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
Enable the service and turn it on.
sudo systemctl daemon-reload
sudo systemctl enable --now radiance.service
sudo systemctl start radiance.service
Now connect your favorite OpenAI API to the service and make sure it can read the models.
Optionally, you can curl that (my local server IP is 192.168.x.x). It’s important you don’t curl localhost because the server is only listening on the LAN Address of 192.168.x.x
curl -s http://192.168.x.x:8080/v1/models