So this is a strange one. I’m running Proxmox on one of my homelab boxes, and have a GPU passed through to an LXC for some llama.cpp local inference, partial GPU offload etc.
It’s clear that during generation the Moe layers that aren’t GPU accelerated are not running fast enough to keep up with the GPU (GPU utilisation is around 35%)
So, I’m trying to determine which part of the system is the biggest bottleneck.
It’s an old Ryzen 2400G, 64GB DDR4 2666, Quadro RTX 4000.
the CPU is clearly the weakest component in the stack, but it has been suggested that system RAM bandwidth may be the bottleneck.
I can see that the CPU is essentially maxed out during inferencing, but I can see no way of monitoring the RAM bandwidth utilisation, only the RAM capacity utilisation.
Like the ram in dual channel mode should be delivering 42.66 GB/s max. If I could see that the RAM was only delivering say 30GB/s I could confidently upgrade the CPU and expect to see a performance uplift. But if the RAM is genuinely supplying the 42.66GB/s it’s capable of, that would indicate that the CPU is already ‘keeping up’ if that makes sense.
Any ideas? Seems odd that I can’t find any tools or information that monitor that particular metric.
