Is there a way to basically use the system ram as (what did we call it back in the day?) hyper memory? Basically a slower extension to GPU memory and the CPU really doesn’t touch it
Or is that what’s already being done?
NVLink may be needed for both the GPU cores to be simultaneously be used
There are some systems especially newer and NPU devices just coming out with UNIFIED_MEMORY which llama.cpp does support (nvidia DIGITS and AMD Ryzen AI Max and Mac Metal). I believe you can kinda assign up to a certain amount to “GPU”.
I’m on a big dual socket Intel Xeon server with no GPU currently though, trying to see how much these CPUs can produce…
lmao… I got some franken torch intel stuff half going and still:
42x P cores at ~75% utilization on a single NUMA node (memory seems allocated within same node afaict) and getting a whopping 0.3 tok/sec with the safetensor version of Qwen-2.5-1.5B-Instruct haha… nope.
INFO 03-03 22:05:46 [metrics.py:470] Avg prompt throughput: 5.0 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, P
ending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 03-03 22:05:54 [metrics.py:470] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, P
ending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 03-03 22:06:02 [metrics.py:470] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, P
ending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
.
.
.
lol
I’m sure I’m doing something wrong or it defaults to using the wrong quantization of the model when auto-downloading it…
My old TI-85 calculator can get over 1 tok/sec on this model… lol jk…
omg the vllm github has over 1.4k open issues, 450 open PRs, and countless “discussions” (which just confuse me, are they new?)… I need an ai agent to scrape all that and give me the magical command to make this thing actually move… lol…
So you’ve got the layers of the model, the thing you downloaded.
Then when the model is running we need memory for the KVcache that’s the attention part of the model.
TL;DR You can offload, but it cost performance due to the wait for data to copy across.
Double check you’re using the right instruction sets
I recall Intel being a nightmare with multiple versions of similar names 512 and 16 bit instruction sets and being real picky about it
Progress! Got triton flash attention to compile for the older Mi50 cards, and I was able to see tensor parallelism working briefly with 2 cards. I’m now running up against the limitations of my testbench hardware–not enough cooling for these passive cards and not enough power from my test PSU. So now I need to move everything into my server chassis and (hopefully) see what 4 of them can do with adequate power and ventilation.
Edited to add: Just in case anyone wants to follow what I did to get the Mi50s working with vLLM: I installed the ROCm drivers as described on AMD’s site. After a reboot and checking that Linux sees the card(s), I moved on to the vLLM install. I built everything from source based on these instructions. A couple notes:
https://download.pytorch.org/whl/nightly/rocm6.3https://github.com/triton-lang/triton.gitgfx906). Then compile Triton as described in the vLLM instructions. I did not use CK flash attention.If you’ve got triton lang working, then you could also use this benchmark script for giggles xD
https://triton-lang.org/main/getting-started/tutorials/06-fused-attention.html
I am testing it on my machine: 7965wx, 8x64GB 4800 RAM, and an RTX 4090.
With IQ4_XXS on llama.cpp I get pp ~25 and tg ~5 at 0 context, down to ~3 with 5k context.
With UD-IQ1_M on llama.cpp I get pp ~50 and tg ~6 at 0 context, down to ~4 with 5k context.
With IQ4_XXS on ktransformers I get pp ~75 and tg 11 and some.
With UD-Q2_K_XL on ktransformers I get pp ~85 and tg 14.6 and some (tested with 4k context).
For ktransformers I am using:
ktransformers \
--model_path DeepSeek/DeepSeek-R1/ \
--gguf_path ../models/DeepSeek-R1-UD-Q2_K_XL/ \
--cpu_infer 22 \
--use_cuda_graph \
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml \
--max_new_tokens 8192 \
--cache_lens 32768 \
--total_context 32768 \
--cache_q4 true \
I am not really sure what the --use-cuda-graph and --optimize-config-path do, TBH. I have tried with and without and there is no difference.
According to the (somehow confused) docs and discussions on ktransformers, the .yaml file should be updated to use larger context, but to me it’s not clear how. Another mystery is the context size. With the context flags above, GPU memory usage is at 13.5 GB. --cache-lens should be deprecated. I have not yet tested the thing with 32k context.
Have you understood how ktransformers works? I have yet to understand if it limits the number of used experts, or just put the more commonly used ones in VRAM.
EDIT: again with the 2-bit quant on ktransformer, I have tested 8k context. tg remains very fast at 13.9.
EDIT 2: tried now at 25k context. It occupies 21.5 GB VRAM and tg is reduced to 10.7. It found in one take the issue I was asking (a vulnerability in a piece of javascript code).
Oh nice job! Thanks for the numbers across the different quants and inference engines. You can probably get llama.cpp faster using the experimental branch supporting custom tensor offload e.g. -ngl 62 -ot exps=CPU, but it still hasn’t landed in main 
Over 10 tok/sec is very usable though on ktransformers!
--use-cuda-graph
Maybe you are newer version than me (I’m a few days old now on ktransformers@7a19f3b`, but double check the actual argument as on this version:
$ python ktransformers/server/main.py --help | grep -i cuda
[--use_cuda_graph | --no-use_cuda_graph]
Basically if you disable it, it gets much slower. CUDA graphs only work in the situation for single user inference like we’re mostly using it here. It allows the kv-cache calculations to be sped up given they are not changing much for single-user inference.
--optimize-config-path
Yeah you specify that DeepSeek-V3-Chat.yaml which mainly allows you to map layers to different GPUs if you have multiple. The default one is best for single GPU.
Right, the arguments are clear as mud. I totally ignore that other yaml file and explicitly specify everything on the CLI. I show a verbose example in the github guide i linked above
They added a new long context feature based on storing the compressed latent space MLA in cache that supposedly fits 20k context in less VRAM but I haven’t tried it out yet, but hope to add it to the guide soon.
I can’t keep up, lol… Anyway, yes I’ve found it is quite useful for rubber duck debugging code as well as refactoring and making my comments more uniform or adding extra type checking stuff etc.
It is very usable on a 24x core threadripper like that given single NUMA node is the only way to go for now.
Ah yes, I was forgetting. Tried to set the BIOS on nps4 and passed --numa distribute in llama.cpp, the result is a significant slow down with respect to nps1.
Hi, what cache_q4 for?
Key Value Cache quantised to size float4
If people have questions about the cache, it’s part of the transformer architecture that stores attention as it runs.
With cache_q4 you can use a larger context with the same RAM. But it probably will be slower.
Yup, that is the idea. However two things:
sm89) can use the hybrid fp8 quants stuff, and If you enable the “long context” FlashMLA stuff it isn’t normal kv-cache like in regular multi-head (non-latent) attention models. I’m still trying to figure this out.If you want to dig in already here is a starting point to do your own research. It maybe possible ik_llama.cpp is faster than ktransformers with GPU and faster than vanilla llama.cpp@main with GPU+CPU and CPU only configurations.
Probably not the best tool for Apple ][ e Macintosh Metal backends, but there is a whole different world of mlx for that.
Sorry I could have answered that previous question myself by reading a little, my fault. Have a different one though. Can you people tell me your impressions from activating mmap with llama.cpp and having it disabled. As I understand mmap will load data from SSD for CPU inferencing and the OS decides which data it keeps in main memory. Have you all observed performance degradation by using this or is it mostly just as good as having the entire model in RAM in everyday use?
I would be really interested on the performance of a 9950X3D with 96GB or 128GB of memory and a PCIe5.0 SSD.
I have tried llama-bench with the same quant and the results are in the same ballpark: 89.59 pp, 14.13 tg. I have your same platform so the -20% speed on pp is surprising. The only difference is the GPU (I have a 4090 vs your A6000) and I was expecting the 4090’s pp speed to be superior to the A6000’s.
Llama-server has not the same performance for pp (using the 11446 Q2 quant). It gets 45-60 t/s. Not bad, still way worse than ktransformers, but way superior to vanilla llama.cpp.
The great advantage is that llama-server is a mature product full of features, while ktransfomers has just an openAI-compatible server which is absolutely underwhelming.
EDIT: with last updates I am running Unsloth 2.51-bit quant on ik-llama-server. Pp is 90-105 and tg is 13.5 at 5000 context. Impressing.
First time here, but long-time Follower of the channel on YT. I wanted to ask whether or not RAM of this design would operate better if the entire system was running off of different software that utilise a RAM Cache, like RAMDrive from AMD, purchasable license, but free one’s are like IMDisk that allow you to go as basically high as you want. I would be interested in seeing whether this Form Factor is any different in any regard/test methadology within a RAMDisk that the OS is running on, is everything just faster than an SSD, In theory it should be faster than a RAID-0 GEN 5 NVME, or 3 or 4. OR used to be. Worth a try. I joined to ask this, but no rush!
So if you do not have enough RAM to hold the entire model weights, you must use mmap() to allow paging weights off of your drive into the page cache. This is slower given even with RAID0 striped array of Gen5 NVMe you will be limited by async i/o to around 5GB/s max. But using this configuration, I can get over 4 tok/sec on my 9950X 96GB RAM + 3090TI FE 24GB RAM system and 32k context using this ik_llama.cpp fork specific quant
If you do have enough RAM to fit the entire model weights, you can choose to disable mmap() for one reason or another. This takes longer to startup as the model must be fully allocated and not simply sitting around in page cache.
You may get speed improvements if you do this using either Explicit Huge (EHP) pages or Transparent Huge Pages (THP). It is quite confusing and took me a while to understand and test. EHP is the older Linux Kernel mechanism and requires you to manually allocate a number of either 2MiB or 1GiB huge pages big enough to fit the model weights. Only the above mentioned ik_llama.cpp fork can use EHP by starting it with -thp 1 (confusing I know lol).
With vanilla llama.cpp or probably ollama and other downstream engines you can probably get it to load into THP by twiddling some system configs and clearing cache before starting the model. You have to check to make sure the OS actually is loading all the weights into THPs. On some test systems if the weights are not fully loaded performance is actually worse.
tl;dr; is if your system is fully dedicated to ai inferencing and you don’t use it for anything else, I would recommend trying to pre-allocate EHPs on the kernel boot parameter and loading into that and confirm it is actually faster by benchmarking.
If you use your system for various other things, don’t bother with it and maybe use THPs if you have enough extra RAM and benchmark it to confirm it is faster.
Otherwise the convenience of mmap() outweights the ~10% speed-up that you might get using huge pages.
Full discussion on how to enable and benchmark EHP and THP here on this ik_llama.cpp PR
I get just over 4 tok/sec generation with this hardware setup and discussions on how I tuned it in other threads on this forum.
Hey, great, glad you got ik_llama.cpp going! It is the best fork out there at the moment in my opinion. While ktransformers may be slightly faster in certain situations, I like the flexibility of quant types and easier ergonomics of the usual llama-server CLI. I uploaded a IQ2_K_R4 quant to hugging face linked above that likly has better perplexity than the unsloth one given mine keeps all the GPU tensors full q8_0. I figure this is fine as the GPU is not the bottle neck anyway. I’m getting around 100 tok/sec pp and yeah around 12 tg or so. Yeah there are some good llama-bench commands to test tg with longer prompts.
I don’t think a RAM drive for the OS would help anything. The software already loads the weights into RAM if you have enough. If you had enough spare RAM to create a RAM drive, I’d recommend turning it off and allocating it to hold the model weights like normal.
I have tested RAID0 and the issue is not O_DIRECT synchronous read capability, so much as it is operating system async i/o limitations. If you are running with mmap() paging data off of disk you will see kswapd0 spike as it frantically frees up overcommitted virtual memory TLB pages…
Check out above conversation on huge pages, and also I haven’t seen any noticeable gains from using google/tcmalloc dynamically linking.
@wendell
Not sure which thread you’re referring to when you said to hit you up from the CXL video that you just posted, but I would LOVE to see the CXL being used as scratch disk space for HPC/CAE/FEA applications.
(Less for for CFD, because CFD isn’t really memory intensive, contrary to what people believe.)
For FEA, especially if you are using the direct sparse solver, you can really hit the RAM/CXL hard. It would be nice if you can create a tmpfs space on the CXL device, and then use that as scratch disk space where you don’t have write endurance limits to worry about.
The only downside is that if you have a power outage, it would probably be impossible for you to dump the contents of the CXL device to some kind of non-volatile, stable storage, that you would be able to restart the FEA run from.
re. our discussion on easiest way to ask an LLM questions from a PDF document etc…
The easiest way is assuming you have enough context to fit the entire pdf text you can do this:
sudo apt-get update
sudo apt-get install poppler-utils
pdftotext --help
pdftotext input.pdf output.txt
cat output.txt | xclip -sel clip
# now push <control>+v to paste into your LLM UI after your question
I do this a lot for shorter PDFs, man pages, websites, full text scraped from youtube videos, podcasts, etc… Works great assuming you have at least 8k to 16k+ context available with your LLM setup.
If you only have a small context like 4k to work with, and the document is too big, or if you have many PDFs and many many documents then you have to look at Retrieval Augmented Generation (RAG).
There are various open source projects that essentially handle the details of chunking up all the text, store it in a vector database, etc. Then when you ask a question, it only pulls relevant “chunks” and stuffs them into the context along with your prompt and provides “references” to the chunked data.
Open WebUI can do this kind of thing for example. A friend of mine is currently exploring arc53/docsgpt to index his companies internal wiki to create a chat bot that provides references (to double check for hallucinations lol).