I have now experienced this accross everything I can think to test. I have testing qwen 3.5,3.6 27B and 35BA3B and 3.8 27B. All at various quants from different sources(unsloth mainly) in llama.cpp. I have tested the latest builds, built from source, custom builds with some patches applied for other things, unsloths build they package, vulkan and rocm, lemonade. UD q4 all the way to q8_0. I have tried MTP, non mtp, cache mechanisms turned off, turned on, turning off vision, and yet everything I try I still cannot get it to stop randomly, and more frequently now devolve into //////ās.
Hereās my current active profile and I am currently running a patched version to re-enable rocWMMA but it was doing that before this change as well.
Different inference engine builds, different models, different configurationsā¦ Is your PCIE topology allowing p2p? Is that a setting you have tested disabling? I donāt know what your cards are but that kind of weird decode corruption smells multi-gpu-symptomatic. Probably a wrong guess, but 2 cents is more than nothing.
I currently do have p2p working and enabled but it happened without it as well, Iāve had it happen on bare metal, layer split, tensor split, p2p enabled and not, single gpu q4 quants I believe I had an issue with as well.
I will say though and I almost hate to admit this because I had forgotten I had done it but I had the memory overclocked a bit and it was stable for testing but now that I have realized it, I have disabled it and have not seen the issue resurface. It is quite possible that it was causing memory corruption after sustained loads for longer periods I suppose and caused this. Unfortunately I am not sure everything I needed for ECC logging to be reliable was not enabled and I did not think of this till after I disabled the overclock and restarted. As of right now I suspect this is the most likely culprit in my eyes after Iāve tried everything else.
As far as testing goes I have done that quite a bit using pretty much all defaults, the reason I pass them in right now is I hate nothing more than defaults silently changing out from underneath me and it helps me see the differences between the 5 or so profiles I switch between for reasons easily but yeah for diagnosing the issue I usually do pass much less in there the issue is its so inconsistent itās been stable for a day then itāll happen 5 times in 20 minutes.
I get this with Qwen 3.8 (ggml-org, Q4_K_M) using ROCm on my Ryzen 7840ās 780M with the HSA_OVERRIDE setting, even reducing the reasoning budget to ālowā didnāt help. I think that the model ask ROCm to do things that the hardware canāt support, so these loops of reasoning yield only a string of /. (Qwen 3.8 is thorough, diligent even, but slow on Vega.)
For me it was the memory overclock being unstable and me forgetting that I had even done it. I am not sure if you can apply it to your scenario in any way but once I resolved this issue it runs perfectly stable. As far as asking the hardware to do something it canāt handle. Not so much the hardware can handle it, there is just some instability somewhere in your chain you need to locate.
I have this problem with Gemma4:26B. it is fine at low context, but as the context starts approaching 80k tokens, it starts entering frequent loops, and above 80k it starts getting really bad (every other prompt!)
I did some research suggesting that hardware faults were unlikely the cause of this, and that it is just the nature of things in LLMās, especially lower parameter ones (which MoE models are particularly sensitive to since they have smaller active weight sizes)
The recommendation I found was to try adding a repeat penalty to my llama-cpp flags.
Default is 1.0 (in other words, none) but raising that to 1.1 to 1.15 can apparently help reduce those annoying loops. You donāt want to go too high though, as that can hamper the model in other ways.
This is set with:
āārepeat-penalty 1.1ā
I set mine recently, and havenāt had any large context prompts yet, so I canāt say if it solved things for me or not yet, but it might be worth trying.
Edit:
Oh, I didnāt realize the pattern repetitions you were seeing were nonsensical characters. That might actually point to hardware problems.
I thought you meant logic reasoning loops.
Either way, doesnāt hurt to try the setting and see if it does anything for you.
I had tried them before but this wasnāt that issue. I have run into the issue your describing quite a bit particularly with low quants of a model but after a few days ive pushed a few million tokens through it now and can be confident mine was definitely the memory overclocking. I really try not to add the repetition penalty because I donāt want to hurt the models performance.
I donāt have enough time on it yet to notice any difference with or without --repeat-penalty, but my understanding is that the key is to use it at low settings.
This is why I set mine to 1.1 Itās if you crank it up it can have some problems, but usually just in vocabulary use variety, not in logic. At least that is what I have read.
I think your probably right, in this instance I donāt need it but I will keep it in mind. Iāve read similar things, specifically I had major issues with the Ornith model which doesnāt surprise me as it is based on Qwen 3.5 which I also had major issues with. It would get into loops repeating two statements over and over, often in its reasoning output usually. Biggest things for those specifically I found is keep KV at bf16, they seem to be real sensitive to it, and lower quants suffer more for model weights as well.
I am not sure what hardware youāre running, but I had similarlyā¦ strange behaviour off of a Strix Halo box I was using. I would constantly get responses like you describe. A reboot would often set it right again, but it seemed to happen more and more. I tried tweaking with all kinds of parameters to try to get it to improve but to no avail. I was also getting terminations at prompt fill. The model wouldnāt crash or OOM, but it would simply terminate during prefill and I thought it was some kind of timeout issue but my investigations there didnāt prove fruitful either.
I read about a lot of software stack issues with AMD and it opened my eyes to some of the challenges that are hidden underneath rocm. I eventually started applying a bunch of different vLLM patches for rocm to try to fix the issue. I got frustrated since Iām admittedly pretty impatient.
What ended up working for me was to migrate away from rocm entirely. I was able to get it stable for background agents by just using the vulkan-radv toolbox instead of rocm. The performance is roughly the same, and I donāt have any of these quirks to try to work out with rocm. If youāre on the same hardware platform and donāt want to mess around with custom patches to make it work, this is a path you could take. Iāll link the docker references and my startup parameters below:
What ended up working for me was to migrate away from rocm entirely.
I was going to suggest switching to Vulkan as well. I believe that the issue is well known and is (or was) caused by āāflash-attn onā, though I thought this had been fixed.
The more time I spent reading rocm PRs the more I laughed whenever I heard Wendell talking in his videos about the maturity/growth/future of the rocm platform. It seems very optimistic to me.
Iāve lost patience with it, myself. I see many people doing good work in the community but I question why AMDs offering is soā¦ incomplete.
Thatās not to say itās not a great and accessible platform. Itās got a market, but I donāt think Iām it.
To be completely fair, the Ngreedia alternative with SM120ās is probably just as shit of a software situation.
The blackwell performance community came together to create a multitude of their own custom cuda kernels, fused gemms, mathematical optimizations, parallel vllm inference engine, benchmarks, models/quantizations, speculative decoding, and so on.
The trillion dollar titan only cares about the SM90, SM100 kind of customers. Screw everybody who ONLY dropped 10k on a rtx6k (or twelve). Luckily the continuous improvement agents running to test validate and improve the software ecosystem are banging out winner after winner, so it seems like they are ahead. If somebody gifted me a 8 pack of MI350Pās id be doing the same kind of work in parallel over there.