The llama.cpp and vLLM projects move INSANELY fast. do we have a good grasp on how good tensor-parallel inference on Intel Arc is at the end of Aug 2026?

High-Performance Computing Machine Learning, LLMs, & AI
1

this is the most recent stuff I can find: Reddit

everything more recent I find just ends up linking back to that post. which like, sure, I get it, that was where folks pointed out the PR for TP SYCL that got merged into llama.cpp. I just want to know if folks around here are running it and whether or not you’ve seen improvements in performance over the past two months, especially on everyone’s favorite, Qwen3.6/8-27B. I am slowly leaning in the direction of selling my two 16GB 5060Tis and purchasing a couple of Arc Pro B65s (if I can find them) in order to have the VRAM to run that model specifically at a reasonable quant without quantizing kv.

2

Dunno.

Have to admit that NVidia buying HuggingFace makes me uncomfortable.

NVidia is making insane sums of money, mostly on accident. NVidia came out of the gaming space, and those gaming GPUs turned out to work well for AI in the form of LLMs. But big successful companies tend to be very bad at new things.

The next generation hardware tuned for LLMs will likely look very different than NVidia GPUs. That is a problem for NVidia.

Jensen in public comes across as massively delusional. Keep in mind, he got here largely on accident.

1 Like
3

Lol, that’s a pretty desperate take.

He got there through compute leadership, which nvidia have been leading for some years now.

LLMs aren’t a new thing they’ve been around several years at this point.

its not dumb luck.

I say that as someone who has not purchased an nvidia product since 2013.

1 Like
4

You are not wrong, but seems to me that NVidia got here through gaming, which lead into supercomputing, which lead into LLMs … and that only somewhat lately has GPU architecture optimized for LLMs and AI.

You do not need fp32 for LLMs, at least to what I know. :slight_smile:

I kind of got here in the opposite direction, as a few years ago I was evaluating GPUs for use in an advanced radar. Far as I could tell, some of the newer GPUs were less suited for that use, as they shifted resource away from fp32.

5

GPUs were always good at shifting and operating on vast quantities of data - that was the whole point, in fact.

Jensen getting Nvidia to the top of the pile wasn’t accidental at all, it’s down to the fact that he and his team were the first to spot that processing vast quantities of data would be incredibly useful in a world where there’s a vast quantity of data at everyone’s fingertips, so they jumped in and optimised their stack for the exact operating environments and languages which were commonly used for analysing large chunks of 0s and 1s. Quite how AMD missed the boat on that is beyond me.

Then LLMs came along, and the early devs realised that GPUs were both great for it and much cheaper than anything in the TOP500, and so they picked the only brand whose products seemed to be optimised for the task when everybody else was asleep at the wheel. Fait accomplit.

The rest was just good management and shitting on the customers who got you there in the first place. The American Dream.

And then, of course, once Nvidia had won the entire game five times over, Intel came along and gave it a half-hearted try with the GPU project they’ve been trying to get off the ground for 20 years. They failed, of course, because it takes them two or three years to get one generation’s products out the door and for some inexplicable reason they insist on putting out the weakest cards first, so everyone except YouTubers (and a few diehards) ignored them and they ended up having to ask Nvidia to bail them out.

2 Likes
6

The hardware from NVidia, to me at least, isn’t the best. But they did start their CUDA software development very early. And that is (and will for quite a while to come) a masterstroke.

Although their GTX 1080 card is a very well designed GPU and certainly in my top 3 best GPU’s of all time, the newer models from NVidia weren’t as inspired. After seeing some recent YouTube videos of Jensen addressing large crowds, the inspiration doesn’t seem to be there anymore.

Besides, I have nothing but trouble with NVidia cards in my Linux computers, while all my computers with Intel and AMD GPUs (including iGPUs) all work splendidly.

Playing around with AI on my computers I find more entertaining than gaming, but a lot of AI code only starts to perform well with CUDA. Which I find to be very unfortunate, given the problems I experience with NVidia hardware.

Personally, I don’t think Jensen/NVidia got where they are currently by accident. But if they should remain there, even given their new pretty nice (consumer) hardware, which also lacks in luster on the other end, that is debatable.

Current AI hardware is far too dependent on HBM memory. Even the new Jalapeno hardware from OpenAI is designed to use that. Not that HBM memory is that bad performance-wise, but the production process is too involved and therefore prone to manufacturing errors. And as the capacity of HBM memory needs to grow as more AI is demanded, the risks of errors in manufacturing grow exponentially. Limiting the wafer production capacity from all those companies even further.

Taalas is a company that designed an AI chip that doesn’t seem to rely on HBM memory at all. Yet, it is fast. As in: at minimum 1 order of magnitude faster than the best NVidia can do right now. It proves, to me at least, that HBM in any AI chip design isn’t the way to go.

Apparently there is also already a successor for HBM, called: zHBM. Very inspired, I know. Seems to require a lot less wafers and/or chip “packaging”, and is therefore a lot less prone to hardware errors. Which should alleviate wafer production capacity, and hopefully also a drastic reduction in purchase + operating costs for AI servers, while the surplus capacity can then be used again for every other product that nowadays needs memory to function.

1 Like
7

if Taalas or Etched end up producing a card with Qwen3.8-27B running (presumably!) at FP16 with max 262k context and it is reasonably priced (which it theoretically should be since at scale these things are fairly cheap), they are going to completely seismically shift this market.

8

also I feel like @wendell has been really diving into what you can do with Intel silicon lately, so if anyone knows the answer to my first question I’m sure he does…

1 Like
9

sycl and the hardware is no joke but good performance is not reachable on every single model/model config you could possibly run.

I am working on a howto with my 4x b70 system (128gb, lots of context) or qwen 3.8 dense but it’s been challenging to get the gpu utilization as high as it needs to be.

It seems sycl is the path but I wouldn’t characterize it as stable yet. However vulkan works great

and llm scaler is probably the most “stable” option out there even though it’s not on the bleeding edge. They must have a really good test/regression suite for llm scaler.

1 Like
10

thanks! yeah, thinking specifically of getting a pair of Battlemage somethingorothers in the Bxx class (60/65/70) because I’m finding that my pair of 5060Tis giving me 32GB just isn’t quite cutting it for Qwen3.8-27B at max ctx, plus a smart enough quant, plus unquantized kv, plus MTP. if that config can do well then it seems like Intel is the smart move when most of what we need for this model is just “more than 40GB RAM, close to the GPU.” I’m not even worried that much about pp/tg tps being a screamer, just looking for good-enough performance without degrading my output quality.

given that secondary market values even for these fairly basic Blackwell cards are starting to get ludicrous while other vendors’ hardware stays mostly sane, this seems like a good idea, but definitely open to other takes on the matter.

11

As a slight ironic aside, have a datacenter GPU (an AMD Instinct MI25) that lives in an old HP z820 (dual Xeons so more PCIe lanes). The z820 has an GTX 1050 bought as at the time a passable video card.

Deploy an LLM on that box, and it really wants to use that 1050! This makes no sense, as 2GB is too small, and transfer across PCIe 3 too slow. NVidia is kind of winning by default, though the 1050 is incidental. :slight_smile:

Is the CUDA tie to AI still as strong? I seem to be getting decent performance from LLMs on the AMD MI25, though I am unfamiliar with benchmark in this space. There seems to be quite a lot of progress with Vulcan and use with LLMs.

To my mind, LLMs are barely out of the “dancing bear” stage. A very useful bear, but when your algorithm is: step 1 - read all of memory; step 2 - read all of memory … this does not seem especially efficient. Seems like recent work aims to be a little more selective. Would not be surprised if better algorithms got us an order of magnitude improvement, just in software, over the next few years.

We might see a factor of ten improvement on the same hardware.

We might also see another factor of ten (or more) improvement with new hardware.

And to return to the original topic, I just re-built llama.cpp as quite a lot landed in the last week. There are more recent reports that suggest I might be able to get a tweaked Qwen3.8-27b to run well on my 16GB GPU … maybe. Lots of churn.

12

to your point about selectivity and efficiency, I think Qwen3.8-Flash-Next is starting to point the way with stuff like its n-gram table. there’s a lot of interesting architectural changes in that model that are letting folks do more with less.

what I’m not really clear on is that if I judge from Reddit, people increasingly seem to think there is some kind of Free Lunch™ running very aggressive quantization levels on 3.8-27B, like Q3 for the model and q8 or worse for kv. lots of “oh my output is great.” like. bro. are you sure about that? I would love to believe that the newest kinds of quantization are somehow magic but 3 is a VERY SMALL number of bits.

2 Likes
13

Hey. Folk used to apply blue marker to the edge of CDs, as they thought the sound was better. To this day, there are folk buying “premium” ethernet cables to use with their audio system. With any technology, there is an edge of somewhat fuzzy folklore.

To my eye, it looks like there is a distinct “knee” to the curve around Q4 and Q5. Seems the gain is greater than the loss, down to about Q4. But at Q3 you have fallen off the edge.

Then again, is the gain of running a larger model worth the loss in quality? I dunno.

1 Like