Wondering if this is the disruptive step function in the AI accelerator market.
NVidia is a gaming-video company that once-served the supercomputing community as a secondary market. WIth the rise of AI. the head of NVidia is announcing all sorts of delusions.
Google has more AI-focused chips. But they (so far) do not sell to the wider market, which allows NVidia to continue to sell delusions.
What I always expected is that once we better understood the “AI” (LLM) needs, there would be a next generation of silicon designed to efficiently serve those needs.
One word: CUDA. GPUs were never particularly good at AI workloads, they’re just better than CPUs and a big reason we use them still is the huge library of software behind them. We’ve shaped the software to fit the hardware we have as much as we have evolved the hardware to fit the software. Whatever comes next needs to be “better enough” to be worth discarding that nearly two decades of familiar software. So, it will be an uphill battle even if the hardware is killer.
Tenstorrent designs ASIC for AI training and inference. There are quite a lot of ASIC companies like Tenstorrent.
Google’s TPU is another one but for its own use. Meta is also working on its own ASICs. Not to mention other independent vendors that intend to sell their ASIC to enterprises and consumers.
IMO, ASIC will be a major pillar in AI inference both in the cloud and locally.
The problem is programming API for ASICs. But we already have ROCM and Huawei. DeepSeek already ported to Huawei APIs.
So in the near future we may have: CUDA/ROCM/Huawei, and proprietary APIs. In the longer run, the industry may standardise on one. We may end up with two: the future industrial standard, and CUDA.
How familiar are you with the offering and their approach? While you’re right about there being other companies utilising ASIC, there’s plenty that TensTorrent is doing that presents a strong case.
In addition to having Jim Keller, they also offer, or are aiming to offer;
High-bandwidth chip-to-chip networking as a first-class part of the design, rather than just “here is another accelerator card” - you get the mesh the cards without having to send that compute through PCIe (!!)
Fully open source while exposing enough of the stack down to bare metal, and providing tooling so people can actually adapt and customise vertically, instead of just waiting for a vendor runtime to support their use case. (See: AMD, Intel).
But most importantly: They are trying to offer something you can actually buy, own, run locally, and build on, that runs in a home office, business, or at scale in a data centre.
Other companies like Meta etc will use ASIC, sure, but you’ll be locked onto their platform paying rent and it probably won’t be something you can put on your desk.
It even looks increasingly like this sort of “drm-style” future will be possible even hardware you buy for home, with what Nvidia are doing with fully encrypted state on the cards (and y’know, the way the economy increasingly works).
With the size of crypto mining hardware companies, ASIC fab is not going to slow down, but they might as well repeat the things they did with mining. Take the work out of the GPU and make is ASIC. LLM’s are a bit more challenging with bus requirements, but the ASIC LLM is already here, they get more advanced in every rendition, and the big labs are constantly testing on edge devices to attempt to inject LLM’s on your person, in your house, etc.
No doubt the only competition nVidia will face is ASIC market, but given the lead, I bet there is already a plan to squash anything like this early, so most of these startups are really elaborate pitch decks to sell to nVidia, ARM, or Googs/fBook
For unknown reason you tried to make a strong case for Tenstorrent but I didn’t argue Tenstorrent in a good or bad position in my previous post. I pointed out:
Tenstorrent belongs to ASICs
ASICs will be a major pillar in AI (and yes that means Tenstorrent is only one player)
ASICs need a common programming API
The API will likely be standardised industry wide in future
In the long run, we may have an industry-standard API, and CUDA.
I think ASICs for consumers and enterprises are certainly in its INFANCY stage. Tenstorrent has a head start but it means very little to the market at the moment and in the long run.
Because big cloud providers are designing their own ASICs as I previously said. The US will certainly sanction cutting-edge ASICs for China, one of the largest AI markets, if not the largest.
China won’t buy even if not sanctioned. Many firms there are designing their own ASICs. Huawei is already selling to non-domestic markets. Other firms want to sell their chips to both domestic and non-domestic.
On AI ASICs. To make an analogy of what I believe, think of ASICs like AC97/HD Audio chips in future. Capable firms can design one and will adhere to a common API.
I didn’t mean to come off argumentative. Relating to OP’s questions re: disruption, and specifically TensTorrent, I figured there was more to TensTorrent than what was covered by generalising it into the milleu of ASIC production without highlighting the specific differences that make TensTorrent interesting. I also intended to validate what what you were saying - was intended to be a “yes and” not a “yes but”.
CUDA certainly gives NVidia something of a lock. But will this continue to be so?
Looking at the Tenstorrent architecture, that fabric of tiles and interconnect looks - rather clean. Looks to scale well. But it is sufficiently different that software will be a problem, at least at first.
In the end this becomes a question of cost/compute, and they just might have advantage.
To realize potential, this is going to come down to how well the hardware architecture fits the problem, and also critically on the software stack.
Anyone (with $$$$) can develop problem-specific ASICs, but for their solution to go anywhere, they also need software. The industry needs (preferably) one software stack. Also they need something of an external focus. For the companies building silicon for internal use, the motivation to meet outside needs is less.
Tenstorrent has no internal market, so must focus outside. Are there similar players?
The global big players, let’s call them frontier transnationals, have a completely different business model that operates more like a cartel — a term created to be applied to corporate entities which collude to fix markets and pricing. Nvidia is peak in this category with all the large transnationals clustered with.
The interests funding and heavily influencing these “frontier" transnationals want everything in the cloud, their cloud and not our own private, data sovereign cloud. They all go to Davos and drink the UN-WEF Strategic Partnership Kool-aid — you know, the “you’ll own nothing and be happy“ cultists.
Along comes Jim Keller with real hardware chops and a long history designing highly successful compute. He wants to utterly commoditise ai compute and make it available to everyone. He licenses everything and builds small, medium, large and hyperscale products that cost less and are optimised for ai moreso than raster.
It is a totally different model and motivation to the frontier Davos lovers. He is running in the OPPOSITE direction. He is grounded in a reality most ordinary people share and want to build.
CUDA is designed for GPUs, for raster engines. Excellent software can be developed for excellent hardware and non proprietary excellence will explode — especially in this era of cost and supply crisis. I believe Jim is building excellent hardware highly optimised for ai and very, very cost effective to fabricate compared to the frontier extreme-hyperscale prohibitively expensive super HPC infrastructure.
Jeff Bezos just put $700M into a Series D Round with Tenstorrent and Jeff IS a devout Devos cultist so let’s see what happens to Tenstorrent.
From an enthusiast point of view I have considered in the past to get some of their cards, still waiting to see if they make their last card available -it is in the workstation but not as a standalone sale-. The software challenge sounds like painfully fun.
In terms of industry though my doubt is if Tenstorrent products scale in terms of tokens/watt and if that is relevant. They have a high ratio of networking/GB, so the power budget might be pretty high and not scale well. If they are not focused on the inference side of the business it might make sense but still I would love if someone could clarify how that is working for them.
My impression is that Tenstorrent is limited by progress in related software. The hardware model is different from current GPUs, so existing AI software may not yet be efficient on the new hardware. Put different, there might yet be a lot of headroom on the software side for improving performance on Tenstorrent.
If I were in Keller’s place, I would make my first designs a bit heavy on networking, as we cannot yet know what will be optimal in the final design. Lots of bandwidth means lots of potential for scale. Would not be surprised if Tenstorrent scales better to very large models. But that might be unclear until the software gets further.
I am sure they took their best-guess at the start, but actual use will likely differ in ways we might not expect. Put different, there yet be headroom for optimizing the hardware design, once the software has settled out.
Bit of a chicken-and-egg problem.
Likely someone(?) has a better grasp on where Tenstorrent is with software.
About scaling my question is not about the hardware performance, but the power. Bear in mind I barely understand what I am talking about.
The networking does not come free, and I don’t talk about the cost, I mean the power budget. Tenstorrent has a lot of networking per GB of RAM-weird metric, just trying to picture power budget for networking-. If I want more RAM I keep adding more and more networking at a proportionally higher ratio than other architectures.
The architectures aren’t directly comparable but Tenstorrent might have 25% more networking than Rubin has if we talk about the 32GB cards, it is a mesh so there is no switch, so some savings there. But still that is a lot of power in networking, isn’t it? And copper if I remember correctly. Nvidia has been working hard to reduce that considerable slice of the pie.
Keller has been involved in every major chip success of the last 30 years so I have no doubt there is a vision there. That is why I wonder. I mean would a Blackhole with 128GB or 256GB make sense or for their strategy to work they need to keep RAM numbers low and average Gbps high.
It is just me wishing to understand the strategy, how does it fit in the big picture. If I remember correctly when he first talked about Tenstorrent he explicitly stated something about moving complexity from the hardware to the software, so my guess is there is a relation there with smaller highly connected nodes in a mesh.
I saw some things in sanfrancisco, oddly not amd specific, around tooling to do machine translation of code. This is nothing new, that’s been around forever. But doing machine translation algorithmically with an AI layer to constantly sanity check or help pick which way the initial flowchart goes (in terms of which approach the machine translation should pick).
I don’t think CUDA is the moat anymore. And that’ll benefit everyone. It makes a bit more sense, I think, why the corporate funding for Zluda dried up in that Nvidia sort of gently tapped the IP sign and that was enough for many to pull funding… but they have created something much, much worse for themselves in the wake of “no” on cuda compatibility.. they might not realize it yet.
It’s an ASIC, it would be strange if it wasn’t absolutely wiping the floor with anything running on a “general purpose” parallel processor like a GPU. The graph shows like 50x the performance of Nvidia and their marketing claims “1000x more efficient” so they are cutting power consumption down massively as well. That’s what you can do when the logic is in the silicon instead than in software.
Imho it’s a “long term” kind of investment. Since this cards are permanently tied to a specific model at the hardware level, and that fabbing silicon is hilariously expensive, this stuff is only making sense for mature models that are already good enough for mass deployment in embedded devices and robots and whatnot.
Which I don’t think it’s the case for a lot of them. This will be mostly used for image recognition modules or sound recognition or whatever relatively limited single purpose modules for a while still. I don’t think it makes sense to burn ChatGPT on them yet.
They would become obsolete too fast and won’t recoup the manufacturing cost
Had similar thought from reading the article. Burning to silicon is a lot of overhead in cost and time. OTOH, the structure is pretty regular, and they are not re-doing the lower layers, so coming up with a new version every year might be no big deal.
Yes, expected the orders-of-magnitude speed-up. Yet getting response from actual card live on the web, was different.
Decades ago in human factors studies, we came up with response-time thresholds that mapped to roughly 0.1, 0.3, 1, and over 3 seconds. Each “feels” different in kind to a human. We are used to all LLMs landing in the “slow” response time group (some slower than others). Dropping all the way down to near-real-time is a difference in kind.
On the high performance end, this puts us in different place in mind.
On the scale-out end, this displaces a lot of hardware on the lower end. Could obviate half of NVidia’s order book.
