I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device.
"Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption.
Probably this will also happen for software engineering. Some usb-powered AI accelerator with Kimi K3 (and in future even better) performance running at 10K+ tokens/sec under 50W of power purchasable for almost no cost. Need a better model? Buy the new hardware. Old hardware is probably still fine for a lot of other use-cases. I expect China to be a big player here, it fits their open-model and hardware-manufacturing strategy.
I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition.
Baking models onto silicon would've been the next logical move to get a moat.
Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
I'm surprised there's not more discussion about potential inflection points here. When technology gets faster, it opens up whole new classes of UX that were hard to predict
For example, faster internet didn't mean being able to view 100x as many HTML4 web pages. It brought SaaS, streaming media and interactivity.
I'm not good at predicting, but some ideas:
1. All information gets augmented in real time with personalised context.
2. AI interaction seems more like find-as-you-type than a back and forth.
3. AI produces many outputs to pick from. Either the human, AI, or another system can do the deciding.
Even if it's last year's model, speeding up LLM inference could open up all sorts of opportunities.
I see a lot of discourse about it being fast-to-deprecation. But I see it a different way personally.
Modern LLMs are trying to do more with less. Focus on doing the right thing the first time. Even if we squeeze dumb LLMs, the significantly faster speed means quicker iterations. So a bad decision doesn't cost the time and inference costs that it cost before. It theoretically changes the scale of errant token spend.
I compare it to the 1 thousand monkeys on a typewriter. In this case it's 1,000 monkeys with stale training data of everything ever written and the ability to search the web.
Thinking that five or six years from now, Fable-level intelligence could be provided at 100x the current speed... makes me feel lost. I cannot imagine what the future will look like.
People are not talking enough how huge this is for robotics and IoT. Current robotics arhitectures are limited by tok/sec. How cares if its not a Fable model?
What I like about this, is that it significantly increases the probability of a sci-fi scenario where you're picking up a hot chip on the black market; rumor has it, Mythos 9 weights baked in...
AMD could have saved their money and used their own hardware! I've got a language model doing 60k tok/s on AMD hardware already, a Xilinx Kria K26 SOM, with the weights baked into URAM/BRAM with zero DRAM in the token loop. Same thesis as Taalas: single-stream decode is bandwidth bound, so stop fetching weights from far away.
Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though.
I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two
Something I personally haven’t seen much of, in all the discussions of model benchmarks and AI breakthroughs, is a distinction between “peak performance” and “reliable performance”. The “peak performance” of frontier models is very high: they’re solving open math problems, analyzing large codebases, etc. But my subjective impression is that “reliable performance” is mid at best: out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least.
I think there’s inherent tension between the two: the more a model reaches or outright hallucinates, the more likely it is to come up with tricky, subtle solutions to problems (I think people are somewhat like this too: Terry Tao’s brother is nonverbal, Jim Watson’s son has severe schizophrenia, etc). But then the less likely it is to generate a sensible email reply.
I use models all the time for coding, but I would not let one take over my daily correspondence. If the idea here is to run frontier models at high speed in data centers, that could be useful (the speed would be cool), but I’d be surprised if the cost of that hardware churn is worth it to frontier labs. But if the idea is to turn this into a chip that goes in your phone as some kind of routine, low-power inference thing…taking something too kooky to be relied on and baking it into your phone’s hardware like that doesn’t make sense to me.
This is probably a win-win. The team gets paid, and we get greater assurance that their best ideas and architectures -- which are truly impressive -- are going to see the light of day in actual products.
Question: we currently emulate neural networks by performing matrix math in synchronous clock CPU architectures. Would it not be better to abandon synchronization and etch neuron synapses directly in silicon, keeping only the weights variable? I think some researchers are pursuing this, but I forget what the approach is called.
1. This chip for an 8B model even if it was done at 5nm would still be twice the size of a conventional CPU die so what are the yields for this going to be like for even a 30B model?
2. They say 2 months but llama 3.1 was released 2024, ~2 years which is normal lead time for silicon, I suspect this would take longer if the architecture is not llama?
3. Can google do the same thing in house with their Gemma 4 series (two year lead time puts Gemma 4 on silicon April 2028) ?
I can see the benefit for hyper scalers but at the rate of model turn over does this type of investment make sense?
I believe this is big news. Speculations here on out: I imagine this being baked into consumer products, greatly increasing the local token capability for consumers. They will suck the cloud-oriented companies' milkshake. Most users do not need extremely capable models, they just need some automation to do better web-searches, and get simple facts etc. If it can do simple coding tasks too, but at thousands of tokens per second, in stead of tens or hundreds, the development will benefit so much. It will benefit AMD in other ways too. I imagine they can start selling physical chips, usb-drive like devices, that just does llm. If you want a newer, better, model, you simply go to a store and buy one. Need more capability, buy more drives. Similar to physx back in the day, but with usb-c and a smaller footprint.
There are market verticals where this makes a lot of sense. Embedded systems and IoT devices comes to mind.
Even in data center space, I believe several layers can use fixed weights and remaining layers will compensate for the variations. Power savings will be huge so I think there is an incentive to do more of these.
I've been eagerly awaiting their 2nd gen HC2, which uses multiple chips to host a "mid sized reasoning" [1] model. Its due in summer according to the article, I wonder if it will ever be released in that form now.
People are missing the point if they think this is useless because frontier models keep changing every few months.
We really, really need better secondary models that can do things fast and do them cheaply for lots of dumb tasks. Not only because it can be used as sub agents by frontier models, but also because it can be like a universal grease for all kinds of software.
I've got an app I am building and I don't want to tie myself with frontier models because I'll never be able to beat openai/anthropic. I just want a simple, cheap, instantaneous model that can just go through my documentation and tell the user what to do next and how to integrate with whatever ai subscription they have.
While this design is self-limiting I think its a good approach. It doesn't take an entirely new architecture or infinite memory to produce significant performance improvement.
Take an open model and put it on chip, it becomes a dedicated sub-agent. You can still use a frontier model to check things or orchestrate but having 1000 sub-agents becomes very possible. I was already long AMD but just added to my position.
While the demo is incredible, I think that in most practical use-cases, models aren't very useful without tools (search, code execution, etc.).
Even if we assume reasoning latency drops to ~0ms (AFAIK this demo doesn't include reasoning at all), these use-cases will still remain relatively slow due to I/O of tool calls.
It would be quite ironic if this technology would render all those AI data centers practically useless. If the next step are just a much smaller amount of expensive chips, and the bottleneck becomes manufacturing those chips fast. Not building huge data centers and fighting for electrical power.
I can actually see this being useful for fairly narrow workloads in dedicated devices where the model doesn't need to change very often and low-latency inference matters more than flexibility
I don't see it replacing general-purpose GPUs but it seems like a reasonable option for that kind of workload
It obviously won’t be continuous delivery but could make sense if the lifecycle of a model (train, deploy, iterate (meaningfully) is about 1-2 years. In that case it fits nicely in the “this year’s model” already established with cars, phones, etc.
Honestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.
I'm a big AI user, and have some minor experience training models, and follow some AI stories - but this is the coolest thing imo. The AI model/weights written directly on the chip!
Compaction, busting the cache, and other issues like that will lose significance when you're running at 15k tokens per second like chatjimmy. Very interesting to think about what will change in the future.
I suspect this acquisition was a response to NVIDIA's acquisition of Groq.
Cerebras is already public. AFAICT, there are 8 other startups in the space, some of which have mature products: Etched, d-Matrix, SambaNova, Tenstorrent, Positron, FuriosaAI, Rebellions and Fractile.
I have an AMD CPU as well as an AMD GPU. Their open-source philosophy and Linux compatibility, including a clean core distribution (No big.little for desktop) is what keeps me in. I think Taalas is in safe hands.
TBH Taalas was a company I was existed about as a consumer. A dense model like gemma4-31b or qwen3.6-27b running at 10k t/s sounds like an awesome thing to have. Would be willing to pay GPU prices for it.
What does it take to go from here to a model on a pcie card or an m.2 card, so I can plug one into my workstation / laptop? Will 'intelligence' become much like a gpu, where most people just live with the performance of whatever they have installed, outside large companies that must have cutting edge, or prosumers that have a incrementally better version than the masses?
Are we a couple years away, a decade away, or something else?
One advantage that I see in models that are implemented as code is that they can quickly and cheaply be modified using LoRAs. What would the equivalent be in hardware? Another piece of hardware you would attach like adding a graphics card to a computer?
I guess the idea is, gains from inference speed could offset the cost of upgrading the chips to a new model when really required. I think general purpose models would consolidate and release frequency might flatten out, favoring this strategy.
My understanding is that the chip carries the neural network itself while the weights are loaded on top, so any model that uses that same network topology should be loadable on demand.
On thing with this enormous speed is that you could create multiagent validation and correction and still have it super fast and achieve greater quality thanks to speed alone.
With web search and tool call a decent current generation model at the speed of the chatjimmy could do a lot. People saying it would be out of date are missing the point. It’s not going to make much sense for frontier companies that’s chasing the SOTA. But for a lot of business use cases if someone can put GLM 5.2 and sell it as a box, it would make so much sense.
My partner has been asking for a “completely private” model for doing research and shifting through volumes of data that can’t leave the office and $$$ for the current hardware makes no sense. It would be an easy sell if someone walks in with a black box that contains “ChatGPT”.
I would've hoped the company stayed independent instead of being engulfed into a behemoth. I'd like to see more diversity in the hardware ecosystem, but I guess the economics of hardware manufacturing aren't there.
I've been showing people chatjimmy for months - it's incredible. Both reasoning and tool use generation scale with TPS. Imagine 100x more reasoning on a model, or 100x parallel tool uses.
They moved from HBM to dedicated silicon and only got a 48x speed up? That is so, so, so much less than I would've expected. Any numbers on how it scales?
given their specs [1] show ~100 mm^2 of silicon for each billion params (albeit in 6 nm process), what will come first: nuclear fusion or one of these that can handle the multi-trillon param models of today?
I think the big news is that AMD is getting into memory-business so they won't be so dependent on Hynix and what have you. Memory is the bottleneck currently.
HN is rightly pointing out putting a model into an ASIC is kind of dumb.
HN is failing to understand that AMD knows this well.
Taalas has WO2025217724A1 pending and AMD wants that because it is immediately a function block they can sell to anyone doing FP math, since large (mostly) read only memory banks are ideally suited for that micro-code type stuff.
I’m surprised I haven’t seen anyone mention video models yet. I don’t know how many fps 17000 tok/s translates to exactly but it’s gotta be a lot. Might make real time AI video possible.
Now that I think about it, real time AI video might be a clear case of “You scientists were so preoccupied with whether you could or not, you forgot to ask if you should”
I feel like NAND process tech could become useful at solving some of these problems. A GPU where you can update the weights a few thousand times may be sufficient.
Enjoying the Ian Cutress / TechTechPotato video on Taalas. Some ok good technical details on the tech, and some good insider baseball, whose who stuff. (What a treasure having tech discussions like this about.)
https://youtu.be/3MKRjt59hh4
>"While the tech is blazing fast, if you hadn’t already figured it out, it comes with a pretty substantial downside. Once the chips are deployed you’re stuck with that model. Any change bigger than something like a LoRA adapter is going to require a re-spin of the chips, which is not only expensive but time-consuming."
That is a downside to be sure, but from a pure business perspective, "that's not a bug -- it's a feature!"... from a pure business perspective it's the ability to sell and resell, to purchasing and re-purchasing customers, way into the future -- that is, recurring revenue from the perspective of the company being able to make those future recurring sales...
In the above case, that company is AMD...
(Also, on a related note, it would be interesting to see what open source / open hardware work has currently been done to offload LLM weights (and/or anything else that could be offloaded to silicon ASIC's) to FPGA's...)
Might be an interesting motivation for looped LLMs to cut the gate count down. Perhaps even a collection of mixed programmable layers and baked layers in a loop.
"... the chip serve Meta’s Llama 3.1 8B at a blistering 16,960 tokens a second — when announced last February, that was 48x faster than Nvidia's GPUs and 8.5x faster than Cerebras' accelerators. "
There is a big risk in etching a model into silicon like this. We are still evolving what small models look like and improving their performance. When do you decide to etch one into silicon knowing that right now an improved one can be 3 months away.
Models, probably first open weight ones like Kimi K3 class, are etched into silicon like this and sold as cartridges almost like old school game cartridges.
You buy a USB-C dongle that the cartridge goes into, or for data centers you have PCI cards that take these in slots.
This is a sign of the AI bubble bursting. The 2000 bubble also saw a flood of hardware accelerators from SSL to packet switching to firewalls to graphics, most of which failed.
taalas is great for llama 3.x 8B models, really bad for one board serving Kimi K3, it seems like you would bottleneck at a few hundred tokens no matter what you do.... spreading the big model against multiple cards seems the only way to get into the 1k+ tok/sec range. Another thing taalas is doing is masking the model weights into the silicon itself, not a flashable firmware, which would increase latency....
> At 20 billion parameters per chip, you’d need just 50 accelerators to support a trillion-parameter model
I don't see any evidence that this is possible. From my understanding, the whole model needs to be on a single chip. Which rules out any popular frontier models with several trillions of parameters. Even smaller sub-frontier models have hundreds of millions of parameters, so these would be ruled out as well.
AMD acquires Taalas to boost inference performance by etching models in silicon
(theregister.com)898 points by itvision 6 August 2026 | 677 comments
Comments
"Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption.
Probably this will also happen for software engineering. Some usb-powered AI accelerator with Kimi K3 (and in future even better) performance running at 10K+ tokens/sec under 50W of power purchasable for almost no cost. Need a better model? Buy the new hardware. Old hardware is probably still fine for a lot of other use-cases. I expect China to be a big player here, it fits their open-model and hardware-manufacturing strategy.
Baking models onto silicon would've been the next logical move to get a moat.
Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
For example, faster internet didn't mean being able to view 100x as many HTML4 web pages. It brought SaaS, streaming media and interactivity.
I'm not good at predicting, but some ideas:
1. All information gets augmented in real time with personalised context.
2. AI interaction seems more like find-as-you-type than a back and forth.
3. AI produces many outputs to pick from. Either the human, AI, or another system can do the deciding.
Even if it's last year's model, speeding up LLM inference could open up all sorts of opportunities.
Modern LLMs are trying to do more with less. Focus on doing the right thing the first time. Even if we squeeze dumb LLMs, the significantly faster speed means quicker iterations. So a bad decision doesn't cost the time and inference costs that it cost before. It theoretically changes the scale of errant token spend.
I compare it to the 1 thousand monkeys on a typewriter. In this case it's 1,000 monkeys with stale training data of everything ever written and the ability to search the web.
This move undercuts NVIDIA directly.
Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though.
I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two
Something I personally haven’t seen much of, in all the discussions of model benchmarks and AI breakthroughs, is a distinction between “peak performance” and “reliable performance”. The “peak performance” of frontier models is very high: they’re solving open math problems, analyzing large codebases, etc. But my subjective impression is that “reliable performance” is mid at best: out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least.
I think there’s inherent tension between the two: the more a model reaches or outright hallucinates, the more likely it is to come up with tricky, subtle solutions to problems (I think people are somewhat like this too: Terry Tao’s brother is nonverbal, Jim Watson’s son has severe schizophrenia, etc). But then the less likely it is to generate a sensible email reply.
I use models all the time for coding, but I would not let one take over my daily correspondence. If the idea here is to run frontier models at high speed in data centers, that could be useful (the speed would be cool), but I’d be surprised if the cost of that hardware churn is worth it to frontier labs. But if the idea is to turn this into a chip that goes in your phone as some kind of routine, low-power inference thing…taking something too kooky to be relied on and baking it into your phone’s hardware like that doesn’t make sense to me.
Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out.
Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.
Economic and financial ripple effects would be huge aside from the obvious:
- reduction in electricity usage
- OpenAI / Anthropic are dead in the water unless they start to license their models to fabs.
- Every single one of those GPUs that all of those massive data centers contain become paperweights.
1. This chip for an 8B model even if it was done at 5nm would still be twice the size of a conventional CPU die so what are the yields for this going to be like for even a 30B model?
2. They say 2 months but llama 3.1 was released 2024, ~2 years which is normal lead time for silicon, I suspect this would take longer if the architecture is not llama?
3. Can google do the same thing in house with their Gemma 4 series (two year lead time puts Gemma 4 on silicon April 2028) ?
I can see the benefit for hyper scalers but at the rate of model turn over does this type of investment make sense?
In Aug 2025 you had
- OpenAI o3
- Opus 4.1
- Gemini 2.5 Pro
- Grok 4
Even if those were almost free to run, you'd be way better off with Deepseek flash 0731 or GPT 5.6 Luna, which already are almost free.
Other than for things where the t/s are critical, it seems like a bad idea to etch a model into silicon.
Guess we can look forward to picking these up ex-enterprise on ebay for under $5k a pop in a decade or two
Even in data center space, I believe several layers can use fixed weights and remaining layers will compensate for the variations. Power savings will be huge so I think there is an incentive to do more of these.
[1] https://www.forbes.com/sites/karlfreund/2026/02/19/taalas-la...
We really, really need better secondary models that can do things fast and do them cheaply for lots of dumb tasks. Not only because it can be used as sub agents by frontier models, but also because it can be like a universal grease for all kinds of software.
I've got an app I am building and I don't want to tie myself with frontier models because I'll never be able to beat openai/anthropic. I just want a simple, cheap, instantaneous model that can just go through my documentation and tell the user what to do next and how to integrate with whatever ai subscription they have.
Burnt in, it needs a zif socket and easy access in every car, aircraft, a pull out slot in a phone, or it's new era planned obselescence.
100% local and no leaks.
Even if we assume reasoning latency drops to ~0ms (AFAIK this demo doesn't include reasoning at all), these use-cases will still remain relatively slow due to I/O of tool calls.
Would it be realistic for a frontier lab to deploy this or would the turnaround time mean the model is always too out of date?
Assuming the weights and architecture are eventually stable, how much cheaper would this end up being?
Cerebras is already public. AFAICT, there are 8 other startups in the space, some of which have mature products: Etched, d-Matrix, SambaNova, Tenstorrent, Positron, FuriosaAI, Rebellions and Fractile.
Are we a couple years away, a decade away, or something else?
My partner has been asking for a “completely private” model for doing research and shifting through volumes of data that can’t leave the office and $$$ for the current hardware makes no sense. It would be an easy sell if someone walks in with a black box that contains “ChatGPT”.
At least we can be sure that's the model we wanted. Service providers could be serving modified versions and nobody would ever know.
[1] https://taalas.com/products/
"You are not prepared" --Illidan Stormrage
HN is failing to understand that AMD knows this well.
Taalas has WO2025217724A1 pending and AMD wants that because it is immediately a function block they can sell to anyone doing FP math, since large (mostly) read only memory banks are ideally suited for that micro-code type stuff.
Now that I think about it, real time AI video might be a clear case of “You scientists were so preoccupied with whether you could or not, you forgot to ask if you should”
Enjoying the Ian Cutress / TechTechPotato video on Taalas. Some ok good technical details on the tech, and some good insider baseball, whose who stuff. (What a treasure having tech discussions like this about.) https://youtu.be/3MKRjt59hh4
That is a downside to be sure, but from a pure business perspective, "that's not a bug -- it's a feature!"... from a pure business perspective it's the ability to sell and resell, to purchasing and re-purchasing customers, way into the future -- that is, recurring revenue from the perspective of the company being able to make those future recurring sales...
In the above case, that company is AMD...
(Also, on a related note, it would be interesting to see what open source / open hardware work has currently been done to offload LLM weights (and/or anything else that could be offloaded to silicon ASIC's) to FPGA's...)
Models, probably first open weight ones like Kimi K3 class, are etched into silicon like this and sold as cartridges almost like old school game cartridges.
You buy a USB-C dongle that the cartridge goes into, or for data centers you have PCI cards that take these in slots.
so instead of RTX xx70 series, I can buy xxTA that have kimi integrated ??? is that right ??
I don't see any evidence that this is possible. From my understanding, the whole model needs to be on a single chip. Which rules out any popular frontier models with several trillions of parameters. Even smaller sub-frontier models have hundreds of millions of parameters, so these would be ruled out as well.
Edit: Lol, downvotes? Stay jelly, meanwhile Talas goes brrr.