Quirks from the article:

1) The performance numbers are simulated

2) The details say that the performance numbers require 2.7x the overall power, so around 13.5W. Power needed for all the memory is not explicit in the numbers.

3) The compiler is also simulated and it's a prototype, so it's not a drop-in replacement for anything like Nvidia ISAAC

4) 5W seems to refer to sub-4B models, so not the 100B param ones.

I'm getting questions weekly about running AI in-house now, in Europe trust in both the US and China seems to have hit an all-time low and AI is already scary for a lot of people.

I have no doubt that local first is the future that we're heading towards. We're seeing "algorithmic progress" towards more efficient models and _some_ hardware being available (NPUs, this chip, etc).

At the moment it doesn't seem like an appealing offer since it's just a little bit too constrained and since internet is universally available but if we can run a DeepSeek 4.1 Flash class model cheap and local, it becomes a no brainer.

80 TOP/s and 150GB/s memory gives us the fundamentals of where this sits performance wise although its not said if the TOP/s is for INT4 or FP16 or what which changes the prefill performance considerably.

A consumer graphics card (say a 9070 XT) will be something more like 750 TOP/s on INT4 (20TOP/s on FP16) and 650 GB/s. So its probably about a tenth the performance for prefill but maybe its only a third if its FP16 and can't handle more than 15GB per token due to its memory bandwidth (at Q4 that would be ~7B active parameters) for decode to achieve 10 token/s and to get its 30 token/s it would be max 2.5B active parameters at Q4. A full FP16 model on this seems very unlikely unless its a lot bigger than we think it is.

This I think is just about enough for a decent home AI, its a little slow and its only going to be of use for MOE (so no qwen 27B) but depending on price I can see it being of interest to a lot of people as some flash models aiming for 100B and 2-7B active parameters could be really quite a lot of home capability in a relatively low power consumption unit.

If it can work with 100B parameter models then stopping with edge devices makes no sense. This kind of compute-in-memory technology being so much more efficient would make most or all existing data centers obsolete. Or so it seems?

Also, they are saying this efficiency is going to become standard. I think this is the type of technology that will allow AI companies to keep increasing the size of AI models. So they will not be bottlenecked by energy needs to make the next qualitative step in intelligence.

It also points towards the next step for local AI after chips like RTX Spark. So say 18 to 36 months from now (or pick your more optimistic timeline) we expect performant local AI to be everywhere.

This mentions targets for inference of 10 to 30tps and I (albeit just skimming through) didn't find any info about ingestion speed. They also mention that the 1st gen chip is used for vision, though I suspect it's only for specific cases and not anything general.

Overall I like power efficient optimizations like this and let's hope that this is adopted broadly. Although the compute in memory sounds like a big increase in memory prices

My M5U Studio is using about 72w running a long Gemma4 job (and normal day-to-day stuff). I'm in San Diego so that's a tangible addition to electrical bill :'(

autonomous hunter drones running 100B models

> "large large language models"

Wonder how it would feel because the details shown seems a bit slow to edge device use