That the table contains what seems to be absolute numbers for score, cost/task, time/task and output tokens, makes it seem like they've only made one run for each task/model combo, but that can't be right, right? I don't see any mentions of how many times they run each task, so if it's just one run per task/model, isn't this more noisy than useful?
I have yet to order any or program it, but it was enough to make me push on with a PCB art project for ST-style guitar pickguards - no netlist, no problems.
I'm also foolishly toying with NeXTBus dev boards for the Cube. Is it cursed? Probably. https://github.com/itomato/NeXTBus-Dev-Board
My boards are for hobby use and are ridiculously simple though compared to anything professional (breakout boards for specific components in FPV drone video transmission subsystems). That's probably an important detail. I think its like anything else in AI right now. It can do it 90% of the time but that 10% can be really rough and if its a task you can't do or verify yourself, you won't know the difference.
But I started like others, I would build manually, then run drc then sleep on it and check again and ask an llm to double check for me then order. Llms catch quite a few things but like with code like to make things more complicated than that have to be.
Super excited to see real world feedback added into the agent loops we have gotten used to working with. Could you let the model print and test the circuit boards it is prototyping with a jig?
The way I found this out is I propped it up next to a webcam so it was more or less full frame, and I had the (then new) Fable write a python script to bezier warp the camera capture to a flat projection of the screen. At that point I couldn't address the whole screen. Once I'd guided the capture script I just left the LLM overnight with the instruction to get full control confirmed by a capture round trip, and it was meaningfully finished in a couple of hours. I don't really have the skills to attempt that myself in a reasonable time frame.
Recent models can generate mostly competent schematics if you're using well known parts, feed them data sheets (and you _must_ feed them the errata too) and it's not too complex. Any complexity analog or RF, everything falls down quickly if you know what you're looking at. Maybe Astra will do better? There's still so much implicit knowledge that a good designer (not me by a long shot, but I know a little) will bake into a board, and even as cheap as JCLPCB is now, you don't want to have to spin a half dozen revs because Claude or Codex hallucinated. It will be genuinely interesting to see what happens on this front.
It will use the part datasheets and decent EE logic to cross reference pins and parts and polarities and generally check that SMD caps and resistors have realistic specifications for their footprint, switching regulators and communication ICs are configured correctly (e.g. when you have resistor settings for an ethernet PHY indicating 100Base-T and RMII, you also won't need TX/RX 2 and 3 and two clocks and a COL net, which it would check), all nets are named and linked correctly, buses have the correct and consistent termination, and so on.
It's game over once AI figures out autorouting.
Evaluating a model for specific use-cases like this that are bit more broad scope than a lot of typical microbenchmarks turned out to be quite the challenge.
Determining deterministically what a "good" electrical design is non-trivial and most electrical-engineers rely on good ol' intuition and decade long experience, so we tried to formalize it.
The only reason it was somewhat possible within a reasonable amount of effort is because atopile provides most of the complicated infrastructure for the benchmark: constraint solver, simulation, code-first electronics modeling, erc & drc checks, high level models of passive electronics components etc
It's a thinly veiled advert for Atopile. I don't know if it's a company behind it or just someone with a lot of money to spend on advertising since their about page (and docs) both 404, and the "packages" link sends you to a login screen that tries to unlock 1password. I guess claude can make circuit boards but not functioning websites?
All robots designed by robots.
Likely a crazy youtuber first
For circuits I would trust a netlist or DRC/ERC review, checking a design against reference circuit values, probably not placement as that tends to be guided straight from a datasheet where it makes a difference and I like the artistic side of it.
The next pain-point is sourcing the components from Digikey, LCSC, etc., and finding suitable substitutes if necessary.
Of course this assumes the LLMs can already read datasheets because that's the biggest pain-point in designing electronics. It's like filling out tax forms.
Finally it would be great if LLMs could extract simulation models from datasheets!
Not sure whether you played with llm datasheet extraction lately, but they are working their way up there. Even started reasoning about curves and footnotes.
Where do you think is the biggest pain point: part discovery, datasheet verification or substitute determination?
> I'm excited by the option in Astra to route PCBs.
Eurisko was used for VLSI chip design, then used rules discovered there to design TCS Traveler winning fleet.
It is unbelievable how artificial intelligence walk in circles.
[1] https:/atopile.io/
Frankly, I am surprised that it isn't already a solved problem, as we've had silicon compilers, for many years, and I always figured that IC design is more difficult than PCB.
PCB design on the other hand more often then not has outside constrains like mechanical, thermal an RF design
I’ve had similar results asking AI to design 3D models for 3D printing.
“Coach me on optimizing error on my printer”… helps.
“Build me a print ready file to these specs.”… it’s like a drunk cat attacked my computer. Just confidently puts out total nonsense.
Anyway... the posed question reminds me of an anecdote I cannot find because Google is contaminated to hell and beyond, some researchers a decade ago let a machine-learning algorithm loose on an FPGA, and it "found" a design that worked but made no sense, because it exploited unique physical features of this specific chip.
Thanks!
[1] https://www.damninteresting.com/on-the-origin-of-circuits/