Simplest general way is literally asking the model for the X and Y coordinates to click, given an input image. Tends to work with most multi-modal models, although some have better accuracy than others, obviously.

More involved integration, would be to expose reading and writing to the game state or character controls via tools that the LLM can use (either in your own harness however you want, or via MCP). This is basically what most "AI tooling" connected to editors do today, one way or another, with fallback to "click on this UI element after taking a screenshot" often, or "simulate click on this widget with #id".

So given a goal, the LLM would first call check_inventory(), then craft_building(:building), then walk_to_coords(x,y), then build_building(:building) and so on, which tends to indeed be a lot faster than doing it by screenshots and inferring coordinates.

Ultimately I think the harness and prompt matters more than the model here, but I guess what they're exploring, is the shortest possible prompt to have it do it correctly. Otherwise, setting your harness to "Continue until you've launched a rocket in the game" together with a bunch of added rules to the prompts as you discover where it gets stuck, is an easy way to probably get most LLMs to be able to do this correctly and relatively fast. Some of them requires more handholding than others, which I guess might be another thing they're testing for here, it isn't 100% clear what the intention/motivation nor exact environment was/is.

Edit: As fitting as it is, I was just trying out the Unreal Engine Editor MCP/tools for the first time with Codex, which ended up also being a good example for how a "real turn" would look like, with a LLM connected to some other editor: https://gist.github.com/embedding-shapes/b16c50ed591569d3c70...

There are people much smarter than me to answer this, but the basic idea are that models have a harness around them that include something called "tools." Each tool is basically a text description of its functionality. So for something like this, for example, the model might have a "move" tool. All of the tools are added to the model's context. Then the basic idea is that when the model detects it needs to "move west 5 tiles" it will spit out a bunch of JSON to "call" the tool, and then at that point a normal program picks up that tool call output and actually performs the action. So in other words, the LLM isn't directly manipulating the environment, it's spitting out JSON that tells the harness it is in what to do.

Sleep is a bottleneck

There's vod: https://m.twitch.tv/videos/2867906622

What does Ultra offer in this context? Better code? Or is the problem being solved extremely complicated that requires Ultra?