Like are you prompting like:
--- I need code that does X,Y, and Z. Write it so that the Roslyn compiler on this machine can compile and the code passes the repo's styling and formatting requirements. ---
Or something else.
So it would be something like:
Rewrite this Python code to use match/case instead of if/elif/else chains, write a script using the ast module to rewrite the code, do not edit it yourself, also write some tests with clear inputs and outputs I can inspect.
Or something.
https://learn.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/t...
For example, with browser automation, giving the LLM raw access to the literal DOM generally results in disaster for tasks that need to be stable across more than 5-10 interactions. The better approach is to write an intermediate layer that understands each view and can provide a list of tools that are precisely tailored for each case. E.g.:
https://myapp/Login
- <raw dom - hundreds of kb>
- Available Tools: <arbitrary javascript>
https://myapp/login
- We detected that this is the application's login page.
- It has the following visible elements:
+ Username
+ Password
+ Login Button
- Available Tools:
+ PerformLogin
+ Quit
BTW, you should probably fix the Beagle link on your homepage: https://replicated.live/beagle/
In other words, are there places where a one liner for the agent would be more reliable than markdown instructions and crossing fingers?
I look at it this way... I wrote scripts over the years to make my life easier. Do the same for your agents and free their attention for the parts that matter.
[1]: https://github.com/gritzko/jab
> validating that the LLM didn't disable tests it didn't agree with
Provide a test runner and force the agent to call it. Have it emit something if you want evidence.
I also aspire to make one post a day. To be continued.
Formal methods, as in proof of correctness, have been around for decades (I was doing that stuff in the 1980s) but pushing the proofs through was too laborious. The seL4 verification effort reportedly used over a decade of people time.
The idea is that if you have a formal specification of what you want to happen, you can get a LLM to do the struggling with the proof system to get it right. It's a good task for an LLM, because there's feedback from the prover.
I'd like to see more non-trivial examples of this. People keep republishing verifications of greatest common divisor or stack algorithms, which was done decades ago.
Coming up with simple specs is not necessarily easy. You could say that is kind of what math is about. That’s how we actually make progress: find those cases where simple specs are possible and build upon them. That’s the kind of library made for eternity.
Very often, the spec is indeed just a very simple implementation. Often you can make the spec especially simple if there are no constraints on the resources it can use, at times even infinite ones.
LLMS should be abstracted out of a process as soon as practicable, replaced with deterministic processes or procedures. Otherwise you’ve built the world’s most fragile process at the mercy of token cost, vendor hostility, geopolitics, and model deprecation.
And then... yeah. You got it exactly right. Once a problem or process is deterministic, that's the wrong application of an LLM.
But I had never quite thought of it in these exact terms. The way I've been thinking about it up until now is that the very best way to use LLMs is to have them produce tools. The tools get to stay reliable and predictable. They boost your performance. But I think you found the more general abstraction of the same idea. Tool-making is not deterministic. But the tools themselves can be. That's why it fits. Trying to stuff LLMs into what's otherwise a deterministic process is an absurd waste and error-prone.
Smart. I like it.
This sounds made up or your workplace is rather odd to say the least. Maybe english isn't your first language and "threatened" is not the correct word?
Just to be clear, software development itself is not deterministic, though? The software developer pushes a given business process from less-deterministic toward more deterministic? When we say we’ve “abstracted LLMs out of a process” we’d also say that we’ve abstracted software developers out that process as well?
Would love to know how you’ve managed to counter this as the drive to throw everything at LLMs is driving me insane.
I've genuinely never considered it from this angle before.
Thus why we replaced computers (flesh and blood people writing out calculations) with computers (silicon-based number-crunching machines).
It doesn't matter what we are, what matters is what we want, and whether what we built actually works the way we want it to work.
[0] Discworld's Ponder Stibbons would be rolling in his, grave, or more likely his "Early Death package" pocket-dimension jar.
Claude or any other model just translates your natural language instructions into formally defined tool calls. You cannot replace this layer with a formal tool like Ragel. You can write code for Ragel directly, in which case the responsibility for this is yours and not Claude's. (duh)
>What about Claude? Well, my instructions say in all caps: DO NOT PARSE ANYTHING MANUALLY, EVER. (...) It tries anyway
This needs a self-verification loop. It still won't guarantee that model's interpretation will match yours, but it will improve the accuracy. Almost every model will know that it went off the rails upon checking what it's trying to do. Harness has to provide the loopback for this, because the transformer architecture doesn't.
Developers and developer-adjacent, technical people tend to think this way on their own... but every business has dark corners where repetitive, manual things still happen. We're leaning a lot on training and even org-wide LLM instructions to try and let the LLM (by its own assessment) be the vehicle use to codify a process and turn it into some good old-fashioned reviewable, deterministic automation.