Why harnesses matter for energy (and token) efficiency(neuralwatt.com)
22 pointsby ccgibson2 hours ago4 comments
  • randysalamian hour ago
    This article mainly argues around harnesses but a breakthrough for me was understanding why harnesses. Without a harness and only a raw LLM, exploring and modifying codebases is not feasible. It’s just text in, text out, and has no way of handling the context of the codebase (if large enough). Plug an LLM into a harness, and all of the sudden it has an exosuit that lets it funnel its thinking into discrete actions. It can now use strategies to manage that large context in pieces.
    • LaurensBER26 minutes ago
      Wait until you realize that deep down everything on a computer is text. We juist build "harnesses" to manipulate in-memory "text" in different ways.

      One level deeper is all numbers though.

      And eventually it's all 0 and 1s!

      • Terr_13 minutes ago
        "It was just a dream, Bender--there's no such thing as 2."
  • zbyan hour ago
    I don't know - all of these cache hit rate advocacy is kind of goodhearting. Cache hit rates are less important than the accuracy of LLM answers and people seem to forget about it. For example I've seen articles how sub-agents poison the cache. Sure you can fill up your context with irrelevant data to have high cache hit rate - but that degrades the LLM work - sub-agents are about providing the LLM with a clean context - just the information needed to do its job, no distraction.

    All of the main providers now offer cache management that can work with sub-agents - but it was kind of bolted on - and all of them have different apis - so nobody uses them anyway.

    • ccgibson11 minutes ago
      I agree with you, accuracy of answers and the ability to complete task is the most important thing. But once a system can achieve a goal, throughput and cost come into consideration. In this regard, I think cache rates help make things both faster and cheaper.
  • hedgehogan hour ago
    The thing that surprises me is that any of the popular harnesses break cache this way. Is it just an artifact of too much LLM code and not enough tests?
    • ccgibson12 minutes ago
      One of the main things I've noticed is that several of the techniques are well intentioned for other reasons. For example, when we debug low cache rates with users probably culprit #1 is usually some memory system. Many of those memory systems are trading a stable prefix for the ability to augment the current turns request with some new memory data. For some -- this may be the right trade-off, but the impact is slower responses times due to the extra prefill compute.

      So when a memory system promises better LLM performance the other trade-offs aren't necessarily well understood (at the expense of caching).

      Features like OMP trying to clean up history to shrink context I think are also well intentioned. But I found these trade-offs interesting hence wanting to share.

    • LaurensBER23 minutes ago
      It's easy to break the cache and there's a number of very valid usecases (mostly context management such as compaction or tool shaking).

      Not to mention that it can be very strategic to break context. E.g, after chasing a bug for a while it does make sense to summarise all the approaches that did not work so that you have more context available for newer approaches.

      Getting 99% cache hit rates while providing a good user experience is the hard part!

      • hedgehog17 minutes ago
        Compaction for sure, but putting git status in the system prompt? Why? (also I'm not sure about common approaches to compaction, they make context window management tricky and cost more generated tokens)
  • ccgibson2 hours ago
    [flagged]