Beating GPT-5.6 Sol on retrieval with 100x cheaper open models(neon.com)
93 pointsby moonikakiss2 hours ago7 comments
  • mrinterweban hour ago
    There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
    • phainopepla27 minutes ago
      > Claude code does some of this by handing off the "explore" agent work to haiku

      This is no longer necessarily true. As of 2.1.198 [0] (released July 1st): "The built-in Explore agent now inherits the main session’s model (capped at opus) instead of running on haiku"

      [0] https://code.claude.com/docs/en/changelog#2-1-198

    • Malpan hour ago
      There are! Chroma has Context1, SID has SID-1, and you'd actually be surprised at how easy it is to post-train your own with pretty good pass@ recall@ ndcg@ etc.

      There's also Hornet who have shared some interesting talks & blogs lately. I don't know that I'd exclusively use agents for retrieval the way Neon outlines here as well. I think distillation similar to what ZeroEntropy has done for bespoke retrieval & reranking with _some_ agent manipulation on top-k results works better (IME).

    • footaan hour ago
      I feel like the future is people building applications with tightly integrated LLMs that work hand in hand with the application's own lifecycle and code.

      I also didn't realize that people were using agentic harnesses for search, it's an interesting idea. If the context length is short enough it should be fairly cheap compared to running "normal" agentic coding workloads where you have O(100k) context length for doing almost anything.

    • devolving-devan hour ago
      Models keep on improving though, so doesn't fine tuning become an ongoing task with ongoing maintenance burden?
      • kumamaan hour ago
        (one of the blog post authors here) -> once you set up a finetuning pipeline, it's often trivial to rerun it on top of a new open weights model. so, it's orthogonal to base model improvements
    • Razengan19 minutes ago
      > There is so much opportunity for purpose built models like this.

      OpenAI etc could themselves do this, and maybe they already do? Where the public-facing interface delegates to multiple little goblins behinds the scenes

      • mrinterweb13 minutes ago
        Exactly. There could be a lot of value for inference companies to do this. Could save a lot of money being able to hand off highly repetitive known tasks to far smaller specialized models.
  • aliljetan hour ago
    There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.
  • JCharantean hour ago
    I have done my own testing and found that smaller models can beat their larger siblings on fact retrieval from documents. I haven’t investigated it in depth with a large enough dataset but my guess is that larger models overthink it while smaller ones just do it. I would like if they compared this with 5.6 Luna instead.
    • barakean hour ago
      Anecdotally, it feels like Opus, Fable, and Sol "get distracted" when you use them for writing code. Great at reasoning and coordination but they will go off on a tangent and refactor half the code base. I only use them for reasoning (of course) and coordinating subagents.
    • andrenotgiantan hour ago
      Any data or public links you can share? That surprises me
  • breadislove37 minutes ago
    On what do you guys test the model. Its very dubious that there is no common retrieval benchmark such as browsecomp plus or similar tested. And what metric do you report?
    • krm0113 minutes ago
      Keeping track of any AI progress is becoming harder by the day, because there's ambiguity around common/clear/consistent benchmarks. Everything is constantly skewed into favourable directions.
  • ramon156an hour ago
    Bit unrelated, I realized that z.ai gives you access to deepseek 4 flash. It's incredible how well it performs when given a detailed spec. I'm not sure I've seen a model one-shot like that, and I was already impressed by gemma 4's speed and efficiency.
    • swiftcoderan hour ago
      Deepseek flash (especially after the recent update) has to be one of the most slept-on models. Price-performance is ridiculous, and its available on a number of cheap coding subscriptions
  • an hour ago
    undefined
  • richwateran hour ago
    One thing that plagues [insert current FAANG] is the large amount of corpus knowledge that is outdated/misleading or just plain wrong. I'm curious how this addresses that if it's deriving the reward function from the corpus itself.