Fast KV Compaction via Attention Matching(arxiv.org)
57 pointsby cbracketdash12 hours ago6 comments
  • WarmWash2 hours ago
    Considering the insanity of the AI arms race going on now, and the incredible sums of money be thrown at any slight advantage, is there any reason to believe that any meaningful AI breakthrough would be openly published for anyone to leverage?
    • 5424582 hours ago
      These folks are MIT, so citations are valuable to them. Citations convert into prestige, academic career progression, or a favorable exit from academia into industry.

      Also, I don't see why you couldn't patent this if you wanted to monetize it.

    • mikodin2 hours ago
      I would say yes.

      The reality is that the money being thrown = the time of humans. I guess compute as well, but in terms of people doing innovation - openly published things are the same thing, minus the money.

    • abeppu2 hours ago
      I do sometimes wonder -- if the transformers paper wasn't published, what would the industry be like? Would the same ideas have been put together in almost the same way weeks or months later somewhere else?
    • cmaan hour ago
      The inventor's grace period under first to file changes still gives them/their university a year to file if they publish openly.
  • cs7022 hours ago
    This looks promising. I've added it to my reading list.
  • cadamsdotcom5 hours ago
    Superficially it sounds like this could create a bit more of a move toward doing compaction on some continuous basis, or compacting in batches once you hit the context limit, rather than starting fresh with a summary and system prompt..

    Feels like high fidelity, fast compaction could be a path to “solving” long context.

  • 4 hours ago
    undefined
  • speedping4 hours ago
    This is big for long-horizon tasks
  • esafak3 hours ago
    None of the compaction accuracies look impressive.
    • yorwba3 hours ago
      I think matching or exceeding the original cache at 20% compacted size is fairly impressive.
      • esafak2 hours ago
        The original cache had 70% accuracy, and the alternatives were only worse.
        • yorwba40 minutes ago
          It sounds like you looked at figure 1 but not figure 3.