Mercury 2.5 LLM hits 770 tokens per second(artificialanalysis.ai)
30 pointsby Retro_Dev2 hours ago4 comments
  • bearjawsan hour ago
    If you care about speed Cerebras gpt-oss-120b is 1400tk/s and "just as smart" in ranking.

    I've used it on a few for fun projects and its decent but the speed is crazy to watch.

  • walrus012 hours ago
    Pricing at $0.25 and $0.75 already puts its cost well above reasonably reputable inference providers for deepseek v4 flash or qwen 3.8-flash-next or similar class of open weight LLMs that fit in under 170GB of RAM, so I don't see the point. I think this is probably also stupider than laguna s 2.1 which can also be very cheap to serve.
    • an hour ago
      undefined
  • nylonstrungan hour ago
    I honestly think the diffusion LLM approach is a dead end

    It's telling that frontier labs like Google toyed around with it but didn't invest further even for their most speed and cost sensitive small models

    Still unclear for what, if any use cases this is pareto frontier

    • LarsDu8843 minutes ago
      You can't think that a small startup versus Anthropic's training setup is anywhere near the same scale to make apples to apples comparisons.

      Not sure how the Chinese labs pull it off though using autoregressive models. The secret sauce is probably going to be in the training data.

      The main reason Google hasn't switched over to DiffusionGemma is because serving at larger batch sizes loses the speed gains you get from diffusion, and most of the primary use case is serving many users at once off a single device with a large batch size.

      If you were to move to on-device low latency... like say in a robot or something, then the story might be different...

  • rvz2 hours ago
    The speed means absolutely nothing when it is finishing almost dead last when compared to the frontier AI companies.
    • copperx2 hours ago
      Ah, the old "good, fast, or cheap; pick two" proves true once again.
    • glouwbugan hour ago
      Some of us want fast food
    • voiceehan hour ago
      Not if your use case needs speed. For one of my products I can't use an LLM that has a p99 of >700ms for TTFT.
      • hansvm10 minutes ago
        If it could output 1k tokens per second but needed 4 seconds to produce the first batch of 4k, would that not be viable?