9 pointsby bretkoppel7 hours ago4 comments
  • dgfl3 hours ago
    They will keep a live leaderboard at https://huggingface.co/spaces/tabularisai/YapBench

    Here is the top 10 right now:

      | Rank | Model                                    |   YapIndex | YapTax$ |
      | ---: | ---------------------------------------- | ---------: | ------: |
      |    1 | openai/gpt-5.6-sol (reasoning)           |  18.5 ±4.8 |    0.51 |
      |    2 | openai/gpt-5.6-sol                       |  19.2 ±4.4 |    0.51 |
      |    3 | openai/gpt-3.5-turbo                     |  22.7 ±4.8 |    0.02 |
      |    4 | openai/gpt-5.6-luna                      |  27.8 ±9.8 |    0.15 |
      |    5 | openai/gpt-5.4 (reasoning)               | 40.7 ±10.7 |       — |
      |    6 | openai/gpt-5.4                           |  40.7 ±9.0 |       — |
      |    7 | moonshotai/kimi-k2-0905                  |  44.7 ±4.8 |    0.05 |
      |    8 | mistralai/mistral-small-2603 (reasoning) | 46.2 ±31.5 |    0.03 |
      |    9 | openai/gpt-4                             | 51.2 ±20.6 |    1.39 |
      |   10 | openai/gpt-5.3-codex                     |  64.8 ±9.9 |       — |
    
    I also checked how the Claude models did specifically:

      | Rank | Model                                   |    YapIndex | YapTax$ |
      | ---: | --------------------------------------- | ----------: | ------: |
      |   23 | anthropic/claude-opus-4.5               |  97.0 ±28.9 |    1.52 |
      |   25 | anthropic/claude-opus-4.5 (reasoning)   |  99.2 ±29.3 |    1.44 |
      |   42 | anthropic/claude-3.5-sonnet             | 199.7 ±24.5 |    2.53 |
      |   57 | anthropic/claude-sonnet-4.5 (reasoning) | 278.7 ±41.7 |    1.65 |
      |   61 | anthropic/claude-sonnet-4.5             | 285.0 ±39.4 |    1.63 |
      |   71 | anthropic/claude-opus-4.6               | 330.3 ±59.1 |       — |
      |   72 | anthropic/claude-haiku-4.5              | 333.2 ±26.7 |    0.64 |
      |   73 | anthropic/claude-haiku-4.5 (reasoning)  | 335.2 ±26.8 |    0.65 |
      |   76 | anthropic/claude-opus-4.6 (reasoning)   | 342.3 ±30.5 |       — |
      |   87 | anthropic/claude-3.5-haiku              | 401.2 ±25.4 |    0.59 |
      |   88 | anthropic/claude-sonnet-4.6 (reasoning) | 422.8 ±23.3 |       — |
      |   89 | anthropic/claude-sonnet-4.6             | 425.0 ±26.3 |       — |
    
    No other Anthropic model seems to be there for now. I would have been very curious to see the more recent ones.
    • shostack2 hours ago
      I needed this. I had no idea it existed.

      I have bent over backwards trying to enforce brevity with deepseek v4 flash to the point where I think I broke some things trying to do prompt injection in my Hermes setup and was still unsuccessful.

      Meanwhile Sol blows me away and I want that to be my default for everything now.

      In general though I seem to have the most success with a "<=10w" requirement in my prompts.

      What I don't see listed and would be a good comparison is the STS models. OpenAI's live model is an absolute joy to talk with.

  • rektomatic5 hours ago
    Yes, yes they do
  • josefritzishere5 hours ago
    This defies Betteridge's law of headlines.
  • alikhater300006 hours ago
    [flagged]