12 pointsby b--l4 hours ago6 comments
  • kasey_junk24 minutes ago
    > We trust them that what is behind the string "opus-5-5" is consistent and what we signed up for.

    Do you run evals as a matter of course? Everyone I know that does shows high levels of variance across them under the best case.

    That’s not to say that the labs don’t downgrade models at times, just that without any measurement it’s a non-interesting assertion.

  • faangguyindiaan hour ago
    It seems like pre-IPO companies need to balance "cost" vs. "optics" and it's one of the metrics they optimize.

    So, launch at an "exceptional" state and then cut costs after some time following the release, then repeat to make it appear like lots of advancement is happening.

    • richstokes25 minutes ago
      I have zero evidence. But anecdotally (and working with Claude every day) I see wild swings in Opus/Fables abilities day to day. It really does feel like they are twiddling knobs behind the scenes and hoping no one notices.
  • Bender3 hours ago
    For me Claude was nerfed when:

    - They put the shock collar on it [Cyber] Your session has been paused - meaning I have to start a new session. That is also on Opus 4.8 now, just not quite as sensitive as on 5.5. (Defensive coding, defensive system configuration) In my opinion Claude is not permitted to write secure code or analyze ones existing code for bugs that may be security related. Instead of the shock collar, just report anything illegal I ask Claude to do to Palantir and other related platforms like everyone else does.

    - They downgraded my 20x account to a 10x, so I downgraded it to the lowest one. I would have paid for Mythos if they would have permitted me to verify myself.

    - I keep hitting context memory limits and Claude starts acting like it has dementia, accusing me of doing things that it did. Claude is not proactively compacting when it should, especially on max. It says max does not matter but in my experience it very much does.

    I have not logged in for a couple weeks so I have no idea if the model itself has been nerfed. The feeling I got is that they want to focus on enterprise customers and would prefer individuals went away so I cleared out my data and moved my projects elsewhere that does not block me for wanting to secure my code. Perhaps that is their way of sending more individuals away? Their message has been received loud and clear.

  • turtleyacht3 hours ago
    Why not use a distributed benchmark? Tokens are shared across volunteers; everyone gets informed of the nerf simultaneously. No one need be surprised.
  • oleksg43 minutes ago
    Oh man, can't wait for open-source to catch up...
  • gigatexal4 hours ago
    I noticed the same at work using Opus 5.5 on medium which was the sweet spot. It’s now just not great.