Yeah, of those 6 tasks, only "halluc-doc-http-handler" isn't within 1% of the previous result. 86.6% is 13/15 rounded down, so if they sampled 15 attempts for that task, the probability of getting 100% when the true success rate was 13/15 would be (13/15)^15 > 0.11, which is not all that unlikely.
Thank you. Those are not the same test at all. I agree that something weird is up with Opus. But this post doesn't actually prove anything, and the title is misleading editorializing.
See original opus 4.6 sitting at 16% hallucination and the retest on 12th of april at 33%
They definitely must be doing some quantization or optimization to meet demand, otherwise why would model performance degrade this much? It's been crazy for me personally
do people get the simple fact of data sets becoming more and more polluted, one from AI, and secondly from an increasingly deranged human population that is hyper focused on getting some extra financial advantage at ANY other cost, and a huge part of that is about "reputational management", again with zero limits on missinformation.
a sane society would pull the plug, now.