I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.
Write-up: "Jev Does Not Play Dice" https://kantahayashiai.github.io/posts/jev-does-not-play-dic...
I think this is a misunderstanding of what that 83% probability means. You think you're asking Jev to assign a probability to a dice roll, and if so then obviously it should pick each answer 1/6 of the time.
However, what Jev is doing is assigning a probability to the answer it should choose. And it's choosing 1 with 83% probability, as the correct answer to give, when there's no additional knowledge available.
1 is a perfectly valid answer here. In the absence of other data there's no more (nor less) valid answer than that. and I assume that just like LLMs, the token prediction must assign higher probability to one outcome.
If you keep feeding the previous answers back in (if you can do that with Jev) I assume it would start giving other numbers because it's no longer operating from zero info. Maybe that would even reach ~1/6 after enough rounds, although maybe not.
It continued alternating between the two until I got bored (around a dozen turns).
Unless your specific test is baked into its training, real probabilities require math and rough approximation at a minimum needs reasoning to sanity-check. Jev does neither. This isn't a new problem or anything unique to Jev.
From TypeSafe docs:
"Higher probability should correspond to a greater chance that the answer is correct."
"Outcomes assigned a probability of 0.2 should occur about 20% of the time."
https://docs.typesafe.ai/introduction/machine-learning-prime...
Humans also don't give a perfect 1/n probability when asked for a random number.
(Or just use a model to re-calibrate the probabilities, I like the conformal approach though as those rates are what I often care about.)
You just need some labelled data to generate the "corrected" probabilities (or thresholds to meet the specified error rates).
If you have just one example you're sending to a model, how would they guarantee 80% over your data?
FYI, for an overview, scikit's page on calibration is great [1], and my answer on Quora from a long time ago covers a specific type [2].
[1] https://scikit-learn.org/stable/modules/calibration.html
[2] https://www.quora.com/How-is-isotonic-regression-used-in-pra...
If we could get machine learning type results on images without training, that would be fantastic.
Good job to them for putting out something that does seem quite nice to use, and will likely get a bit of wider traction.
And if I'm being charitable to Jev (which is nearly impossible at this point), detecting whether blob of text is AI generated is not a "system one" question.