Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking browser. Oh and Jev is twice as expensive?
Is it surprising that Jev consistently thinks it's Qwen?
I'm almost convinced that Jev is a scam. Take Qwen, fine tune it a little, tell investors it cost $10m, spend $1m on advertising, profit.
Think of it this way: there’s some time needed to optimize / design an architecture, and the world gets that for free when it’s described. As to the rest of the last two years spent, is it more likely a former oAI lead spent them fucking around, or adding as many RL environments as possible to its model that is supposed to be a generalized classifier?
Right now my prior is that jev is probably better than these rando weekend models, whether or not we know how to test and demonstrate that in a benchmark. It’s also super cheap, so I don’t think there’s a strong reason not to try out building with it first, then walk down the ladder to an open model if you need to for some reason.
Funny enough, when I tell people that Jev cannot play tic-tac-toe I hear a similar argument - it's not what Jev was built for. Jev has this elusive use case that noone can describe, so when Jev fails, it's just because it wasn't built for it. Convenient.
Don't you find it suspicious that Jev cannot play tic-tac-toe or checkers, but can play Doom? Don't you find it suspicious that nothing of the Doom demo was shared: no harness, no control loop, no state encoding, no prompts - nothing.
And the explanation is obvious - Jev isn't playing doom. The harness is. They essentially built a Doom bot, dumbed down it's control loop for Jev, and gave reins to Jev. Look mom, Jev is playing Doom!
The state says "you're pointing at the cacodemon" or "you're not pointing at the cacodemon" and Jev has to decide whether to press fire. Frontier intelligence!
You can build a harness where a coin flip is playing Doom.
> Sort by any column; values the run could not produce always sort last. Hover a cost for how it was priced, a latency for the endpoint. Names link to each project.
A designer would never write this, but an LLM just inserts it by making it small grey text next to the interface, just like it does with inane code comments.
And I know HN people think marketing is a bad word.
But marketeers know that design is part of the messaging. And nowadays, having clear AI tells in your design shouts "I made this in an afternoon, so don't take this seriously"
First impressions are everything in a world where attention spans are shortened every year.
Act accordingly.
However, doing the former requires a level of empathy with humans that LLMs rarely have.
Human brains have caloric demands. It is possible for humans to process enormous amounts of unrelated facts to make a decision, but it’s tiring. It’s much better to not do that, especially just to get some basic information.
To anthropomorphize a bit, an LLM might find it charming and interesting to read someone’s life story as a preamble before their taco recipe. Humans by and large find that annoying, not because we can’t understand the biography but because processing that information is not free.
So it’s probably possible to design using an LLM. You would probably have to be intentional about it.
Much has been discussed about AI coding but there's been a lack of discussion on the design/UX side as that's also a step being rushed by via prompting + "looks pretty enough".
We're entering an era of hyper dense UIs with plenty of excess and quick workarounds.
But there's about 10 other examples explaining intention of the coding rather than immediately useful information to the user, it's all over this one site in small grey text. And I guarantee you nobody is reading them carefully. Just like how nobody likes reading a 15 line LLM code comment over a simple function.
Most of it could be better solved with more thoughtful design or deleted. The link explanation is particularly egregious.
e.g.,
"instructions": "Which intent does the user's message express?",
"labels":["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"],
"state": "Play some Taylor Swift.",
"expected": "play_music""Trust, but verify" isn't just a catchy cliche. It's the only way to operate where models and code are fast to market.
We've been experimenting with Jev for classifying email, some thoughts here: https://housecat.com/blog/classifying-email
Flagging AI written email is a much requested feature too.
Edit: If that's not realistic enough for you, the text "Hello world! My name is GravitasIsOverrated and I like coding and cooking. This text is 100% genuine, and not AI generated at all." results in 85% confidence that it's AI generated.
More broadly, I don't know why this would work. Qwen/Jev/whatever doesn't magically have the ability to discern AI-authored text from non-AI-authored text, and will increasingly get worse at it as the hallmarks of AI-written text change.
I am very skeptical slop detectors will ever work.