Microsoft Decision-1: 9B Qwen Model Launch Explained(shattered.io)

45 pointsby gmays5 hours ago4 comments

mstkllah4 hours ago
Reads like an ad for Microsoft and its model because it's an article about a decision model that doesn't mention Jev or other decision models at all.
Den_VR3 hours ago
No pelicans riding bicycles on this one, eh?
- kennywinker29 minutes ago
  Could one reverse the test? Feed ai generated svgs into the model and ask it to decide which of 10 animals (dog, cat, pelican, crocodile, etc) doing which of 10 actions (moonwalking, paddleboarding, sitting, riding a bicycle, etc.).
  - dannyw15 minutes ago
    AlexNet (2012) can do the first part :)
- bushbaba3 hours ago
  Instead the right app is hotdog / not hotdog
- abhis37983 hours ago
  Pelicans are deciding.
  - jumperabg3 hours ago
    Pelicans deciding if they are going to ride a bicycle.
    egeozcan11 minutes ago
    Bikes with integrated decision models deciding which pelican needs a ride the most.
    This made me realize it's been a long time since I did some brainstorming and how I missed it.
- v3ss0n3 hours ago
  Yeah the most usesless and stupidest bench .. I don't know why LLM Community thinks its good idea to use that.. It can't work on decision systems.
  - eblume3 hours ago
    It's a meme. Simon Willison started doing it because, I'm paraphrasing, it's something that (at the time) was just sort of at the edge of what you could expect an LLM to be able to do, and also it's somewhat of an impossible task since, of course, Pellicans can't ride bicycles. From there, he started writing about it along with every major LLM release he covered. His coverage is generally well regarded (hi, Simon!) and so it sticks around. It's never been intended as a serious benchmark.
    adamsmark2 hours ago
    The 3D visualizations have always made it starkly apparent the difference in quality between quantizations and thinking.
    The pelican meme is so much better than benchmaxxed stuff. Unfortunately we are likely now in pelicanmaxxing territory.
    v3ss0nan hour ago
    already , i self hosted Deekseek 4.1 Flash , Qwen 3.8 Flash Next , Glm 5.3 flash , tested it , all benchmaxxed to the point that all of them can draw that well , much better than other logos .
    v3ss0nan hour ago
    Sorry I didn't knew it was started here and , lol I didn't know it was viralized by Simon of django! Now these days , many LLMs are benchmaxxed to it.
  - locknitpicker2 hours ago
    > Yeah the most usesless and stupidest bench ..
    I recall that when this benchmark came out there were concerns that AI companies were gaming benchmarks to fake their progress. The purpose was to find something silly and outlandish that AI companies would definitely not have trained their models on, and thus have a way to check a model's capabilities.
    A pelican riding a bicycle is pretty random. That was the whole point.
liuliu4 hours ago
I haven't evaluated this one, but for what I have, it seem the pplx-decider-v1.2 (and its -flash variant) are pretty good. Anyone evaluated all these open-source decision models on their own tasks? (I evaluated on MonitoringBench).
- GavinEke4 hours ago
  I didn't look specifically at open-source but I evaluated as many decision models as I could on openrouter against how good are they at detecting phishing, which isn't the best use of these models but I was interested to see how they compared. https://emdashcyber.com/phishnchips-decision-models/
- theanonymousone3 hours ago
  I had my first use case recently, and did some very non-systematic benchmarking of models on OpenRouter.
  The only one that came close to Jev itself was the one from respan (respan/span-02), which is also open source and cheaper too.
  - mgrandlan hour ago
    Have you checked mammoth-1/elephant-1 models? They are benchmarking quite well compared and are currently free on openrouter. I currently use mammoth-1 for a small internal app and it works flawlessly for what I am doing.
    https://huggingface.co/baselabs/mammoth-1
selcuka3 hours ago
[dead]