GPT 5.6 Sol was outperformed on all benchmarks by Gemini 3.5 Flash, apart from a single exception (OCR) where Fable was the winner.

Gemini 3.5 Flash not only outperformed GPT 5.6 Sol, but did so at 1/3 of the cost.

Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.

ChatGPT Pro with GPT 5.6-sol: https://chatgpt.com/share/6a834217-ca8c-83e8-a8e8-45d5b8797b...

The puzzle: https://activityvillage-files.s3.eu-west-2.amazonaws.com/s3f...

We have vision models for our pharmacy and I could never imagine taking the latency hit to use a Sol in our robotics, it would be likely 25-50x slower.

In the next bench, Sol looks like it’s correct again but the bboxes are rotated 90 degrees for some reason.

Overall I've been hooked on using agents from different companies for what they are best at (Thanks to Theo). Fable is expensive, but unmatched for planning and top level organization of other agents. Sol is fast, will persistantly go after goals (sometimes to its detriment), and does well with computer use.

Anecdotal but I've seen it use python to crop, zoom, and "enhance" (fiddle with sharpness and brightness) images to read sections of handwritten census data from the 1800s. Feels like that there might just be a mismatch of capabilities when it comes to straight outputting coordinates but I bet the model is better at actually finding the answer given any tools available. Which I get is a bit of an apples and oranges situation.

I've also tried to use it to identify an old pair of glasses and it didn't stand a chance, so I do think it's not quite there yet when it comes to some vision tasks.

Also it seems to be more capable, need to test more, but I think it's at least getting on par and it's fully open-source and open-weights.

Here's some benchmarks:

https://benchlm.ai/compare/gpt-5-6-sol-vs-qwen3-8-max

https://qwen.ai/blog?id=qwen3.8#full-benchmark-table (incredible UI/UX demos)

https://venturebeat.com/technology/qwen3-8-max-arrives-with-...

EDIT: Am I early to the discussion, or is none else using Qwen3.8-max?

When reasoning got introduced a year ago to GPT 5, on average the model performed worse than GPT4-o for short video clip captioning (Ie hallucinating actions that didn’t happen). The old GPT 5 was extremely finicky in terms of fps sample rate.

The other SOTA LLMs (like Gemini Pro) have clearly been optimized for long video understanding, since they can’t see almost anything sub-second (even if you up the frame sampling rate).

Sol is the first model we’ve seen to accurately caption complex sub-second movements (eg woman suddenly turns heard head to right). It’s robust to different fps sample rates so I can only guess that they trained on videos sampled at different fps.

In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.

The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models.

Their feedback loop of training on user data is incredibly strong. I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set.

Or the future will develop in llms using fine-tuned models as tools? Inference cost and speed does still seem to be below user expectations.

But for being able to one shot with this accuracy... IMPRESSIVE

I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful.

I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well.

I will add that all frontier models were fine with such tasks from the early 2024's.

I usually go to https://arena.ai/leaderboard/vision/pareto for a nice overview of current models.

I hope that whatever was lost at GDM in the last few months, didn't include their extra focus on vision capabilities.

I keep waiting for these AI companies to assemble the parts into a great autonomous driving module.

I asked it to recognize and draw the very faint reflection of what I was wearing, visible in only a tiny black part of a very brightly lit poster behind glass.

In addition, the poster itself also happened to contain similar clothing.

You can see the reference images and its output in my writeup here: https://medium.com/@rviragh/gpt-5-6-sol-very-good-image-reco...

While a human can focus on the reflection easily, this is an enormous challenge for a vision model. It's very impressive.

After I increased the game's resolution, I asked it to increase the image's size while keeping the same scale and existing content, and gosh, it constantly keeps getting something wrong no matter what I tell it, even on Sol Max with the $100 Pro subscription.

An organically-grown meat-based pixel-artist could have recreated the image and more within 2-3 days, in exchange for food and shelter.

Still very impressive, but nowhere near the text chat revolution. OpenAI still trying to strike their second lightning

It seems pretty counter intuitive that we can't do vision significantly better with specialized techniques.