is this good? so Mo costs more money per bug? I'm confused.
The main thing we're trying to show is that Mo finds over 3x as many correct bugs in the same amount of time as the next-best QA agent. This is the main thing we optimize for since the worst thing you can do in QA is miss a real bug and ship broken code to prod. Currently, the tradeoff you make for that better recall is higher cost. I would say the precision is more or less comparable, but marginally lower.
Obviously we'd love to be beating every other agent on all of these numbers, and we're actively working on it, but for now this is an honest and transparent look at how the tool is doing.
Feel free to take a look at the full writeup for more detail!
I also do believe search quality is recall dominated, but the link between the metrics and results are not as clear as they could have been. I think you have a stronger sell here than it looks at first glance.
Congratulations to the team tho! Those are very nice results
Some questions are top of mind after having done POCs and trials with other companies that have products in this space (excuse me if they are answered somewhere obvious - I skimmed the site and your post before drafting this):
1. What is vendor lock-in like with Momentic? In my mind there's space for a mature QA platform that agentically generates and maintains playwright/similar tests, but perhaps I'm wrong.
1.b. will Momentic code or testing standard or some aspect be open enough that it is feasible to move away from it if needed, or continue using it if Momentic goes under/deploys major breaking changes?
2. How deterministic are Momentic tests?
3. Who is your target user? Product owner / developer / SDET / ?
----
Also - is this saying Codex is better and cheaper than Mo?
"... though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin."
1. Mo generates its own test cases in natural language, but you can ask it to write Playwright and maintain it in your repository if you really want. We actually do have a regression testing product as well where tests are just natural language in your repository. For that product yes there is some vendor lock-in since this is a proprietary format that only our CLI translates into browser actions. But ultimately since it's natural language it's pretty portable :)
2. Mo is focused on exploratory testing and so is inherently non-deterministic. But the regression testing product has a ton of caching built into it that we've written about in the past (not just a single css selector). https://momentic.ai/blog/teaching-browser-agents-user-intent This is a big part of reliability and latency optimization, otherwise your CI would just hang for hours on any reasonable suite.
3. I would say the regression testing product was very much SDET/developer focused since it required you to have a codebase and use Git. Mo is meant to be super easy for anyone to onboard onto and lower the barrier to entry, so we're hoping we'll see more product owners, PMs, QA and even leadership use the product. But devs are always going to be our bread and butter, we ultimately tried to build the product we wanted to use ourselves.
4. We're optimized for recall (# of bugs you find in a given amount of time). The bench shows we find ~3-5X more bugs compared to Codex and Claude Code, with approx. the same precision. The service costs more than the raw harnesses for sure, but that's the all-in cost including our infra, parallel agents, etc.
1/ General purpose models are still not good at this task. I would personally love it if I could trust my agent when it says its done after a 24h build. But whether it's because of sycophancy, context degradation over time, or something else coding agents still struggle to verify their own work. Maybe with 1B context windows this will change.
2/ Infra is still annoying to setup. We wanted to provide a unified interface across web, mobile and soon desktop as well. So users don't have to worry about provisioning, concurrency, security, etc.
3/ As with any SaaS, even beyond the initial build, maintenance is a headache. I think most companies would rather not consider agentic QA as one of their core competencies and investments. It takes a ton of eng effort to keep the lights on, continually update the models used, train new ones, groom ground truth, onboard infra providers, etc.
This strikes me as a company that is late delivering a product for November 2025, when it is November 2026.
To be more constructive: let's say I have 2 frontier subscriptions and I end up with 20% left on each at the end of the week. It feels awful knowing that I would _love_ to use that with Momentic but I can't because I have to pay for yet another set of proprietary non-transferable tokens. It's like dealing with airline rewards.
That doesn't mean we won't eventually support byok, especially as OSS models catch up! We have our finger on the trigger there.
how does this fit in to fixing the bugs? how do we close the loop with the agents we have coding?
Everything has to be reproducible and runnable by the general public