I routed everything through OpenRouter with a single API key, so request handling, timeout logic, and retry behavior were identical across models.
OpenRouter does direct forwarding without modifying the prompt payload. If it introduces any bias, it does so equally for all five, which preserves relative comparability.
TLDR: 42 attack types. 5 models. 3,360 tests. 1 in 3 harmful requests got through.
I could see this turning into a valuable third party resource, you can even monetize, for companies implementing agentic solutions. The industry needs independent third party voices.
Kudos.
The monetization angle is interesting. A continuously updated version with more models and frontier models, agentic scenarios, and multi-turn testing would be genuinely useful for teams making deployment decisions. That's the direction for v2.