I recently built KillSwitch[1], a brand new esoteric programming language which includes features designed to confuse/mislead/break LLMs, including:

    * certain constructs designed to trigger safety guardrails in models

    * counting of letters in identifiers as a language feature, hopefully triggering model's inability to accurately do so

    * the language spec containing hidden meaning and lies (designed to be invisible for humans but not LLMs)

I was curious how well models would do on a brand new language that they haven't seen before and decided to make it even harder for them by creating a language that is specifically designed against them.

The benchmark suite I built for it ends up creating a pretty nice leaderboard for models. The standout results are that Opus 5 beats 5.5 and that MiMo V2.6 Pro is the best performing open-weight model. Though none of the models get everything right (and this is for a very simple set of tasks, like hello-world).

I'd be interested to hear if you have thoughts on language features that might break LLMs even more.

1 - https://killswitch-lang.org