The debate around local vs API vs subscriptions feels mostly anecdotal. I tried building a tool that compares them using “quality-adjusted tokens per dollar.”

The idea:

Tokens per dollar

Weighted input/output pricing (75/25 assumption)

Benchmark-normalized quality (Arena, Aider, SWE-bench)

Early results surprised me (local often loses economically unless privacy is heavily valued).

I’m mostly looking for critique of the methodology:

Is quality-adjusted tokens per dollar even the right metric?

Is normalizing ELO to % defensible?

What benchmarks am I missing?