It'll be interesting to see if we get better / worse at figuring out what effective LLM interaction should look like, since that seems prerequisite to this kind of evaluation. Right now there's so little data on whether e.g. casual conversational interaction works better than structured prompting, and so, so, so much superstition - I don't know if I'd really trust judges to have good instincts about what's effective rather than what resembles their own use patterns.

I would like to preemptively acknowledge that this is not the center of the article, but:

> Did you know IQ tests are effectively illegal for hiring in the US because they disproportionately screen out minorities? I learn things when I research these posts.

The provided link discusses how this is itself a common misunderstanding and they're not illegal if they have to do with the job, but you have to justify that they do have to do with the job.

> Using fully automated scoring would also be against the law in some places: New York City requires annual bias audits of automated hiring tools, and you don’t want that heat.

The elision of the gap between "[it's illegal to do this]" and "[it's illegal to do this without checking for bias]" is interesting.

Why hire juniors? There seem to be plenty of unemployed mid- and senior-level people and the market is so poor that some of them (e.g. me) will work for cheap.

Here is the problem with almost all of these attempts: there is no need to be a “pop quiz” where you have the answer, and the candidate is made to guess what you feel is right (even if you're wrong) and is walking in blind. This literally never happens in the real world - and if it does it means you’re getting fired. It’s just sadism you are pushing to make the in group people feel smarty pants and superior.

Tell the candidate up front what you want to talk about. Do it on a subject you dont know either so you are on even footing. Let them do research before the “quiz”. Since tokens are pretty cheap, ask them to build a working application and then discuss it with them.

This really isn't hard if you get off your imaginary high horse, and show candidates some basic respect.