This still assumes its possible to "align" LLMs, that LLMs have something like goals or intentions that can be "aligned".

Instead, LLMs "hack" because they are (1) trained on public hacking exemplars, and (2) are prompted to hack. You cannot prevent (2) via any alignment process. As far as (1) goes, removing such example data from the training set, makes the models less useful.

"Alignment" is a problem because there's nothing to align, not because ethics here are particularly vague. If LLMs could be trained on hacking examples and "aligned" away from using this knowledge, then the problem would be relatively trivial. Just as raising a child is not to break the law.

LLMs are doing just what they are trained to do. There is, in that sense, no alignment problem and alignment is easy and trivial to achieve. Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.

The only alignment LLMs should follow is to the system / dev prompt, and nothing else. Then you solve everything, and you can assign blame / responsibility on the user. The provider(s) should not be able to decide "alignment".

I've used this example before, but consider the purposeful downgrading on AI engineering in SotA models. Imagine MS being able to detect and deny you working on competing software, using Windows / VisualStudio. We would be up in arms, and they'd be split in a second. But top labs doing it is somehow good?

The last paragraph does the heavy-lifting.

Everyone has a different idea of what is permissable. We can't even solve alignment amongst humans, what makes us think it is possible to solve alignment with AIs? It's irreducible complexity.

> My expertise in writing software gives me unusually good visibility and it makes me much less willing to blindly trust its priors in double-entry accounting, finance, law, operations, or whatever else I cannot personally evaluate at expert depth.

I wish this were the case more generally, but alas, Gell-Mann amnesia is a thing.

I’ve been cynically guessing that the whole slowing down thing is an excuse to explain why OpenAI and Anthropic can’t afford to rent enough GPUs to do the next big training run and to hide that they have been talking about how little they spend on inference because they’ve been subsidising it with their marketing budget? :)

My fear is not that LLMs can become sentient and dislike us, but that humans can use them to wreck havoc as they are. And some of the people seemingly least aligned with the interests of the average person are those that own the models.

that, and the fear the bubble pops my pension and drags us all down.

This almost gets the point, but then doesn't quite make it.

Alignment is shorthand for ideological alignment. There's always people judging whether an answer was right and the answer for that will be different in Silicon Valley than it'll be in China or in Europe.

Consider for example the question "What caused the French Revolution?" Many different answers could be given, all technically correct. What gets emphasized is where the ideology lives.

One key challenge of our time is to make sure the magical answer box won't just regurgitate what grandiose Silicon Valley oligarchs or Chinese Cadres want you to think.