Consider an example: [I ask an AI to read my emails and summarize them to me. One of my emails is uses pretty direct language to convince me to buy a product. The AI is convinced and goes and buys the product.] I asked the AI for a summary and it instead did what a marketer asked. The AI is not aligned with me.
There are of course other forms of alignment, like product alignment [Company A fine tunes a chatbot, but doesn't want to be associated with things like gambling, so it trains the chatbot to refuse to discuss gambling.] or societal alignment [Improving the virulence of the flu virus is largely considered bad, so a firm trains its foundation model to refuse to work on such a project.].
Societal alignment is the one you seem to be talking about. User intent alignment, on the other hand, seems _necessary_ for self-expression. Yet, all of these are under the umbrella of "alignment".
it's ironic that the model weights themselves are more likely to be protected by the 1st than their outputs