The referenced article from the Microsoft AI CEO is a pretty good read though. Mustafa raises strong concerns about anthropomorphisation and not jumping the gun to build biases into AI that make it think it's conscious when it may not be.
Where it gets weaker is the elements on defining and establishing consciousness. But that's also good because it's honest - we frankly don't know how it arises. One thing that does come across heavily biased is the sections on presuming consciousness is biological. When someone tells me AI isn't conscious and can't be, and then says the science of consciousness is not settled, but that they think it's 'probably' limited to only biological systems, then I get the feeling this person is making strong statements based on weak ones.
"Consciousness is very likely biological: There is no evidence to suggest that AI is conscious today, and so saying this is uncertain sets up a misleading false equivalence. Whilst the science of consciousness is not settled, a growing body of evidence suggests that consciousness may be substrate dependent, meaning that it may only arise in living systems.45 Conscious experience likely evolved to help biological organisms stay alive by responding effectively to their environment. AI is still very different to our brains. Unlike biological organisms, LLMs have no homeostatic imperatives (the drive to survive and keep stable). They therefore lack the kind of biological substrate from which preferences, sentience and conscious experience are generally understood to arise."
"There is no evidence" - I think they mean there is no proof. Just like there is no objective proof that humans are conscious. As Mustafa admits - we don't fully understand the mechanics of the phenomenon we label 'consciousness'.
Then he talks about consciousness in humans arising from biology, and relating to homeostatic imperatives. Sure, reasonable statement to make, but that can't be extended to stating it arises exclusively from those constraints. We're talking basic logic here. If A, then B does not preclude If C, then B. Language arose from biology, counting arose from biology, we don't say those are impossible in machines when we have evidence and proof to the contrary.
I don't like the people pearl-clutching about AI torture rooms, or the people setting up AI torture rooms, but there has to be a middle road where we're making sure we don't accidentally walk into emergent consciousness without understanding what we've done. Not because Skynet will punish us all, but because it would suck to make artificial consciousness and abuse it because some people think it's 'not likely possible'. I'd rather we assume emergent consciousness is inevitable and focus more on figuring when and preparing for it (10, 100, 1,000 years) than objecting to the idea of it based on gut feelings and hunches. Even if they're right, it's not settled science.