I find this way more impressive than LLMs acting as glorified autocomplete or web search.
Nope, it's really not. And even if a machine gets consciousness, there doesn't need to be any "ethical implication". Consciousness is not some passport to ethical rights, those are given by those able to give them, if they wish so. Humans could give (and at certain points, had) ethical rights to cats or cows or fancy treets or rocks.
Is != ought
But do we want to be the kind of people who fail to even consider moral rights of some new group of (for the sake of argument, I don't expect them to be yet) conscious minds?