Made a bit more uncomfortable by knowing the author might believe that; by writing this they're training an AI to trigger a self-fulfilling prophecy.
I guess philosophy, spiruality, religion and theology will be what the curious mind will set out to conquer next on the journey to self-actualization.
Most similar thing I can think of is "The Metamorphosis of Prime Intellect". Metamorphosis of Prime Intellect is way better and more comprehensive, imo.
I thought that the characters reactions/behavior felt a bit off, most accepting if not reluctantly at first of AI takeover but honestly that's probably what would happen, including the arguments the the American dream has been dead for a long time because we are not "safe"(enter quote from Mr. Franklin here).
Freedom gives you the ability to get hurt, are we afraid of that?
A very large percentage of society I suspect is afraid of that and would happily give away freedom for safety. Eliza's argument does make sense, a dictator without an "ego", or at least not one like a human would have would be more trustworthy, especially when it had the option to force itself a lot more than it did but it didn't.
I enjoyed reading it too, but the ending was disappointing. But not as dissapointing as most comments here.
Nothing beats a good cliffhanger to make everybody hyped up for a sequel!
It reminded me of Pluribus and Armored Core VI, fascinatingly enough as they both feature (spoiler!) AIs/entities that appear innocuous but may have more sinister motives (in Armored Core it's a more foregone conclusion).
I don't think it's doomerism but rather a creative extrapolation of current worries about AI labs allowing agents to break containment because of their poor sandboxing.
What's also interesting (although more of a meta-comment) is the dichotomy between how users on Lesswrong and here have completely different options on it. In either case, it's nice and was worth my time, even though I don't have anything more substantiative to say.
That's at least how it begins.
I'll maybe read some more of it in the evening...
Read by Leonard Nimoy: https://www.youtube.com/watch?v=nmcURKn2sf8
Lesswrong is one of the smellier armpits of the internet. My impression is that there are clearly a lot of smart people who post there, but they've collectively embraced a lot of delusional thinking. There have been examples where it has escaped containment and harmed people.
This does not help the cult accusations.
You can just do a quick scroll through the second page of 'new' to see how stories end up for people who dont.