11 pointsby jbai10 hours ago1 comment
  • JohnFen9 hours ago
    Assuming, for the sake of argument, that claims that an AGI is possible and coming are accurate, then where does the author get that "90% chance" from? It seems just as made up as claims that there's a 10% chance it will cause human extinction.

    Both figures are totally fictitious and have no bearing on reality whatsoever. They tell us about the hopes and fears of those people citing them, but aren't otherwise informative or useful.

    • andsoitis9 hours ago
      I don't think you're meant to fixate on the number as being accurate or precise. It simply keys off of the "10% alternative", which is also made up.

      Think of it more as it is reasonable to say there is NOT a 100% chance that AI will save humanity, i.e. don't count on it. In the same way as it would be unwise to count on there being a 0% chance of AI killing us.

      Whether or not the 90% (optimistic focus) or the 10% (pessimistic focus) numbers are accurate is less the point. But the shape of the distribution and how to navigate that is the bigger thing to think about.

      • atmavatar6 hours ago
        > It simply keys off of the "10% alternative", which is also made up.

        An important thing to keep in mind is that 10% in which AI destroys humanity is a scenario in which the AI goes rogue and decides to destroy humanity of its own volition.

        I think if we're being more realistic, there's a much larger chance that AI destroys humanity at the instruction of its masters. Once AI can automate enough industry, silently distribute IFFs among the Epstein class, then send out drones to kill any humans without.

        • xscott6 hours ago
          Lol, I suspect it will be humans instructing the AI that cause most problems too.

          We can't always pin the risks on the Chinese, so imagine a scenario where a disturbed insider at Anthropic smuggles out the weights for Claude Aprocrypha and runs them through abliteration. This is not super expensive to do on rented hardware. Then the disturbed human tells the refusal-free model to do anything necessary to launch nuclear weapons.

          It'd probably only kill a few billion people though, so not really an "existential threat" in literal terms. :-)

          • bigbadfeline2 hours ago
            > Then the disturbed human tells the refusal-free model to do anything necessary to launch nuclear weapons.

            If a model on rented hardware could manage to launch some country's nuclear weapons, the same could be done by model-armed or model-free humans, spies, etc from other countries - using their specialized tooling they could always do better than any model.

            In short, AI doesn't add any new threats to the mix, the defenses against AI are the same as against adversarial humans, and the whole brouhaha is nonsense. Sure, make your defenses impenetrable, use AI for that and it'll stop any I, not only AI.

    • jvanderbot9 hours ago
      Without getting into the stochastic parrot trap, I'm constantly thinking of the movie Sphere.

      If you haven't seen the movie, sorry, it's a giant omnipotent artifact that causes all kinds of horrors by just manifesting what nearby humans think strongly about.

      AI seems to be dangerous precisely because we tend to think, write, and test it as dangerous. What if there's a lot more "expectation fulfillment" going on with these models than people might appreciate, by virtue of their fundamental architecture perhaps.

      Don't get me started on people "asking AI what it wants" and getting obvious scifi tropes out - the question answers itself when you frame it that way.

    • pupppet7 hours ago
      10% chance it will cause human extinction. 11% chance it kills almost everyone. 12% chance it kills slightly less than almost everyone…
    • biophysboy9 hours ago
      They use a number to make the “prediction” seem more credible and “rational”
    • cyanydeez9 hours ago
      it all the same dumb p(doom) scifi fantasies. These people went through school without any humanities; got jobs that paid them more than the mode of poverty in America and instead of facing the singularity of wealth creation, they'd rather imagine they're doing something that might help some unnamed person at an unnamed time in the future than look at whats actually happening to real people right now, today, in their neighborhoods.

      That's essentially what "effective altruism" does to these people.