Superintelligence: The Idea That Eats Smart People

While I agree with Maciej's central point, I think the inside arguments he presents are pretty weak. I think that AI risk is not a pressing concern even if you grant the AI risk crowd's assumptions. Elided from https://alexcbecker.net/blog.html#against-ai-risk: The real AI risk isn't an all-powerful savant which misinterprets a command to "make everyone on Earth happy" and destroys the Earth. It's a military AI that correctly interprets a command to kill a particular group of people, so effectively that…

What really terrified me wasn't skynet or Hal, but what Charles Stross describes in his novel Accelerando. Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with. It terrified me because there is efficiency incentive for the buying/selling of goods and services to be realtime and computer based. Not just stock markets but also e-comm, smart contracts…

A lot of people seem to miss this point, so I'll reiterate it. I wrote this talk shortly after the book Superintelligence came out. The first half of this talk presents the strongest case I could make for a "fast takeoff" AI scenario à la Bostrom, while the rest of the talk lays out why I think this argument is fallacious. Please limit your dunking on me to the material in that latter half of the talk. As for how/whether…

> If you encountered a cheetah in pre-industrial times (and survived the meeting), you might think it was impossible for anything to go faster. Fun fact, there is no historical evidence of an adult human ever dying from a cheetah attack. They are naturally shy, and a lot smaller than you may realize.