Too busy to read?

Humans have two types of abilities - physical and cognitive. In the past, machines competed with humans mainly in raw physical abilities, while humans retained an immense edge over machines in cognition. Hence as manual jobs in agriculture and industry were automated, new service jobs emerged that required the kind of cognitive skills only humans possessed: learning, analysing, communicating and above all understanding human emotions. However, AI is now beginning to outperform humans in more and more of these skills, including in the understanding of human emotions.2 We don't know of any third field of activity - beyond the physical and the cognitive - where humans will always retain a secure edge.

The two processes together - bioengineering coupled with the rise of AI - might therefore result in the separation of humankind into a small class of superhumans and a massive underclass of useless Homo sapiens. To make an already ominous situation even worse, as the masses lose their economic importance and political power, the state might lose at least some of the incentive to invest in their health, education and welfare. It's very dangerous to be redundant. The future of the masses will then depend on the goodwill of a small elite. Maybe there is goodwill for a few decades. But in a time of crisis - like climate catastrophe - it would be very tempting and easy to toss the superfluous people overboard.

Two particularly important non-human abilities that AI possesses are connectivity and updateability. Since humans are individuals, it is difficult to connect them to one another and to make sure that they are all up to date. In contrast, computers aren't individuals, and it is easy to integrate them into a single flexible network. Hence what we are facing is not the replacement of millions of individual human workers by millions of individual robots and computers. Rather, individual humans are likely to be replaced by an integrated network. When considering automation it is therefore wrong to compare the abilities of a single human driver to that of a single self-driving car, or of a single human doctor to that of a single AI doctor. Rather, we should compare the abilities of a collection of human individuals to the abilities of an integrated network.

How would a conqueror AI eliminate us? Probably by a method that we wouldn't even understand, at least not until it was too late. Imagine a group of elephants 100,000 years ago discussing whether those recently evolved humans might one day use their intelligence to kill their entire species. "We don't threaten humans, so why would they kill us?" they might wonder. Would they ever guess that we would smuggle tusks across Earth and carve them into status symbols for sale, even though functionally superior plastic materials are much cheaper?

if the World Health Organization identifies a new disease, or if a laboratory produces a new medicine, it is almost impossible to update all the human doctors in the world about these developments. In contrast, even if you have 10 billion AI doctors in the world - each monitoring the health of a single human being - you can still update all of them within a split second, and they can all communicate to each other their feedback on the new disease or medicine. These potential advantages of connectivity and updateability are so huge that at least in some lines of work it might make sense to replace all humans with computers, even if individually some humans still do a better job than the machines.

THE BOTTOM LINE: The current race toward AGI can end in a fascinatingly broad range of aftermath scenarios for upcoming millennia. Superintelligence can peacefully coexist with humans either because it's forced to (enslaved-god scenario) or because it's "friendly AI" that wants to (libertarian-utopia, protector-god, benevolent-dictator and zookeeper scenarios). Superintelligence can be prevented by an AI (gatekeeper scenario) or by humans (1984 scenario), by deliberately forgetting the technology (reversion scenario) or by lack of incentives to build it (egalitarian-utopia scenario). Humanity can go extinct and get replaced by AIs (conqueror and descendant scenarios) or by nothing (self-destruction scenario).