Too busy to read?

Perhaps your school allows you to select a foreign language: Do you want to install a software module into your brain that enables you to speak French, or one that enables you to speak Spanish? Do you want to learn to play tennis or chess? Do you want to study to become a chef, a lawyer or a pharmacist?

Throughout human history, we've relied on the same tried-and-true approach to keeping our technology beneficial: learning from mistakes. We invented fire, repeatedly messed up, and then invented the fire extinguisher, fire exit, fire alarm and fire department. We invented the automobile, repeatedly crashed, and then invented seat belts, air bags and self-driving cars. Up until now, our technologies have typically caused sufficiently few and limited accidents for their harm to be outweighed by their benefits.

intelligence = ability to accomplish complex goals

Once technology gets twice as powerful, it can often be used to design and build technology that's twice as powerful in turn, triggering repeated capability doubling in the spirit of Moore's law. The cost of information technology has now halved roughly every two years for about a century, enabling the information age.

boring pseudo-controversies caused by misunderstandings. For example, never waste time arguing about "life," "intelligence," or "consciousness" before ensuring that you and your protagonist are using these words to mean the same thing!

Above we considered the very real possibility that we're the only high-tech civilization in our Universe. Let's spend the rest of this chapter exploring this scenario, and the huge moral responsibility it entails. This means that after 13.8 billion years, life in our Universe has reached a fork in the road, facing a choice between flourishing throughout the cosmos or going extinct. If we don't keep improving our technology, the question isn't whether humanity will go extinct, but how. What will get us first--an asteroid, a supervolcano, the burning heat of the aging Sun, or some other calamity

This ability of Life 2.0 to design its software enables it to be much smarter than Life 1.0. High intelligence requires both lots of hardware (made of atoms) and lots of software (made of bits). The fact that most of our human hardware is added after birth (through growth) is useful, since our ultimate size isn't limited by the width of our mom's birth canal.

"the history of human technological civilization is measured in centuries--and it may be only one or two more centuries before humans are overtaken or transcended by inorganic intelligence, which will then persist, continuing to evolve, for billions of years. ... We would be most unlikely to 'catch' it in the brief sliver of time when it took organic form."

Bacteria, on the other hand, aren't great learners. Their DNA specifies not only the design of their hardware, such as sugar sensors and flagella, but also the design of their software. They never learn to swim toward sugar; instead, that algorithm was hard-coded into their DNA from the start. There was of course a learning process of sorts, but it didn't take place during the lifetime of that particular bacterium. Rather, it occurred during the preceding evolution of that species of bacteria, through a slow trial-and-error process spanning many generations, where natural selection favored those random DNA mutations that improved sugar consumption.

the story of nuclear radiation, nuclear EMP and nuclear winter all demonstrate that the greatest hazards may be ones we haven't even thought of yet. It's incredibly difficult to foresee all aspects of the aftermath, and how nuclear winter, infrastructure collapse, elevated mutation levels and desperate armed hordes might interact with other problems such as new pandemics, ecosystem collapse and effects we haven't yet imagined.

The memory in your brain works very differently from computer memory, not only in terms of how it's built, but also in terms of how it's used. Whereas you retrieve memories from a computer or hard drive by specifying where it's stored, you retrieve memories from your brain by specifying something about what is stored.

How would a conqueror AI eliminate us? Probably by a method that we wouldn't even understand, at least not until it was too late. Imagine a group of elephants 100,000 years ago discussing whether those recently evolved humans might one day use their intelligence to kill their entire species. "We don't threaten humans, so why would they kill us?" they might wonder. Would they ever guess that we would smuggle tusks across Earth and carve them into status symbols for sale, even though functionally superior plastic materials are much cheaper?

I made a New Year's resolution for 2014 that I was no longer allowed to complain about anything without putting some serious thought into what I could personally do about it,

the combined wealth of the bottom half of the world's population (over 3.6 billion people) is the same as that of the world's eight richest people

Charles Darwin elegantly explained why: since the most efficient copiers outcompete and dominate the others, before long any random life form you look at will be highly optimized for the goal of replication.

Hydrogen ..., given enough time, turns into people.

Intelligence is the ability to accomplish complex goals.

If animals are conscious, then what are predators supposed to eat?

The Dreaded Comparison: Human and Animal Slavery, Marjorie Spiegel argues that like human slaves, non-human animals are subjected to branding, restraints, beatings, auctions, the separation of offspring from their parents, and forced voyages. Moreover, despite the animal-rights movement, we keep treating our ever-smarter machines as slaves without a second thought, and talk of a robot-rights movement is met with chuckles. Why?

we saw how intergalactic travel becomes much easier once people and other intelligent entities can be transmitted in digital form, potentially making us masters of our own destiny not only in our Solar System or the Milky Way Galaxy, but also in the cosmos.

If we don't improve our technology, the question isn't whether humanity will go extinct, but merely how: will an asteroid, a supervolcano, the burning heat of the aging Sun or some other calamity get us first?

If we don't change direction soon, we'll end up where we're going. Irwin Corey

Life is a journey, not a destination. Ralph Waldo Emerson

when people ask about the meaning of life as if it were the job of our cosmos to give meaning to our existence, they're getting it backward: It's not our Universe giving meaning to conscious beings, but conscious beings giving meaning to our Universe.

Jobs can provide people with more than just money. Voltaire wrote in 1759 that "work keeps at bay three great evils: boredom, vice and need."

In the United States, with its high safety standards, motor vehicle accidents killed about 35,000 people last year--seven

Car accidents alone took over 1.2 million lives in 2015,

THE BOTTOM LINE: The current race toward AGI can end in a fascinatingly broad range of aftermath scenarios for upcoming millennia. Superintelligence can peacefully coexist with humans either because it's forced to (enslaved-god scenario) or because it's "friendly AI" that wants to (libertarian-utopia, protector-god, benevolent-dictator and zookeeper scenarios). Superintelligence can be prevented by an AI (gatekeeper scenario) or by humans (1984 scenario), by deliberately forgetting the technology (reversion scenario) or by lack of incentives to build it (egalitarian-utopia scenario). Humanity can go extinct and get replaced by AIs (conqueror and descendant scenarios) or by nothing (self-destruction scenario).