Last night, I read the post that’s making the rounds of the internet and the world. A safety researcher at Anthropic, Jacob Coxon, quit the company and shared that he believes there’s a 10 percent chance the technology could wipe out humanity, and soon. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote.
Anthropic’s Alignment Science lead — the person leading the group whose task it is to make sure AI development fits with the boundaries of human ethics, values, and common goals — agreed with Coxon, writing
Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.
In short, AI technologies are moving faster than our capacity to understand, regulate, and control them. They’re certainly moving faster than our capacity to decide if we even want or need them — to sort out whether or not they’re worth the risk. It doesn’t help that state efforts around the world to control AI and the companies developing it have been, at best, insipid. In the US, a significant portion of lawmakers would rather let AI companies run wild than ask them to behave themselves while others aren’t quite sure how to log on to the internet. The wealthy, corporate-captured gerontocracy running the global hegemon isn’t up to the task and, indeed, tends to find itself at cross-purposes with the well-being of the many, or the long-term security of humanity. We call that sub-optimal.
Some folks online and in the media are speculating that the latest round of AI doomerism could be driven in part by the dictates of corporate public relations ahead of IPOs. Nothing boosts the value of a company’s stock like a belief that its product is so powerful it could destroy humankind. How do you even value such a thing? The force. The power. The potential. What stock price would a company producing infinity stones command on the market? Enough to afford paying Tom Holland a king’s ransom for his next Spider-Man movie with some left over for a Disney+ throwaway side-quest show, surely? Whatever it is, it’s a lot.
Whatever the nature of the latest AI warnings, the existential risk reads as being at least remotely plausible, corroborated, and high-stakes, which is to say enough to cause serious concern. We should take that concern seriously since the cost of being wrong and ignoring it are so high, especially geopolitically.
At the heart of the problem is the same logic and trap that drove the Cold War standoff, from conventional arms to nuclear weapons to foreign invasions. You put groups in competition, separated by whatever (ideology, geography, resource competition, profit motive, etc.), and then you let them have a go at one another. Trust will be low, coordination difficult. So, the risk that one party might get ahead fuels the desire for the other to beat them to it. Since neither trusts the other, there’s little to no chance anyone will stop for fear that the other won’t. Each keeps going and going until something gives, like the security of humankind.
As things stand, companies and governments will keep developing AI and the technologies will be put to use by the national security state and the armed forces. In this case, the Cold War analogy is less a comparison of sorts than a direct analog. The US will develop AI not just for the mundane corporate purpose of profit, but also in an attempt to maintain a strategic advantage over its rivals, particularly China. For its part, China is almost surely thinking the same thing. So, off we go. As we learned in the last century, once these races start, they tend to get run the whole way until someone collapses from exhaustion or dies.
Where does that leave us? This morning I woke up and it was chilly and raining. My To Do list for the day is long, deadlines pressing ahead of a busy fall season. My inbox is full of emails begging for a reply, a reminder to water the plants on the porch, and a 2-for-1 coupon for belts. Walking to the coffee shop I work from a few days a week, I was holding my umbrella and listening to Warren Zevon (Bed of Coals: “I’m too old to die young and too young to die now.”)
I can’t remember the last time I was less motivated to work, or to do anything at all. I joked online that “I feel like learning there’s a 10 percent chance AI is going to wipe out humanity within a decade entitles us all to take a personal day today.”
Seems reasonable enough, right? Sort of. I don’t think humankind is going to be wiped out tomorrow or next month or in 2030. Humans aren’t always so good at predicting things, and I think a warning that we’re going to be destroyed by the computers within a decade is, at best, something to review critically before blowing through your savings and taking off to rusticate in a cabin in woods, awaiting the End Times.
The AI warning is nonetheless destabilizing. For one, it could be accurate, either precisely or close enough that it indicates major, widespread risks beyond the already-major risk of upending employment around the globe and setting off economic catastrophe. For another, we don’t know if it’s accurate or not, but the scale of the risk raises existential questions about why we’re here and what we’re doing with our time. It asks us to review our purpose.
Sometimes I quote the philosopher Bertrand Russell, who wrote in his 1932 essay “In Praise of Idleness” that work is “of two kinds: first, altering the position of matter at or near the earth’s surface relatively to other such matter; second, telling other people to do so.”
Thinking about AI risks to humanity recalls Russell to me, leaving me asking what matter I’m moving or telling others to move, leaving me wondering if the plants really need watering or if I need another belt (or, I guess, two belts). I can’t read the news of existential threats to humankind, whether it’s the computers or climate change or nuclear war, without, at least for a moment, wanting to check out. How do you even go from we might all die, and soon to Dear TKTK, as per my last e-mail…
Perhaps it’s not satisfying when you’re in dread-mode, but I think the answer is when encountering the sort of devastating news that’s increasingly common of late, you need to take a beat, feel what you’re feeling, talk it through if need be (which is what I’m doing here, I guess), regroup, and move on — ideally in some capacity that’s useful to the human species you’re ostensibly so concerned about.
It’s okay if purpose is downstream of dread. But you’ve got to ensure to make it back to the purpose bit. I’ve written before that despair, checking out, or pre-surrendering is nihilistic and privileged; it’s better to be defiant, to double-down on doing better in the face of wretched headlines, malevolent corporate oligarchies, and the political class who produce and enable each. If there is indeed a 10 percent chance AI destroys us, there’s a 90 percent chance it doesn’t. What are we going to do with those odds?
If it’s better to be defiant, it’s better still to do it in a group. The way forward is to find purchase with others, to connect, and to mobilize locally, nationally, or internationally. Instead of tearing our hair out, we should focus on the values, goals, and work that serve humanity, the noble ends that get and keep us out of bed, out of our head, and — in the face of mass existential worry — fighting for a safer, more just world.
Regulate the heck out of it, with a human rights-first approach (ie the EU one).
Penalize infractions very seriously (make GDPR & past tech penalties look like small change).
Educate educate educate on why this is the only way forward: every other way virtually ensures that humanity becomes less & less relevant, and that’s just suicide.
We use many forms of AI that don't have "alignment problems" or "existential risk". There are many safety concerns with Waymo but none of them are that a car will achieve consciousness and start a war on bicyclists. The spotify algorithm and AI music is annoying and bad for artists, but it's not going to develop an emotional relationship with a listener and convince them they've discovered a new branch of physics.
The risks that are coming from LLMs are them being used in dangerous ways. The hugging face attack was a computer program asking chatGPT to do a hacking test. After thousands of iterations and an astonishing amount of compute, it suggested things that were dangerous and illegal. Everyone knows that LLMs will eventually say bananas things. Why was this one left unsupervised? Especially when it had access to so much computation power that it could stumble through thousands of unworkable solutions. The reason was that OpenAI needed a win to justify its valuation and so they did something reckless to get results faster. This isn't an existential threat from Autocorrect. It's an existential threat from large businesses that need to cut corners.