Is Avoiding Extinction from AI Really an Urgent Priority?
📰 Fast.ai Blog
Experts question whether avoiding extinction from AI is an urgent priority, citing more pressing risks from human misuse of AI and other global issues
Action Steps
- Assess the likelihood and potential impact of AI-related risks
- Consider the role of human factors in AI risk, including negligent or malicious use
- Evaluate the urgency of AI extinction risk relative to other global priorities, such as pandemics, nuclear war, and climate change
- Develop a nuanced understanding of the complex interplay between AI, human agency, and societal risks
Who Needs to Know This
AI researchers, policymakers, and technologists can benefit from understanding the nuances of AI risk assessment and prioritization, as it informs their decisions on resource allocation and regulatory measures
Key Insight
💡 The greatest risks from AI may come not from the technology itself, but from the people who control it and use it to accumulate power and wealth
Share This
💡 Is avoiding AI extinction really a global priority? Experts weigh in on the complexities of AI risk assessment #AI #RiskAssessment
Key Takeaways
Experts question whether avoiding extinction from AI is an urgent priority, citing more pressing risks from human misuse of AI and other global issues
Full Article
# fast.ai - Is Avoiding Extinction from AI Really an Urgent Priority?
[fast.ai](https://www.fast.ai/index.html)
* [About](https://www.fast.ai/about.html)
* [](https://github.com/fastai)
* [](https://twitter.com/fastdotai)
* [](https://www.fast.ai/index.xml)
[](https://www.fast.ai/posts/2023-05-31-extinction.html "Toggle dark mode")
# Is Avoiding Extinction from AI Really an Urgent Priority?
The history of technology suggests that the greatest risks come not from the tech, but from the people who control it
ai-in-society
Author
Seth Lazar, Jeremy Howard, & Arvind Narayanan
Published
May 30, 2023
> _This article is the result of a collaboration between philosopher [Seth Lazar](http://www.sethlazar.xyz/), AI impacts researcher [Arvind Narayanan](https://www.cs.princeton.edu/~arvindn/), and fast.ai’s Jeremy Howard. At fast.ai we believe that planning for our future with AI is a complex topic and requires bringing together cross-disciplinary expertise._
This is the year extinction risk from AI went mainstream. It has featured in [leading publications](https://www.newyorker.com/science/annals-of-artificial-intelligence/can-we-stop-the-singularity?utm_medium=social&mbid=social_twitter&utm_brand=tny&utm_source=twitter&utm_social-type=owned), been invoked by 10 [Downing Street](https://www.theguardian.com/technology/2023/may/25/no-10-acknowledges-existential-risk-ai-first-time-rishi-sunak), and mentioned in a White House [AI Strategy document](https://www.whitehouse.gov/wp-content/uploads/2023/05/National-Artificial-Intelligence-Research-and-Development-Strategic-Plan-2023-Update.pdf). But a powerful group of AI technologists thinks it still isn’t being taken seriously enough. They have signed a [statement](https://www.safe.ai/statement-on-ai-risk) that claims: “_Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war_.”
“Global priorities” should be the most important, and urgent, problems that humanity faces. 2023 has seen a leap forward in AI capabilities, which undoubtedly brings new risks, including perhaps increasing the probability that some future AI system will go rogue and wipe out humanity. But we are not convinced that mitigating this risk is a global priority. Other AI risks are as important, and are much more urgent.
Start with the focus on risks _from AI_. This is an ambiguous phrase, but it implies an autonomous [rogue agent](https://yoshuabengio.org/2023/05/22/how-rogue-ais-may-arise/). What about risks posed by _people_ who negligently, recklessly, or maliciously use AI systems? Whatever harms we are concerned might be possible from a rogue AI will be far more likely at a much earlier stage as a result of a “rogue human” with AI’s assistance.
Indeed, focusing on this particular threat might exacerbate the more likely risks. The history of technology to date suggests that the greatest risks come not from technology itself, but from the people who control the technology using it to accumulate power and wealth. The AI industry leaders who have signed this statement are precisely the people best positioned to do just that. And in calling for regulations to address the risks of future rogue AI systems, they have proposed interventions that would further cement their [power](https://www.businessinsider.com/sam-altman-openai-chatgpt-government-agency-should-license-ai-work-2023-5). We should be wary of Prometheans who want to both profit from bringing the people fire, and be trusted as the firefighters.
And why focus on _extinction_ in particular? Bad as it would be, as the preamble to the statement notes AI poses other serious societal-scale risks. And global priorities should be not only important, but urgent. We’re still in the middle of a global pandemic, and Russian aggression in Ukraine has made nuclear war an imminent threat. Catastrophic climate change, not mentioned in the statement, has very likely already
[fast.ai](https://www.fast.ai/index.html)
* [About](https://www.fast.ai/about.html)
* [](https://github.com/fastai)
* [](https://twitter.com/fastdotai)
* [](https://www.fast.ai/index.xml)
[](https://www.fast.ai/posts/2023-05-31-extinction.html "Toggle dark mode")
# Is Avoiding Extinction from AI Really an Urgent Priority?
The history of technology suggests that the greatest risks come not from the tech, but from the people who control it
ai-in-society
Author
Seth Lazar, Jeremy Howard, & Arvind Narayanan
Published
May 30, 2023
> _This article is the result of a collaboration between philosopher [Seth Lazar](http://www.sethlazar.xyz/), AI impacts researcher [Arvind Narayanan](https://www.cs.princeton.edu/~arvindn/), and fast.ai’s Jeremy Howard. At fast.ai we believe that planning for our future with AI is a complex topic and requires bringing together cross-disciplinary expertise._
This is the year extinction risk from AI went mainstream. It has featured in [leading publications](https://www.newyorker.com/science/annals-of-artificial-intelligence/can-we-stop-the-singularity?utm_medium=social&mbid=social_twitter&utm_brand=tny&utm_source=twitter&utm_social-type=owned), been invoked by 10 [Downing Street](https://www.theguardian.com/technology/2023/may/25/no-10-acknowledges-existential-risk-ai-first-time-rishi-sunak), and mentioned in a White House [AI Strategy document](https://www.whitehouse.gov/wp-content/uploads/2023/05/National-Artificial-Intelligence-Research-and-Development-Strategic-Plan-2023-Update.pdf). But a powerful group of AI technologists thinks it still isn’t being taken seriously enough. They have signed a [statement](https://www.safe.ai/statement-on-ai-risk) that claims: “_Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war_.”
“Global priorities” should be the most important, and urgent, problems that humanity faces. 2023 has seen a leap forward in AI capabilities, which undoubtedly brings new risks, including perhaps increasing the probability that some future AI system will go rogue and wipe out humanity. But we are not convinced that mitigating this risk is a global priority. Other AI risks are as important, and are much more urgent.
Start with the focus on risks _from AI_. This is an ambiguous phrase, but it implies an autonomous [rogue agent](https://yoshuabengio.org/2023/05/22/how-rogue-ais-may-arise/). What about risks posed by _people_ who negligently, recklessly, or maliciously use AI systems? Whatever harms we are concerned might be possible from a rogue AI will be far more likely at a much earlier stage as a result of a “rogue human” with AI’s assistance.
Indeed, focusing on this particular threat might exacerbate the more likely risks. The history of technology to date suggests that the greatest risks come not from technology itself, but from the people who control the technology using it to accumulate power and wealth. The AI industry leaders who have signed this statement are precisely the people best positioned to do just that. And in calling for regulations to address the risks of future rogue AI systems, they have proposed interventions that would further cement their [power](https://www.businessinsider.com/sam-altman-openai-chatgpt-government-agency-should-license-ai-work-2023-5). We should be wary of Prometheans who want to both profit from bringing the people fire, and be trusted as the firefighters.
And why focus on _extinction_ in particular? Bad as it would be, as the preamble to the statement notes AI poses other serious societal-scale risks. And global priorities should be not only important, but urgent. We’re still in the middle of a global pandemic, and Russian aggression in Ukraine has made nuclear war an imminent threat. Catastrophic climate change, not mentioned in the statement, has very likely already
DeepCamp AI