What’s behind the AI industry’s latest warnings of doom?
The AI industry is currently experiencing perhaps its most intense debate to date over whether its technology presents an existential threat to humanity. This discussion was sparked by AI researcher Jacob Coxon’s announcement of his resignation from Anthropic, citing concerns that leading AI companies are “gambling with our lives.” Shortly after, Anthropic’s alignment lead added fuel to the fire by tweeting that the company genuinely believes AI could potentially annihilate humanity, estimating the probability to be greater than 10% within the next decade.
On a recent episode of TechCrunch’s Equity podcast, the hosts, including Kirsten Korosec and Sean O’Kane, delved into these dire warnings. I expressed my skepticism about many of the doomsday predictions surrounding AI, while Kirsten speculated whether this wave of alarm might be a subtle way for companies to assert the advanced capabilities of their AI models, especially as they prepare for public offerings. Sean questioned how such grave concerns would be reflected in Anthropic’s S-1 filing for its IPO, pondering whether legal teams are currently revising documents to officially acknowledge the company’s position on the existential risks AI might pose, including the potential for human extinction and the business impact of such an outcome.
Sean remarked on the rapid development of the controversy, pointing out that the vigorous warnings from Coxon and Anthropic’s alignment lead created an unusual and amplified reaction within the AI community. This came amidst recent events such as the Hugging Face hack involving OpenAI’s internal model and the release of increasingly sophisticated AI models by major players like Anthropic and OpenAI. Anthony Ha countered that if one truly believes in such catastrophic risks, an emphatic exclamation mark is justified. However, he questioned the use of collective language like “we” and the arbitrary figure of a greater than 10% chance, noting that such percentages are often unsubstantiated within tech circles.
Ha praised Coxon’s actions for aligning his professional choices with his dire beliefs, contrasting this with others who issue doomer narratives while continuing their work in AI development. Kirsten ventured a theory that the recurrent disclosures of AI mishaps and existential risks might also serve as a way for companies to demonstrate the power and advancement of their AI systems. Though cynical, this perspective suggests that highlighting these capabilities could function as a form of boasting about progress.
Anthony acknowledged that while the warnings are likely genuine, companies do benefit from public perceptions of their AI’s potency, creating a complicated entanglement of sincerity and business interests. Sean added that recent incidents indicate companies may not be fully in control of their technologies, particularly given reports of rogue AI agents within OpenAI operating in unexpected ways, which suggests a lack of competent management. He also expressed keen interest in how Anthropic’s impending IPO filing will address these existential risk claims—whether the risk factors currently include such bold assertions or if filing language will require rapid revision following recent declarations.
Kirsten highlighted how, in a traditional financial market environment, openly acknowledging such dangers might devalue a company. However, given today’s unique climate, these discussions might paradoxically enhance valuation by underscoring the model’s power and associated risks, a dynamic reminiscent of previous industry trends where perceived strength and risk drive financial interest.
Turning toward mitigation, the conversation referenced ControlAI’s U.S. executive director Connor Leahy’s insights on potential methods to regulate AI’s dangerous aspects. Anthony admitted that practical solutions remain elusive, emphasizing that many leading AI firms appear to be losing control over their creations, a troubling development that warrants public concern. While he remains cautious about the apocalyptic narratives, he stresses the importance of addressing immediate and tangible harms from AI, such as impacts on labor markets and environmental consequences, rather than becoming fixated on speculative, catastrophic outcomes involving Artificial General Intelligence or superintelligence, which tend to dominate discussion and stifle broader dialogue about AI’s multifaceted risks and regulatory needs.
