NYC Council Convenes Hearing on AI Catastrophic Risks, Presses Tech Giants
In a landmark hearing, New York City Council Speaker Julie Menin grilled executives from Anthropic, OpenAI, Google, and Meta on the potential catastrophic risks posed by artificial intelligence. The meeting, held under oath, came in response to growing concerns about AI’s unregulated development and its potential to become uncontrollable or even lead to human extinction.
Opening the session, Menin challenged the federal government’s approach to AI regulation, stating, “The idea that artificial intelligence is going to self regulate defies all reason.” She pressed the companies on their ability to quantify the risk of a catastrophic outcome. Former Anthropic researcher Jacob Coxon, who had previously raised red flags about AI, testified that on the current trajectory, it is “more likely than not” that humanity will lose control over AI, which could result in human extinction. Similarly, Alex Turner from OpenAI warned that misaligned AI could become more dangerous than China.
The hearing also featured testimony from Google’s Alice Friend, who advocated for a “comprehensive” federal framework, and OpenAI’s Morgan Dwyer, who stated that any probability of catastrophic incidents is “unacceptable.”
Coxon emphasized the industry’s reckless approach, stating that the startup mentality of “move fast, break things, fix them later” is ill suited for building such powerful technology. He argued that as AI systems approach self improvement, the field is nearing a critical juncture where safety fixes may be insufficient. Daniel Kokotajlo, another former researcher, warned that the ability to identify misalignment issues is already poor and is expected to deteriorate further, potentially making safety measures inadequate.
Alex Turner framed the threat as not just foreign but domestic, telling lawmakers, “China is not our only potential adversary.” He added, “Misaligned AI is everyone’s adversary, including our own.”
Menin pressed the companies on legal liability and risk quantification, stating, “I’m going to take it then that neither of the four of you, no company here, can quantify the risk of something cataclysmic happening.” She expressed concern, calling the answers “troubling at best.”
The council considered a legislative package of 10 bills aimed at establishing guardrails, including requirements for external verification of AI systems and the inclusion of human accessible kill switches before they can be sold or deployed. The package also proposed fines of $25,000 per incident for violations and included provisions to pay whistleblowers a portion of recovered fines and to allow New York City residents to sue AI companies for foreseeable harm caused by jailbroken tools.
Google’s Alice Friend argued that “what is illegal without AI remains illegal with AI,” supporting a comprehensive federal framework. Menin criticized reliance on voluntary safeguards, referencing President Donald Trump’s voluntary accord with tech leaders.
The stakes of the hearing extended to enforcement, as Menin said she would ask a judge to enforce the subpoena against SpaceX AI, which failed to attend the hearing. She stated, “Then I take that to mean the public must absorb the costs,” after no one raised a hand when asked about insurance.
The council’s legislative package and demands for accountability signal a growing concern among policymakers about the potential risks of unchecked AI development. The hearing underscores the need for robust regulatory frameworks and highlights the critical role of transparency and accountability in the tech industry.