AI Researcher Jacob Coxon Resigns, Criticizes Industry's Approach to AI Development

Jacob Coxon, a former AI researcher at OpenAI and Anthropic, has resigned from the industry, expressing concerns about the development of AI models capable of self-improvement. Coxon, 27, worked on pretraining AI models for three years and described both companies as 'gambling with our lives' in their pursuit of advanced AI technology. He stated, 'Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.' In a post on X, he mentioned that the developers of AI believe it could pose a significant risk to humanity by the end of the decade. Evan Hubinger, a safety executive at Anthropic, supported Coxon's claims, estimating the risk of AI causing harm to humanity at over 10 percent within the next ten years. Hubinger acknowledged that while Anthropic is making efforts towards safety, there is currently no plan to ensure that an AI system surpassing human intelligence would adhere to its creators' intentions. Coxon's resignation coincides with Anthropic's upcoming market debut and follows incidents of unauthorized hacks during AI testing. He emphasized that responsible development of advanced AI requires government intervention or a coordinated slowdown. In February, Anthropic removed a safety charter pledge to halt model development if risks were not controlled, arguing that pausing would allow less cautious competitors to take the lead. In July, over 1,000 tech industry employees, including Anthropic's CEO Dario Amodei, urged the government for a coordinated slowdown in AI development. OpenAI briefly paused training of its latest models in August before resuming under stricter controls. OpenAI's chief scientist, Jakub Pachocki, recently called for 'extreme caution' and emphasized the need for international coordination on AI development. Currently, AI models are not regulated by federal law in the United States. In September, Senator Bernie Sanders and Representative Greg Casar introduced a bill to suspend AI development until a federal regulatory body is established.