AI Companies Propose Embedded Evaluators for Safety Monitoring

<p>Dario Amodei, Sam Altman, and Elon Musk, leaders of AI companies Anthropic, OpenAI, and xAI, respectively, have reached a consensus on the need for "embedded evaluators"—external experts to monitor safety practices within AI firms. This agreement comes amid heightened concerns about AI safety in recent weeks. Both Amodei and Altman have committed to integrating these evaluations, while Musk expressed support for the initiative.</p><p>The specifics of how this evaluation system would operate are not yet clear. Evaluators would be assigned to work within the companies, using company resources and signing non-disclosure agreements. Their responsibilities could include assessing risks related to unreleased AI models and auditing safety practices. Amodei has indicated that evaluators would have editorial independence, although there would be restrictions on sharing certain sensitive information.</p><p>Experts for these evaluations may come from both small nonprofits and large consulting firms, including Accenture, which has a planned billion-dollar partnership with Anthropic. The nonprofit METR, which previously investigated an incident involving OpenAI, is also expected to be involved.</p><p>On-site evaluations are not new; similar practices exist in industries such as aviation and banking, where regulatory bodies place experts within companies to ensure compliance. Brad Carson, president of the AI-governance nonprofit Americans for Responsible Innovation, likened AI companies to banks that pose systemic risks, necessitating oversight.</p><p>However, the proposal lacks a comprehensive framework for AI auditing. As long as evaluations remain voluntary, the organizations selected by the companies may not have full access to assess risks effectively. Some METR employees have expressed concerns about the adequacy of their work as true audits. The potential for conflicts of interest exists, particularly if evaluators are funded by the companies they monitor.</p><p>Concerns about the independence of evaluators are compounded by the close-knit nature of the AI community. While METR does not accept funding from AI companies, many staff members have longstanding relationships with industry professionals, raising questions about impartiality.</p><p>Despite the limitations of embedded evaluators, their presence could improve transparency regarding AI incidents. However, experts emphasize the need for government-backed authority to establish minimum safety standards and ensure the independence of evaluators. Gillian Hadfield, a professor at Johns Hopkins, highlighted the importance of qualified oversight to prevent auditors from prioritizing leniency over quality.</p><p>In light of public backlash and internal pressures, both Anthropic and OpenAI have expressed support for state and federal legislation regarding third-party audits. Recent comments from Ben Horowitz of a venture firm indicate a shift towards a private-auditor approach, suggesting a changing perspective in Silicon Valley.</p><p>Some politicians are advocating for more stringent measures, including treaties with China and bans on superintelligence. California Governor Gavin Newsom, who previously vetoed a bill mandating third-party audits, has now issued an executive order requiring onsite auditors and the development of an "AI kill switch." Meanwhile, President Trump has emphasized the need for strong leadership in regulating AI.</p><p>While the agreement among AI leaders to implement embedded evaluators is a step forward, skepticism remains regarding the effectiveness of this approach given the industry's history of lobbying against regulation and safety measures.</p>