Anthropic gives a METR team employee like access to assess its safety practices, with the right to publish findings Anthropic cannot veto
We’ve committed to having evaluators assess our safety practices. We're in discussions with multiple organizations, and have now given employee-like access to a team from @METR_Evals. In this post, we describe how we work with third-party evaluators:
Context
Outside evaluators with deep inside access and freedom to publish is a new level of scrutiny for a frontier AI lab, and Anthropic is urging rivals to do the same.
This is a first hand announcement from Anthropic, and METR is a nonprofit that evaluates frontier AI models. Anthropic says it signed a White House accord last month in which companies pledged to work with independent auditors or evaluators.
Deep dive
On October 15, Anthropic posted that it had committed to having evaluators assess its safety practices, was in discussions with multiple organizations, and had given employee like access to a team from METR.
In the linked post, Anthropic says it has onboarded six METR employees with company hardware, office space and access to internal tools roughly on par with its own risk assessment teams. Staff are encouraged to share information with them as with a colleague, outside defined categories such as customer data, legal advice and model architecture details. METR will publish its analysis of Anthropic's risk levels, incidents or practices, and Anthropic says it cannot veto publication, though it can require narrow redactions that METR may publicly comment on.
Anthropic describes three tiers of evaluator access. SecureBio is in the first tier, reviewing the chemical and biological weapons chapter of Anthropic's August Risk Report. It says it is also working with Accenture and Faculty, and is in early discussions with organizations including Hugging Face.
Anthropic compares the setup to bank supervisors who sit inside large banks, and says every frontier AI company should embed evaluators voluntarily.
What to expect next
Watch for METR's first published assessment under this arrangement and how many redactions Anthropic requests.
Also watch whether OpenAI, Google DeepMind and other labs that signed the White House accord offer similar access, and whether SecureBio and other evaluators move up to tiers with more access and publication rights. Critics will also look at how independent the arrangement proves in practice.