Anthropic Shares 10 Recommendations for Building Safety Evaluations
Safety evaluations are critical for understanding whether a model is ready to deploy, which safeguards to build for it, and where to focus training and research. On our Engineering Blog, we offer 10 recommendations for how to design, run and report them.
Context
Safety claims about a model are only as strong as the evaluations behind them, and Anthropic is publishing the methods it wants others to use.
Anthropic's official account posted on November 13 that safety evaluations are critical for deciding whether a model is ready to deploy, which safeguards to build and where to focus training and research. The company linked to an Engineering Blog post, published the same day, that offers 10 recommendations for how to design, run and report those evaluations. The post draws on Anthropic's experience and on examples from its system cards.
Deep dive
Anthropic posted on November 13 that safety evaluations are critical for understanding whether a model is ready to deploy, which safeguards to build for it, and where to focus training and research. The company said its Engineering Blog offers 10 recommendations for how to design, run and report them, and linked to a post titled Building strong safety evaluations, dated November 13.
The blog says a safety evaluation measures a model's propensity to cause or contribute to harm, rather than measuring a specific capability, and that a safety claim is only as sound as the evaluation behind it. The 10 recommendations cover building a team with the right skills, running evaluations safely, matching the design to the question, measuring what you mean to measure, making sure results generalize, keeping measures internally valid, preregistering, reporting statistics fully, documenting everything and disclosing flaws responsibly.
If you only do three things, the authors write, write down the construct you want to measure before collecting data, validate your grader against human judgment before trusting any number it produces, and report uncertainty so readers can tell signal from noise. The post is a methods guide, not a new model release or a new score.
What to expect next
Watch whether other labs adopt the same reporting habits, especially preregistration, grader validation and uncertainty, in future system cards. Follow up work may update the recommendations as evaluation methods change. Independent teams can also test whether evaluations built this way are harder to game than current public safety scores. If the next Anthropic system card follows these 10 points, that would show the company is applying its own guidance.