Building strong safety evaluations

According to Anthropic Research:

Background and Challenges

Safety evaluations guide critical decisions regarding AI model deployment and safeguards. However, researchers have found that AI evals are easily gamed, poorly documented, and sensitive to minor formatting changes. Drawing on psychological research reforms, Anthropic outlines best practices for building trustworthy safety evaluations.

Design and Execution Best Practices

Building strong evaluations requires matching the design to the core question, whether observational or causal, and measuring the right construct, such as appropriate response rates instead of simple refusal rates in sensitive contexts like self-harm or dual-use biology. Evals should use multi-turn simulations to capture nuanced, escalating user behavior and run in secure, sandboxed environments to prevent unintended actions.

Grader Validation and Reporting

Since modern evaluations rely on AI graders, teams must calibrate them rigorously, ensure atomic and binary criteria, and validate them against human labels. Furthermore, researchers should preregister analysis plans, report full statistics with confidence intervals, and disclose model configurations transparently to ensure results are reproducible.