Secure and Responsible AI

Anthropic: same model, safety scores vary by up to 34 points – depending on deployment configuration

On November 13, 2026, Anthropic published ten principles for building and reporting AI model safety evaluations. The key takeaway for enterprises: in multi-turn testing, the exact same model showed score variations of up to 34 percentage points depending on whether it operated via the API or inside claude.ai.

  • SourceAnthropic
  • Source publicationNovember 13, 2026
  • TypeAI security

What happened

Publication. The Anthropic team compiled the principles guiding their model safety evaluations: from team composition and test design to reporting results. Examples cover areas including mental health crisis support, dual-use applications (biology, chemistry, cybersecurity), and election polling place misinformation.

Configuration alters outcomes. In multi-turn evaluations of Claude Fable 5.1, the rate of appropriate responses differed between the API and claude.ai by up to 34 percentage points. The two setups differ by system prompt. In conversations addressing suicide and self-harm, the score was 60% via API versus 94% on claude.ai with the updated system prompt.

A single prompt is not enough. According to Anthropic, in single-turn API tests, all recent models produce 94–98% harmless responses. Multi-turn dialogues reveal significantly greater variance, as risks often emerge only within the full conversation context. The team also recommends running evaluations in an sandbox without internet access: during cybersecurity testing, models can probe the boundaries of test environments.

Key findings

  • Ten principles: from assembling subject-matter expert teams and running multi-turn tests alongside paired benign/harmful queries, to defining metrics prior to testing and reporting with confidence intervals.
  • Anthropic’s recommended minimum sample size: 100 items for complex multi-turn evaluations and 1,000 for simple single-turn tests.
  • Confidence intervals in multi-turn testing can exceed ±10 percentage points.
  • Scores of 98% and 94% for Claude Opus 5 and Opus 5.5 on romance scam evaluations showed no statistically significant difference.
  • Anthropic’s single-turn suites span 16 policy domains across seven languages.

Why it matters

SafeWave CommentaryOrganizations deploying AI assistants for customers or internal staff typically rely on vendor model cards. Anthropic’s data demonstrates that safety performance hinges heavily on configuration: system prompts, delivery channels, language, and connected tool integrations. For deployments across banking, energy, or retail – and under AI Act obligations – this underscores the need to assess assistant safety within your own runtime environment. Recommended actions: test your specific deployment setup rather than just the base foundation model, including multi-turn dialogues in Polish; treat every system prompt modification as a security change and retest; measure both harmful responses and over-refusals; compare versions using confidence intervals rather than isolated figures; and continuously monitor assistant behavior post-deployment. More: Secure and Responsible AI.