Anthropic

Evaluating an AI coding tool is not a capability comparison. Capability converges fast, and the questions that decide a rollout — who can buy it, what the meter does, where the source goes, what an auditor is handed — sit in documents nobody reads until the security review.

Scope note: this series covers tools engineers use to write software. It does not cover putting a model inside a shipped product — different decision, different rubric.

Claude Code — an agentic coding tool that reads a codebase, edits across files, runs commands, and works until a condition holds.

One engine, several surfaces. All of them share the same repository configuration, so a team standardizes once rather than per client. Where they differ is where the code executes — and that is the column an evaluation actually needs.

• The row that matters — web and mobile sessions are hosted execution, not local
  • Every other surface keeps the repository on the developer's machine
  • Adopting the terminal is not the same approval as adopting the browser
  • Scope them separately, or scope the hosted ones out

What hosted execution actually involves:

What a team standardizes on:

Six authentication paths. The choice is a data-governance decision wearing a billing costume.

• Personal subscription — not a company seat
  • Identical tool, personal billing, no admin visibility
  • Consumer data terms, not commercial — see § 04
  • Any evaluation without a plan to detect and displace this is evaluating a policy, not a deployment
• Claude for Teams — the smallest real company seat
  • Self-service · collaboration · admin tools · billing management
• Claude for Enterprise — the governance tier
  • The seat is not coding-only. It covers Claude on web, desktop, and mobile, plus Claude Code and Claude Cowork — the vendor's agentic product for general work rather than software. Its capabilities are outside this series' scope; its billing and retention coverage are not, for two reasons below
  • Single sign-on · domain capture · role-based permissions · compliance API
  • Managed policy settings that push one configuration organization-wide
  • Purchasable self-serve online or sales-assisted
• Console — API billing
  • Granular roles: a user can be limited to creating Claude Code keys only, rather than any key
  • Useful where a hard boundary between coding access and general model access is wanted
• Cloud providers — Amazon, Google, Microsoft model platforms
  • Inherits an existing contract, security review, and billing relationship
  • Often the fastest route through procurement, rarely the cheapest per token
• Self-hosted gateway
  • Corporate single sign-on; the gateway-issued token is the session's only credential

The section most evaluations get wrong — the Enterprise plan does not behave like per-seat software.

• Seat fee covers access only — single seat type, per user per month, billed annually
  • No included token allowance · no per-seat usage cap
  • All usage across chat, Claude Code, and Cowork bills separately at standard rates — one pool, three products
  • Same structural trap as the shared meter in the OpenAI episode: a heavy month of non-engineering agent work is a thinner month for engineering, and neither side connects the two unless told
  • Constrained instead by spend limits at organization, group, and individual level
  • This is a cloud-bill model, not a licence model. It needs a named owner before rollout
• Published benchmark — rare, and the most useful number in any of these vendor documents
  • ~$13 per developer per active day
  • $150–250 per developer per month
  • 90% of users stay below $30 per active day
  • Documented advice: pilot a small group, establish a baseline, then widen

A published cost benchmark with a stated distribution is rare. Model the pilot against it — a team landing far outside has a behavioural cause that is findable.

• Cost levers (documented, mostly architectural)
  • Right model per task rather than defaulting to the largest
  • Manage context deliberately · reduce tool-server overhead
  • Offload deterministic work to hooks and skills instead of paying a model for it
  • Adjust extended-thinking settings · delegate verbose operations to subagents
  • Agent teams are called out explicitly as a multiplier — adopt them and the line moves
• Subscription plans differ — usage is included, so the per-session cost figure is not a billing signal for those users
  • Communicate this during rollout; it is a reliable source of confused escalations

Four questions a legal team asks, in the order it asks them.

1 — Who owns the output?

• Customer retains all rights to Inputs and owns Outputs, per the commercial terms
• Anthropic disclaims any rights it receives in customer content, and assigns to the customer whatever right, title, and interest it has in Outputs
• Clean, affirmative assignment — the strongest form of this clause, and one of three vendors in this series that grants it explicitly

2 — Does the source train the model?

3 — Could generated code carry someone else's licence?

• No public-code matching filter is documented for this tool
• Where a peer product screens suggestions against public repositories and can discard or annotate matches, the documented answer here is the indemnity below rather than a pre-emptive filter
• Consequence for evaluation — an organization that wants prevention rather than defense on licence contamination needs its own control: dependency and licence scanning in the pipeline, applied to agent-authored changes exactly as to human-authored ones
• That is a defensible posture. It should be a decision, not an assumption

4 — Who defends the company if a third party sues?

• Anthropic will defend the customer and indemnify for judgments or approved settlements on a third-party claim
• Scope is unusually wide — it covers a claim that the customer's use of the Services violates any third-party intellectual-property right, and the clause expressly includes data Anthropic used to train a model that is part of the Services, as well as Outputs generated through authorized use
  • Reaching training-data claims, not only output claims, is the notable part
• Conditions to read carefully
  • It applies to paid use — a personal or free-tier account is outside it
  • Use must be in accordance with the terms, which incorporate the usage policy by reference
  • Excluded where the allegation arises from the indemnified party's fraud, willful misconduct, or violations of law
• The practical test — the indemnity is only as good as the organization's ability to prove every developer was on the paid commercial account. Which returns to the login-restriction control in § 06

How long is it kept?

• Operational telemetry is two separate streams, each independently switchable
  • Metrics — latency, reliability, usage patterns. Never include code, prompts, or file paths. One environment variable opts out
  • Error reports — messages and stack traces from the tool's own internals, sent to a third-party error-tracking service. Known secrets, file paths, and email addresses are redacted before anything leaves the machine. A separate variable opts out
  • Error reporting is only on at all for personal subscriptions connecting directly, on a recent version, with no zero-retention or health-data agreement in place. A commercial account with either agreement never sends it
• The feedback command detail most reviews miss
  • It sends conversation history including code
  • The submitter chooses the scope: current session by default, or other sessions from the same project over the last 24 hours or 7 days
  • Optionally, a public issue is created on the code host. Worth naming in policy rather than leaving to judgement

Zero data retention — the setting most vendor-risk teams ask for, and the one that surprises procurement.

• Not included in the standard Enterprise plan. Cannot be switched on from admin settings
  • Available to qualified accounts · enabled per organization by an account team after eligibility review
  • Does not inherit — each new organization needs it enabled again
• Covers — coding-tool inference on Enterprise, direct platform only
• Does not cover — web chat, Cowork sessions, analytics metadata, seat and account administration, third-party connectors
  • Worth stating plainly to engineers: moving the same problem from the terminal into web chat or Cowork leaves the zero-retention path, and nothing in the interface says so
• Costs — features requiring stored prompts are disabled at the backend; one model class that requires retention becomes unavailable
• The routing trap — zero retention applies to requests authenticating into the enabled organization
  • A developer signed in with a personal account or another organization's key is not covered, and the tool works identically either way
  • Countermeasure: managed settings that force the login method and pin the organization

Where inference runs

• Direct platform at Anthropic, with disk-level encryption; on Amazon, Google, or Microsoft platforms under that provider's terms and keys
• Microsoft's Azure-hosted option keeps prompts and completions within Azure, egressing only usage metadata and safety-flagged content
• Enterprise adds customer-managed keys in the organization's own cloud and a United-States-only inference option

• Enterprise carries the expected package — single sign-on, directory provisioning, domain claiming, role-based permissions with custom roles, group management
• Group-level spend limits solve a problem the flat model does not
  • Caps at organization, group, and individual level
  • A small group can get a wide budget for agent-heavy work without exposing the whole organization to that ceiling
• The lifecycle question that matters — when someone leaves, does access die the same day?
  • Directory provisioning makes that automatic
  • On Teams, without provisioning, it is manual — defensible for a small team, but it should be a decision rather than an oversight

Configuration resolves through a documented scope hierarchy. Understanding it is most of understanding what can actually be enforced.

• Subtlety with real consequences — most settings resolve by precedence, but permission rules merge across scopes rather than override, and a few security-sensitive keys are explicit exceptions to the managed rule
  • "Managed wins" is true in general, not universally. Read the precedence page before writing policy

Runtime safety model — layered.

• Manual mode — starts read-only; asks before editing files or running system-modifying commands
  • Runs a built-in set of read-only commands without asking
  • Working-directory boundary — writes limited to the start folder and children; prompts before reading outside it
• Auto mode — a classifier model replaces the human approver and blocks what it judges unsafe
  • Explicit allow and deny rules still apply on top
  • An organization can turn auto mode off entirely — the control a regulated environment will want
• Sandboxed command tool — filesystem and network isolation, with read-denial rules when sandboxing is on
• Allowlisting — frequent safe commands per user, per codebase, or per organization
  • This is a security feature: approval fatigue is how approval becomes rubber-stamping

• Attestations — SOC 2 Type 2 report and ISO 27001 certificate via the trust centre. Table stakes, clears page one of most questionnaires
• Telemetry is the strong suit — metrics, events, and optional traces through the open observability standard
  • Destination is the organization's own collector — data lands in its own systems, not a vendor dashboard it must request access to
  • Event catalogue answers security questions, not just adoption: prompts and responses, tool decisions, permission-mode changes, authentication, tool-server connections, plugin loads, skill activations, errors, refusals, retry exhaustion
  • Documented use cases include auditing security events, attributing actions to users, auditing tool-server activity, and forwarding to a security-event system
  • Managed settings can lock the telemetry destination so a developer cannot redirect it. Mutual-certificate authentication supported
  • Together those turn telemetry from a metric into evidence
• Analytics dashboard — usage metrics, contribution metrics via a code-host integration, leaderboard, export
  • Requires installing an application on the code host; described as deliberately conservative — an underestimate by design
  • Contribution metrics are unavailable to organizations with zero retention enabled
  • A direct trade: maximum privacy costs a measurement capability. Resolve that conflict before signing if the spend case depends on contribution data
• Enterprise interfaces — a compliance API for activity logs and content with user and time filtering; an analytics API for aggregated adoption
  • These are what make retention and legal-hold processes automatable

Strong fit

• Genuinely agentic work — multi-file changes, refactors, test authoring, migrations
• One configuration model across terminal and editor, rather than per-surface policy
• Regulated environments: enforceable managed scope + lockable telemetry destination + the ability to disable classifier-driven auto mode
• Budget conversations that need numbers — the published benchmark makes the case arguable

Weak fit

• Where a flat, predictable per-seat cost is the requirement. Enterprise is consumption billing with spend limits; no negotiation changes its shape
• Where zero retention is a hard prerequisite on a short timeline — an eligibility conversation, not a setting, and it costs the contribution metrics an adoption case often leans on

Disqualifiers

• Source cannot leave the network at all — inference leaves in every configuration, gateway included (it controls authentication, not location)
• No device-management channel — the managed scope is what makes every other control real. Without it, the strongest guarantees here are settings a developer can change

Order to evaluate in

1. Confirm the terms — commercial versus consumer is the line between trained-on and not
2. Choose the inference boundary — direct, an existing cloud contract, or a gateway in front of one
3. Ask about zero retention early — it has a timeline and a cost in features
4. Pilot against the published benchmark
5. Deploy the login restriction and managed policy file before widening — every control above is undone by one engineer signing in with the wrong account, and nothing in the product signals that it happened

Next: OpenAI — where the coding tool shares a usage meter with the company assistant, and directory provisioning is missing from the tier most teams start on.