41 How Datadog Built a ‘Universal Machine Tool’ for Claude Code

41.1 Overview

On July 21, 2026, Anthropic published a deep dive into how Datadog — the cloud monitoring and observability company — had built an internal platform called Temper to manage and verify the output of agentic coding systems at scale. The post was notable not just for the technical sophistication of Temper itself, but for the philosophical shift it represented: a move from treating AI-generated code as something to be reviewed to treating it as something to be verified.

Datadog’s context was significant. By mid-2026, Claude Code drove approximately two-thirds of all AI coding at Datadog — a company with thousands of engineers and one of the most sophisticated internal tooling cultures in the industry. The challenge they faced was not whether to use agentic coding, but how to trust its output at the scale and reliability level that a critical observability platform demands.

The answer was Temper, which Datadog described as a “universal machine tool” for agentic systems — borrowing an analogy from manufacturing, where a machine tool is a device that makes other machines with precision and repeatability. Temper was a system that made agents produce verifiable, trustworthy output.

41.2 The Evolution: Courier → BitsEvolve → Helix → Temper

Temper was not Datadog’s first attempt at managing agentic coding. It was the culmination of an evolutionary path through four generations of internal tooling:

Generation	Name	Approach	Limitation
1st	Courier	Human review of agent output	Doesn’t scale with agent volume
2nd	BitsEvolve	Automated test-based verification	Tests alone miss behavioral issues
3rd	Helix	Multi-agent review (agents check agents)	Agents share blind spots
4th	Temper	Specification + multi-layer verification	Most comprehensive

41.2.1 Courier: Human Review

The first generation was straightforward: agents generated code, and human engineers reviewed every change. This worked when agent output was small, but as Claude Code usage grew to thousands of changes per day, human review became the bottleneck. The quality of review also degraded under volume — a tired reviewer at 4 PM was not as effective as a fresh one at 9 AM.

41.2.2 BitsEvolve: Test-Based Verification

The second generation introduced automated verification through tests. If the agent’s output passed the test suite, it was accepted. This scaled better than human review but exposed a fundamental problem: tests verify what you thought to test, not what matters. An agent could produce code that passed all existing tests while introducing subtle behavioral changes that no test covered.

41.2.3 Helix: Multi-Agent Review

The third generation used multiple agents to review each other’s work — an adversarial approach where one agent generated code and another tried to find problems with it. This caught more issues than test-based verification but had a subtle weakness: agents trained on similar data share blind spots. If the generating agent and the reviewing agent both misunderstood a particular API, neither would catch the error.

41.2.4 Temper: Specification + Verification

The fourth generation — Temper — represented a paradigm shift. Rather than having agents produce code that was then reviewed, Temper had agents produce specifications that were then verified through multiple independent methods. Code was a byproduct of verified specifications, not the primary output.

41.3 How Temper Works

41.3.1 The Core Insight: Agents Produce Specifications, Not Code

The fundamental redesign in Temper was this: agents do not produce code; they produce specifications. Code is then derived from the specification through a process that is itself verified.

Traditional approach:
    Agent → Code → Review → Ship (if approved)

Temper approach:
    Agent → Specification → Verification → Code → Ship
                                    │
                          ┌─────────┼─────────┐
                          ▼         ▼         ▼
                    Verify     Verify    Verify
                    Layer 1    Layer 2   Layer 3
                    │            │            │
                    └────────────┼────────────┘
                                 ▼
                           All pass? → Generate Code → Ship
                           Any fail? → Reject → Retry

This inversion solved several problems:

Specifications are easier to verify than code. A specification describes what the code should do, not how. Verifying “this function should return the sum of two numbers” is simpler than verifying that a particular implementation is correct.
Verification can be multi-modal. Different verification methods can check the same specification from different angles, catching issues that any single method would miss.
The specification is the artifact of record. If code needs to change, the specification is updated first, then code is regenerated — ensuring the spec and code never diverge.

41.3.2 The Four Verification Layers

Temper’s verification kernel applied four independent layers of verification to every specification:

41.3.2.1 Layer 1: Symbolic Reasoning

Symbolic reasoning applied formal logic to verify that the specification was internally consistent and logically sound. This was the most rigorous layer, capable of proving (not just testing) that certain properties held.

For example, if a specification stated that a sorting function should produce output where “for all pairs (a, b) in the output, a ≤ b,” symbolic reasoning could verify this property held for all possible inputs, not just tested ones.

41.3.2.2 Layer 2: Exhaustive State Exploration

Exhaustive state exploration enumerated all possible states and transitions defined by the specification, verifying that:

All reachable states were valid.
No state led to an undefined or erroneous condition.
State transitions matched the specification’s intent.

This was particularly powerful for specifications involving state machines, protocols, or complex control flow.

41.3.2.3 Layer 3: Deterministic Simulation

Deterministic simulation ran the specification through a simulated environment, executing it against generated inputs and comparing outputs against expected behavior. Unlike unit tests (which test specific cases), deterministic simulation could generate millions of test cases automatically and verify behavior across all of them.

# Conceptual: deterministic simulation
def simulate_specification(spec, iterations=1_000_000):
    """Run the specification against millions of generated inputs."""
    for i in range(iterations):
        inputs = generate_inputs(spec.input_types, seed=i)
        expected = spec.compute_expected(inputs)
        actual = spec.execute(inputs)
        if actual != expected:
            return Failure(
                input=inputs,
                expected=expected,
                actual=actual,
                iteration=i,
            )
    return Success()

41.3.2.4 Layer 4: Adversarial Verification

Adversarial verification pitted an adversarial agent against the specification — an agent specifically trained to find inputs or conditions that would break it. This was the “red team” layer, designed to catch the edge cases and malicious inputs that the other layers might miss.

Note

The four layers were deliberately independent — they used different methodologies, different assumptions, and different failure modes. The probability of a bug passing all four layers was vanishingly small, because a bug that evaded symbolic reasoning might be caught by simulation, and a bug that evaded simulation might be caught by adversarial testing.

41.3.3 The Three Contracts

Every specification verified by Temper had to satisfy three contracts:

41.3.3.1 Contract 1: Behavior

The behavior contract defined what the code should do — its inputs, outputs, and the relationship between them. This was the functional specification.

behavior_contract:
  function: "calculate_latency_percentile"
  inputs:
    - name: "measurements"
      type: "list[float]"
      constraints: "non-empty, all values >= 0"
    - name: "percentile"
      type: "float"
      constraints: "0 <= percentile <= 100"
  outputs:
    - name: "result"
      type: "float"
      constraints: ">= 0"
  properties:
    - "result is the value below which `percentile`% of measurements fall"
    - "function is deterministic: same inputs always produce same output"

41.3.3.2 Contract 2: Data Contract

The data contract defined what data the code could access, how it could access it, and what it could do with it. This was the data governance specification.

data_contract:
  reads:
    - resource: "metrics_store"
      access: "read_only"
      filter: "tenant_id == request.tenant_id"
  writes:
    - resource: "results_cache"
      access: "write"
      retention: "24h"
  prohibitions:
    - "no cross-tenant data access"
    - "no writing to metrics_store"

41.3.3.3 Contract 3: Authorization

The authorization contract defined what actions the code was permitted to take and under what conditions. This was the security specification.

authorization_contract:
  permitted_actions:
    - "read:metrics"
    - "write:results_cache"
  required_permissions:
    - "user has role: analyst"
  denied_actions:
    - "delete:anything"
    - "write:metrics_store"

Together, these three contracts provided a complete specification of what the code should do, what data it could touch, and what it was authorized to do — all verifiable before any code was written.

41.4 The “Dark Factory” Concept

The post referenced a concept from Simon Willison: the “dark factory” — a manufacturing facility that operates without lights, because there are no humans inside. Applied to software, a “dark factory” was an environment where agents worked without human intervention — generating, verifying, and shipping code autonomously.

Temper was the enabling technology for Datadog’s dark factory. By making verification comprehensive and automated, Temper reduced the need for human review to the point where agents could operate with minimal supervision for many classes of changes.

Note

The “dark factory” did not mean zero human involvement. It meant that humans set the policies, defined the contracts, and handled exceptions — while the routine work of generating and verifying code happened autonomously. The human role shifted from reviewer to architect and policy-setter.

41.4.1 Trust Granularity

Not all work was suitable for the dark factory. Datadog implemented granular trust levels:

Trust Level	Human Involvement	Example Changes
Full autonomy	None	Bug fixes, test additions, refactoring
Notify after	Review post-merge	Feature additions within existing patterns
Review before	Human approval required	New APIs, schema changes
Human-only	No agent involvement	Security-critical, compliance-related

41.5 The Key Insight: Verification Is the Bottleneck

The most important takeaway from Datadog’s experience was this: the real bottleneck in agentic coding is not generation, but verification.

Generating code is fast and getting faster. Claude Code can produce thousands of lines of code in minutes. The challenge is knowing whether that code is correct, secure, and behaves as intended. Traditional code review — whether by humans or by other agents — does not scale to the volume that generation can produce.

Temper’s answer was to make verification automated, multi-modal, and comprehensive — treating it as an engineering problem worthy of the same investment as the generation side.

Tip

For organizations building internal tooling for agentic coding, Datadog’s experience suggested an investment priority: spend more on verification infrastructure than on generation tooling. The generation side is handled by the model (Claude); the differentiation comes from how well you can verify and trust the output.

41.6 Broader Implications

41.6.1 For Engineering Culture

Temper represented a cultural shift from “trust but verify” to “verify, then trust.” In the traditional model, a senior engineer’s review was the trust mechanism. In the Temper model, verification was the trust mechanism — and the engineer’s role was to ensure the verification was sound.

41.6.2 For Code Quality

Counterintuitively, Temper-driven code quality was often higher than human-written code, because:

Every change was verified against a formal specification.
Edge cases were explored exhaustively, not just the obvious ones.
Adversarial testing caught issues that humans would overlook.

41.6.3 For Team Structure

The dark factory model changed team composition. Rather than scaling review capacity with generation capacity (more reviewers for more code), teams could scale generation much faster than review — because review was automated. This freed engineers to focus on higher-value work: architecture, product design, and the specifications themselves.

41.7 Key Takeaways

Datadog’s Claude Code drives two-thirds of AI coding at the company — one of the highest adoption rates in the industry.
Temper is a “universal machine tool” for agentic systems — a platform that makes agents produce verifiable, trustworthy output.
The evolution: Courier → BitsEvolve → Helix → Temper. Each generation addressed the limitations of the previous: human review → test-based → multi-agent → specification + verification.
The core redesign: agents produce specifications, not code. Code is derived from verified specifications, making verification the primary activity rather than an afterthought.
Four verification layers provide defense in depth. Symbolic reasoning, exhaustive state exploration, deterministic simulation, and adversarial verification — each independent, each catching different classes of errors.
Three contracts define every specification. Behavior (what it does), Data (what it touches), Authorization (what it’s permitted to do) — together providing a complete, verifiable specification.
The “dark factory” concept (Simon Willison) describes agent-driven development with minimal human intervention — enabled by comprehensive automated verification.
Trust is granular. Full autonomy for low-risk changes, human review for high-risk ones — not a binary choice.
The real bottleneck is verification, not generation. Generation is fast and accelerating; the challenge is knowing the output is correct.
Investment priority: verification infrastructure over generation tooling. The model handles generation; differentiation comes from how well you verify and trust output.
Temper-driven code quality can exceed human-written code because verification is more thorough than human review can be.