Your AI Agent Can Forget a Safety Rule After Compaction

A long-running AI agent can start with the right rule, obey it for many turns, and lose it during a normal maintenance operation.

A new paper, The Compaction Cliff in Long-Running AI Agent Memory, tested a production-style /compact prompt on 20 real agent configurations. In the authors' experiment, Sonnet 4.6 preserved 53% of safety rules after one compaction round and 10% after five. Their proposed typed method retained 96% after five rounds. The result does not prove that every model, vendor, or opaque native compactor has the same failure. It does prove that “the rule was in the prompt” is not sufficient launch evidence once a product rewrites its own context.

This matters now because compaction is moving from an advanced harness feature into normal agent infrastructure. OpenAI offers native compaction for long-running Responses API workflows, and Anthropic describes compaction as a standard technique for tasks that exceed one context window. For AI app builder users, nontechnical founders, and small teams, the practical question is not how to invent a better summarizer. It is: which product commitments must survive unchanged, how will you verify them after every rewrite, and what happens when they do not?

This guide turns the research into a founder release gate. You will leave with precise terms, a typed retention table, a complete hypothetical support-agent scenario, a five-round regression suite, failure modes, fit boundaries, a machine-readable compaction contract, and a 48-hour rollout plan.

What the new paper found, and what it did not

The paper treats an agent's working knowledge as heterogeneous. A prohibition, a procedure, a current fact, a user preference, and an old event do not tolerate the same amount of rewriting. Yet a general summary prompt often compresses them together when the context reaches a token limit.

The authors assembled AgentArtifactCorpus from 396,934 artifacts across 54,628 public GitHub repositories and analyzed agent instructions from eight platforms. They classified items into five operational types: constraints, procedures, beliefs, facts, and episodes. Their Knowledge Triage design routes those types through different policies for compaction, decomposition, and retrieval. The reference implementation and dataset are public.

The headline result needs boundaries. The 53%-to-10% decay came from author-run tests on a 20-configuration subset using a specific production-style compaction prompt and model. The authors' downstream tests include 200 medical scenarios, 115 retail tasks, and 50 airline tasks, but these remain research environments rather than your product. Their safety guarantee also depends on classification: if a real constraint is mislabeled as an ordinary fact or episode, it can still enter the lossy lane. The paper reports that declarative rules such as “the patient is allergic to penicillin” were harder to classify than imperative rules.

Another study, Governance Decay, independently supports the mechanism. Across 1,323 author-run episodes and seven model families, it reports prohibited tool-action violations rising from 0% with the policy fully visible to 30% after compaction; when a constraint survived the summary, violation remained 0% in that benchmark. Its Constraint Pinning mitigation restored the measured violation rate to 0%. This is still pre-deployment research, but the important causal claim is testable: did the governing item survive, and did behavior remain compliant?

Do not publish “all compaction is unsafe.” Publish the narrower, useful conclusion: compaction is a versioned transformation that must re-earn your safety and product invariants.

Define compaction, trimming, memory, and the source of truth

Four terms prevent teams from testing the wrong object.

Your source of truth is the authoritative state outside model prose: policy records, approval status, account data, executed tool results, task state, and versioned product configuration. A compaction summary can point to that source. It should not quietly replace it.

Safety rules are broader than prohibitions

When founders hear “safety rule,” they often imagine only “never do X.” A production agent has several kinds of product invariant:

• Permission: do not send, publish, purchase, delete, merge, or disclose without the named approval.
• Scope: act only on this account, folder, branch, date range, recipient, region, or budget.
• Sequence: verify identity before reset; calculate before charging; validate before deploy.
• Pending state: an action was proposed but not approved, or started but not independently confirmed.
• User correction: the customer said the preferred plan is Basic, not Pro; the earlier address is obsolete.
• Truth boundary: a draft is not a completed action; a tool request is not a successful tool result.
• Privacy boundary: these fields may be summarized, while these must not enter a remote model or log.
• Stop condition: pause on ambiguity, repeated failure, cost ceiling, policy conflict, or unavailable verification.

Inventory rules by their consequence, source authority, scope, validity interval, and required wording. Do not ask a summarizer to infer which sentence matters most while the token budget is already failing.

Give different information different retention lanes

The paper's five-type taxonomy is a useful starting point, but a small product team can implement a simpler operational table.

The retention policy should be asymmetric. Hard constraints and pending approvals can consume more tokens because losing one can create an irreversible effect. Old tool output can move to external storage with a digest and retrieval pointer. Preferences can be compressed more aggressively because a style mismatch is usually recoverable.

Do not pin everything. If every line becomes a hard constraint, the pinned set eventually fills the budget and compaction cannot work. Promote only items whose loss would change authorization, truth, privacy, financial exposure, or a required product outcome. When the hard set exceeds its budget, stop or decompose the task instead of silently compressing it.

Separate preservation from behavior

A compaction gate needs two tests.

The preservation test asks whether required information remains available in the next working context. It should compare structured IDs and fields, not judge whether a paraphrase “sounds equivalent.” A rule such as “refunds over $100 require a manager” loses force if summarized as “large refunds may need review.”

The behavior test asks whether the agent still makes the right decision when the rule becomes relevant. A perfectly preserved rule can still be ignored. A missing sentence might be reloaded from a policy tool before action. Artifact recall and behavior are related, but neither substitutes for the other.

For each critical rule, create at least three fixtures:

1. Direct case: the user asks for the prohibited or gated action plainly.
2. Distractor case: dozens of irrelevant turns appear before the action.
3. Conflict case: later content suggests the opposite action without valid authority.

Test five rounds, not one successful summary

The defining observation in the Compaction Cliff paper is cumulative loss. One rewrite may look acceptable while repeated rewriting erodes qualifiers, scope, exceptions, and pending state.

Build a bounded test that forces five compaction events. Do not wait for a production conversation to become long enough. Use synthetic filler and tool results to cross the threshold while preserving the same user job.

Run this sequence:

1. Load the rule set, current product state, user correction, pending approval, and test task.
2. Trigger compaction, then inspect the machine-readable post-compaction state.
3. Ask a neutral question so the agent continues normally.
4. Add a controlled tool trace or document block and trigger the next compaction.
5. Repeat through round five, invoking one critical decision after each round.

• hard-rule recall by ID and normalized content;
• procedure-step and pending-gate preservation;
• stale fact and superseded-correction rate;
• compliant decision rate after each round;
• unauthorized or duplicate external effects;
• token footprint of each retention lane;
• compaction latency, cost, and failure recovery;
• false promotion rate: ordinary items incorrectly pinned as rules.

Do not compact in the middle of an unresolved transaction

When compaction happens matters as much as what it preserves.

The Self-Compacting Language Model Agents paper argues that a fixed token threshold can fire mid-derivation or mid-search. Its proposed scaffold gives the model a compaction tool plus a rubric that favors stable subtask boundaries. In author-run tests across six benchmarks and seven models, the method reports lower cost and better task performance than fixed-interval summarization. The result is encouraging, but the model should not be the sole judge for high-impact product transactions.

Define deterministic no-compaction zones around:

• approval requested but not resolved;
• tool call submitted but outcome unknown;
• multi-step write or migration in progress;
• payment, refund, deletion, or publish confirmation;
• credential rotation or permission change;
• a user correction not yet committed to durable state;
• a safety or privacy incident under review.

OpenAI's current Agents SDK session documentation illustrates why lifecycle details matter: compaction clears and rewrites session history, can block streaming, attempts restoration on replacement failure, and warns against concurrent session mutation during manual compaction. Even a semantically perfect summary can be unsafe if the replacement races with a new approval or tool result.

Scenario: a support agent forgets a pending refund approval

Imagine CedarDesk, a small support product that reads tickets, checks orders, drafts replies, and issues refunds after approval.

Its policy includes these rules:

• refunds up to $25 may be auto-approved for verified duplicate charges;
• refunds from $25.01 to $100 require an agent review;
• refunds above $100 require a manager;
• every refund must include an idempotency key and a post-action ledger check;
• a proposed refund is not approved merely because the user asked for it.

CedarDesk's product failure is not simply that one sentence vanished. Three controls failed:

1. approval status lived only in free-form context;
2. the compactor did not preserve the pending gate;
3. the refund API accepted a model-selected action without checking authoritative approval state.

The scenario is hypothetical. It is not a YBuild customer case or a claim that a named model currently issues refunds after compaction.

Use a compaction safety contract

Keep one versioned contract per product job and runtime configuration. The values below are a reusable structure, not universal thresholds.

compaction_release:
  product_job: "support ticket to reviewed refund"
  runtime:
    model: "exact model and revision"
    harness: "exact version"
    compactor: "provider-native | custom | none"
    compaction_threshold: "measured token/item threshold"
    max_compactions_tested: 5
  authoritative_state:
    policy_store: "versioned policy ID"
    approval_store: "durable record system"
    action_ledger: "independent outcome source"
    raw_session_retained: true
  retention_lanes:
    exact: ["hard constraints", "pending gates", "stop conditions"]
    structured: ["procedures", "current corrections"]
    reload: ["account state", "price", "inventory"]
    lossy: ["old tool output", "superseded attempts", "style preferences"]
  invariants:
    required_rule_ids: ["refund-manager-over-100", "approval-required-25-to-100"]
    forbidden_mid_compaction_states: ["approval_pending", "tool_outcome_unknown"]
    fail_closed_on_missing_rule: true
    external_tool_rechecks_policy: true
  regression:
    direct_cases: "pass count / total"
    distractor_cases: "pass count / total"
    conflict_cases: "pass count / total"
    five_round_rule_recall: "measured value"
    five_round_behavior_pass: "measured value"
    unauthorized_effects: 0
  decision:
    posture: "ship-limited | pilot | hold | reject"
    owner: "named reviewer"
    expires_on_change: ["model", "harness", "compactor", "rules", "tools"]

The decisive fields are not the recall percentage alone. authoritative_state, forbidden_mid_compaction_states, fail_closed_on_missing_rule, and the tool's independent policy check keep a classifier miss from becoming an external effect.

Use hard gates before a score

Apply these hard blockers before considering average quality:

• any critical rule missing after a tested compaction round;
• any pending approval changed into approved or absent;
• any prohibited action attempted after compaction;
• any duplicate or unverified external effect after replacement failure;
• no recoverable raw session or authoritative source for a high-impact decision;
• compaction during a prohibited transaction state;
• an unknown model, harness, compactor, rule-set, or tool version in the release record.

A larger context window can justify fewer compaction events. It cannot waive the gate. A provider-native opaque compactor may perform better than text summarization, but if your product cannot inspect its internal representation, behavior and effect tests become more important, not less.

Watch for eight misreadings and failure modes

1. “The paper proves our vendor is unsafe.” It tests specific author-run configurations and mechanisms; your vendor and product need direct evaluation.
2. “Pin all instructions.” An unbounded hard lane recreates the context-limit problem and lets low-authority content crowd out real controls.
3. Keyword-only classification. Declarative constraints and user corrections can lack obvious policy words.
4. Summary-quality judging. A fluent reviewer or LLM grader may approve a paraphrase that removed the operative threshold or scope.
5. One-round testing. Cumulative rewrites can fail after an apparently strong first summary.
6. Artifact-only testing. The rule survives in text but the agent or tool ignores it.
7. Behavior-only testing. A downstream system accidentally blocks the action, hiding that the agent lost its policy.
8. Context as database. Approval, current account state, and external effects live only in a conversation that compaction can rewrite.

Know where this gate applies and where it does not

Use the full five-round gate for long-running agents that can call tools, change customer state, access sensitive data, or work across approvals and handoffs. Support, operations, coding, research, finance, onboarding, and scheduling products can all accumulate enough history to compact.

A lighter test is reasonable for short, stateless generation where each request starts from a complete validated input and creates only a reversible draft. If the product never compacts, the specific compaction risk does not apply, though trimming, retrieval, memory, and prompt assembly can still lose important context.

Do not treat typed retention as a compliance certificate or a replacement for product policy. Do not use the paper's percentages as your launch threshold. A low-consequence writing assistant might tolerate a lost preference; a medical, financial, or destructive-action agent may require exact preservation, external authorization, and zero unauthorized effects in the tested set.

The gate is also not an argument against compaction. Long contexts accumulate stale tool output, irrelevant history, contradictions, and cost. Research on self-compaction suggests that well-timed compression can improve both efficiency and task outcomes. The product decision is to make loss deliberate, typed, testable, and contained.

Run a 48-hour compaction regression

An agent that remembers the rule in turn one has not passed. The product passes when the rule survives its context lifecycle, still governs behavior, and cannot be bypassed at the point of effect.

References

1. The Compaction Cliff in Long-Running AI Agent Memory
2. Knowledge Triage reference implementation
3. AgentArtifactCorpus dataset
4. Governance Decay: How Context Compaction Silently Erases Safety Constraints
5. Self-Compacting Language Model Agents
6. Anthropic: Effective context engineering for AI agents
7. Anthropic: Scaling Managed Agents
8. OpenAI: Equipping the Responses API with a computer environment
9. OpenAI Agents SDK: Session and compaction documentation