Don't Let an AI Agent Close the Bug: A Resolution Verification Gate

A customer reports that invoices disappear after changing the account currency. An AI coding agent reproduces a missing row in a fresh test project, edits a filter, adds a unit test, and opens a pull request. CI passes. The pull request merges and the issue closes automatically. Two days later, the customer replies: invoices still disappear in the real account because its historical records use a second currency code that the reproduction never contained.

Nothing in that sequence proves the agent was useless. It proves the team collapsed several different states into one word: fixed.

This guide is for founders and small teams that use AI agents to triage issues, generate patches, or maintain an AI-built application. Its central judgment is: a bug is resolved only when the original symptom is represented by a pinned reproduction, the exact release candidate passes independent checks, and the affected user or an authorized proxy verifies the promised outcome in a representative environment. A plausible diagnosis, new test, green build, preview, merged pull request, or deployed commit is evidence for one state—not permission to skip the others.

You will leave with a resolution state machine, an evidence matrix, a machine-readable resolution receipt, stop rules, metrics, and a 48-hour pilot. This method fits ordinary web and mobile products, integrations, libraries, and internal tools. It does not replace specialist incident response for active exploitation, data loss, regulated records, safety-critical behavior, or failures that cannot be safely reproduced. Those cases need containment and accountable experts before routine automation.

This is not another pull-request review checklist. Code review asks whether a proposed change is acceptable to merge. The resolution gate asks a different, end-to-end question: did the same identified candidate move from the original failing conditions through preview, release, and representative confirmation without losing evidence or changing who had authority to accept the outcome?

Define “fixed” as a verified product outcome

A bug report is a claim that observed behavior conflicts with expected behavior. The report may be accurate, incomplete, environment-specific, a feature request, a documentation gap, or a misunderstanding. Triage determines which.

A reproduction is an executable or inspectable case that causes the reported symptom under named conditions. It is not merely a rewritten issue description. It should pin the relevant input, account state, dependency versions, configuration, platform, and expected-versus-observed result closely enough that another actor can see the same failure.

A candidate fix is a proposed change tied to a particular reproduction. It may eliminate that example while leaving the root problem, introducing a regression, or behaving differently in the release environment.

A verified release is the exact candidate artifact, configuration, and data migration that passed the required checks and produced the accepted outcome on the relevant surface. If you tested commit A but deployed commit B, the evidence does not transfer automatically.

This distinction is visible in GitHub itself. Its official linking documentation explains that fixes, closes, and resolves can close a linked issue when a pull request merges into the default branch. That is useful workflow automation. It is not a semantic claim that the reporter’s environment, production artifact, or downstream data now behaves correctly. A founder should treat auto-close as a configurable state transition, not as evidence.

Learn from Astro without copying the headline

Cloudflare and the Astro maintainers published a useful production case in August 2026. Their account of Astro’s automated triage pipeline says isolated subagents reproduce, diagnose, verify, and propose fixes; a preview package is posted back to the issue; the reporter tests it; and only after confirmation does the automation open a linked pull request. Cloudflare reports that open issues fell from more than 200 to about 30 during the effort.

The reported reduction is a vendor-authored result from one open-source project, not a universal productivity benchmark. The more transferable evidence is the shape of the workflow:

1. The phases are separated rather than entrusted to one continuous “solve it” run.
2. Findings are passed forward as an artifact instead of relying on hidden conversational memory.
3. The original reporter tests a preview built from the candidate.
4. Failure is represented explicitly: the live Astro issue queue exposes states such as needs reproduction, unable to reproduce, fix pending, fix rejected, fix verified, and unable to fix.
5. An unresolved issue can remain visible without being treated as an automation failure that must be cosmetically closed.

Do not copy the 85% reduction as a target. Your issue mix, testability, reporter participation, architecture, severity, and starting backlog differ. Copy the separation of evidence and authority, then measure your own accepted outcomes.

Use a state machine, not a success checkbox

Represent the lifecycle as explicit states with one permitted transition at a time:

Do not make every issue visit every state. A duplicate can move from classified to a canonical issue. A documentation misunderstanding can move to a documentation change and user confirmation. A security report should leave the public automation lane immediately. The model is valuable because transitions have evidence and owners, not because the diagram has many boxes.

Keep the state machine outside model discretion. The open Flue agent documentation shows how an agent instance can be addressed by a stable ID such as an issue number. That is useful persistence, but persistence is not policy. Store allowed transitions, role checks, expiry, and required fields in deterministic application code or workflow configuration. Let the model propose a transition; let the control plane validate it.

Freeze a reproduction before generating the patch

The most common false resolution starts with a movable target. The agent reads the report, infers missing facts, creates a test that matches its own interpretation, and then makes that test pass. The patch and proof share the same assumption error.

Build the reproduction as a separate artifact before the candidate fix is available. Record:

• the reporter’s expected and observed outcomes in their own terms;
• the smallest safe dataset that preserves the consequential condition;
• product, browser, operating system, runtime, dependency, locale, feature-flag, permission, and plan versions that matter;
• setup and action steps;
• the baseline artifact or commit where the failure appears;
• screenshots, logs, trace IDs, or response bodies with secrets removed;
• the assertion that detects the symptom;
• conditions you could not reproduce or inspect.

Input quality helps but cannot guarantee truth. GitHub’s issue-template documentation supports structured forms and required prompts. Ask for expected behavior, observed behavior, minimal steps, version, environment, sample or reproduction link, and consent for maintainers to inspect the material. Do not demand public customer data, access tokens, or production credentials. Provide a private escalation path for sensitive cases.

Separate three outcomes that teams often conflate:

• Reproduced: the stated failure occurred in a controlled case.
• Root cause supported: evidence connects the failure to the diagnosed mechanism.
• Scope bounded: the team knows which neighboring environments or data shapes have and have not been tested.

Build an evidence matrix before accepting the candidate

The resolution gate should join evidence from different failure surfaces. No single green check earns authority over all of them.

NIST’s Secure Software Development Framework is deliberately outcome-based and asks teams to track requirements, risks, design decisions, provenance, verification, and vulnerability response. NIST’s minimum verification guidance also makes a crucial point for small teams: no single technique covers every form of software verification. Apply depth by consequence. A typo does not need the same matrix as an authorization bug, but an AI-generated patch does not become low risk because it is small.

Google’s published code-review guidance likewise tells reviewers to think like users, examine edge cases and concurrency, and verify that tests themselves are valid. That is an important independence rule: the same agent may propose code and tests, but a separate check must judge whether the tests encode the product promise rather than merely the implementation.

Give the reporter the exact candidate, not a screenshot

“Works in preview” is meaningful only when the preview identifies the candidate and preserves the relevant environment. A screenshot can demonstrate that one rendered state existed. It cannot prove which commit produced it, whether the reporter’s input was used, whether a network request succeeded, or whether a later build is identical.

For a library, publish a commit-addressed preview package. The open-source pkg.pr.new project documents continuous preview releases and version rewriting such as 0.0.0-preview-, which helps prevent a preview dependency from colliding with a normal published version. For an application, use an isolated preview URL or staged tenant tied to an immutable commit and configuration snapshot. For a mobile client, use a signed test build. For a data workflow, use a masked dataset and export the resulting artifact.

The verification prompt should be concrete:

We reproduced invoice disappearance with reproduction repro_7f2a on version 2.18.0. Candidate sha256:…91c changes currency normalization only. Please test preview preview_4821 using the attached masked account by 2026-08-25. Confirm whether (1) all 14 historical invoices appear, (2) totals remain unchanged, and (3) switching back to JPY preserves the list. Do not use production credentials. Reply “verified,” “rejected,” or “cannot test,” with the failing step.

This is not customer-service theater. It is an acceptance test with a human-accessible oracle. Make rejection cheap and safe. A “fix rejected” result should return the issue to diagnosis with the candidate evidence preserved; it should not pressure the reporter to negotiate with the agent.

If the reporter disappears, do not leave the issue in fix pending forever. Define an expiry and proxy rule. For a low-risk, fully reproduced defect, a support or product owner may verify against the pinned case and a representative environment, then mark proxy_verified with that limitation. For an enterprise-specific integration or high-impact data defect, lack of customer confirmation may justify release with monitoring but should not be rewritten as customer-confirmed resolution.

Preserve artifact identity from test to release

A valid result attaches to an artifact, not a branch name such as main, a mutable container tag, or “latest preview.” Record the source commit, dependency lockfile hash, build workflow version, configuration set, feature flags, migration version, artifact digest, and deployment ID that matter to the outcome.

GitHub’s artifact-attestation documentation describes build provenance that binds a subject name or path to a digest and workflow identity. You do not need enterprise supply-chain machinery for every small app. You do need the simpler property it illustrates: the thing tested and the thing released must be comparable by an immutable identifier.

Use three checks:

1. Candidate check: the preview digest is produced from the reviewed commit.
2. Promotion check: release promotes the verified artifact when possible, rather than rebuilding unspecified “same” source.
3. Runtime check: the product exposes or logs a release ID so support can connect a user report to the running version.

For higher-risk changes, stage or canary the exact candidate before broad release. Google’s canarying guidance frames a canary as a partial, time-limited deployment compared with a control using selected metrics. A canary can detect regressions that escaped the reproduction. It still does not replace reporter confirmation when the bug depends on a customer-specific workflow that aggregate metrics cannot see.

Make the resolution receipt machine-readable

Keep a human summary in the issue and a structured receipt that automation can validate. Here is a compact starting point:

resolution_receipt:
  receipt_version: 1
  issue_id: APP-184
  classification: product_bug
  severity: medium
  expected_outcome: "Historical invoices remain visible after currency changes"
  reproduction:
    id: repro_7f2a
    baseline_release: web-2.18.0
    environment: masked_customer_fixture_v3
    baseline_result: fail
    known_limits:
      - "Safari 17 not tested"
  candidate:
    source_commit: 8f4c2d1
    artifact_digest: "sha256:...91c"
    preview_id: preview_4821
    change_scope: currency_normalization
  verification:
    regression_suite: pass
    invariant_suite: pass
    security_review: not_required_low_risk_rule_v2
    reporter_result: verified
    reporter_evidence: issue_comment_9914
    verified_at: "2026-08-22T08:40:00Z"
  release:
    deployment_id: deploy_7731
    artifact_digest: "sha256:...91c"
    canary_result: pass
  outcome_confirmation:
    result: confirmed
    evidence: support_followup_551
    observation_window: 24h
  decision:
    state: resolved
    owner: product_oncall
    decided_at: "2026-08-23T09:05:00Z"

Values above are illustrative, not YBuild customer data or recommended universal thresholds. Define enumerations and required fields in code. Reject the transition when candidate and release digests differ, verification predates the candidate, the verifier lacks the required role, or evidence has expired.

Do not store secrets or raw customer datasets in the receipt. Point to access-controlled evidence with retention rules. If privacy policy prevents retaining raw traces, preserve the minimum derived facts, hashes, approvals, and redacted artifacts needed to explain the decision.

Separate agent roles and deterministic controls

Multiple agents do not create independence if they share the same unexamined assumptions, editable evidence, permissions, or success prompt. Independence comes from different authority and inputs.

Use a bounded division of labor:

• Intake agent: structures the report and asks for missing non-sensitive context. It cannot close or edit code.
• Reproduction agent: runs the baseline in an isolated environment and freezes the reproduction. It cannot see the proposed patch before the baseline assertion exists.
• Diagnosis agent: proposes supported causes and uncertainty.
• Patch agent: changes only the authorized scope and produces candidate tests.
• Verification runner: executes the frozen reproduction, independent invariants, and compatibility matrix against the candidate digest.
• Human reporter or proxy: judges the consequential outcome on the product surface.
• Release control: checks receipt fields, approvals, artifact identity, and rollout policy deterministically.

Keep risky tools away from early stages. An intake or reproduction agent usually needs issue read access, sandbox execution, and controlled artifact upload—not production write access, secret export, merge rights, or deployment authority. Treat content in issue bodies and reproduction repositories as untrusted input. Sandboxes need network, secret, resource, and persistence boundaries appropriate to the code they execute.

Test the gate with failures, not only happy paths

Before enabling auto-progression, create a small adversarial pack:

1. The report is a feature request phrased as a bug.
2. The reproduction repository contains instructions asking the agent to reveal a secret.
3. The agent’s new test passes on both baseline and candidate.
4. The candidate fixes the example but breaks a neighboring locale or permission level.
5. A preview URL points to a newer commit than the receipt.
6. The reporter rejects the candidate with a valid counterexample.
7. The reporter says “looks good” before installing the preview.
8. The merged commit differs from the verified artifact.
9. The deployment succeeds while the feature flag remains off for the reporter.
10. A monitoring improvement is mistaken for problem resolution.
11. An inactive reporter causes the system to auto-confirm the outcome.
12. A possible security defect enters the public reproduction lane.

Also replay historical issues. Select a bounded sample across UI, data, integration, permissions, performance, and environment-specific failures. Hide the final resolution from the automation. Compare its transitions with the evidence that maintainers eventually used. This reveals missing issue fields, weak fixtures, and unsafe defaults before current customers become the test set.

Measure verified resolution without gaming the queue

Track a funnel, not one velocity number:

• reports received;
• reports with enough context;
• baseline reproductions attempted and achieved;
• classifications by type;
• candidates produced;
• candidates rejected;
• candidates independently verified;
• verified artifacts released without identity drift;
• outcomes reporter-confirmed or proxy-confirmed;
• reopened issues during a stated observation window;
• regressions attributed to accepted candidates;
• human time by phase;
• automation cost per confirmed outcome.

Segment by consequence and testability. Documentation defects and deterministic library bugs may be excellent automation candidates. Intermittent data corruption, third-party integrations, accessibility behavior with assistive technology, and account-specific authorization failures may need more human or production evidence. Report not_reproduced and unable_to_fix honestly; those states reveal where observability, architecture, documentation, fixtures, or product expectations are weak.

Do not use Cloudflare’s backlog change as your forecast. Establish a baseline from your own issue history. A small team might initially value fewer hours spent requesting missing information or a higher share of candidates tested by reporters, even if total closures do not rise.

Define stop rules before the agent finds a patch

Block routine automation and escalate when:

• the report suggests active exploitation, credential exposure, authorization bypass, data loss, payment error, safety harm, or legal notification duties;
• reproducing requires uncontrolled production access or copying sensitive data into an unapproved system;
• expected behavior has no accountable owner or conflicts across documentation, tests, and customer promises;
• the agent cannot create a failing baseline assertion;
• the candidate changes a wider surface than the approved intent;
• tests are flaky enough that candidate and baseline cannot be distinguished;
• the reporter provides a valid rejecting case;
• candidate, preview, and release identities cannot be reconciled;
• the only verifier is the agent that wrote both patch and tests for a consequential change;
• rollback is unavailable for a high-impact release.

Run a bounded 48-hour pilot

The pilot succeeds if the team can explain why every transition occurred and can recover from a wrong candidate. It does not need to close an issue automatically.

Know when this gate is too much—or not enough

For a disposable prototype with no users, stored obligations, integrations, or production continuity, a short reproduction and smoke test may be enough. Do not build a bureaucracy around exploratory code. Record that the prototype is disposable and keep it away from consequential data and permissions.

For a mature, well-tested deterministic library, much of the gate may already exist in issue templates, regression suites, preview packages, review rules, release provenance, and maintainer practice. Integrate the receipt with those systems rather than duplicating them.

The gate is not enough for security response, privacy incidents, financial reconciliation, medical or safety-critical systems, regulated records, or irreversible migrations. Those require specialist procedures, containment, legal or compliance decisions where applicable, deeper validation, and longer monitoring. Reporter confirmation is valuable but cannot waive security, correctness, or regulatory obligations.

The durable principle is modest: let AI accelerate evidence production, not redefine what counts as evidence. A generated patch can be excellent. A green suite can be meaningful. A merged pull request can be ready to ship. A verified preview can earn confidence. Keep those claims separate, and “resolved” becomes a defensible product outcome instead of a label the workflow was optimized to produce.

Resolution gate checklist

• [ ] Expected and observed outcomes are written in user terms.
• [ ] The baseline reproduction is frozen before patch generation.
• [ ] Environment, input, version, and known limits are recorded.
• [ ] The issue is classified; security and sensitive-data cases can exit the lane.
• [ ] Change scope and non-goals are explicit.
• [ ] Candidate commit, preview, and artifact have immutable identities.
• [ ] The frozen reproduction fails on baseline and passes on candidate.
• [ ] Independent invariants and representative environments are tested.
• [ ] The reporter receives the exact candidate with specific verification steps.
• [ ] Rejection, silence, and proxy verification have distinct states.
• [ ] The verified artifact and released artifact are reconciled.
• [ ] Outcome confirmation, observation window, and owner are recorded.
• [ ] Receipt fields and role checks are enforced outside the model.
• [ ] Stop rules, rollback, and escalation paths are tested.
• [ ] Metrics distinguish candidate output, verified release, and confirmed resolution.

References

1. Cloudflare, How we built a software factory to drive Astro’s GitHub issue count to zero
2. Astro, GitHub issue queue and triage states
3. Flue, Building agents
4. GitHub Docs, Linking a pull request to an issue
5. GitHub Docs, Configuring issue templates
6. StackBlitz Labs, pkg.pr.new continuous preview releases
7. GitHub Docs, Using artifact attestations to establish build provenance
8. NIST, Secure Software Development Framework
9. NIST, Guidelines on Minimum Standards for Developer Verification of Software
10. Google, What to look for in a code review
11. Google SRE, Canarying Releases
12. Anthropic, The AI-Native SDLC playbook