Your AI Agent Worked Once. That Is Not a Launch Test

Your new support agent looks ready. In a live demo, it finds an order, checks the policy, asks for confirmation, issues the right partial refund, and writes a calm explanation. The team records the screen. A customer tries the same job the next day; the agent refunds the wrong line item after a failed lookup. A third request ends with a polished “done” message but no database change.

The demo proved that a successful path exists. It did not prove that customers can depend on the path.

That distinction became unusually concrete this week. Microsoft researchers released Thinkingbox and Thinkingbox-Bench, a sandbox and 507-task benchmark for stateful business workflows. The paper reports that its strongest tested system reached 65.36% pass@1, yet only 25.25% on its reported pass^20 measure. In plain language: occasional success and dependable repetition were very different properties in this environment. The benchmark also found failed runs that ended cleanly and made valid state-changing tool calls, so a tidy response or an error-free final call was not a reliable substitute for the intended business outcome.

This guide is for nontechnical founders, AI app builders, and small product teams deciding whether an agent may perform repeatable work such as refunds, booking changes, account support, onboarding, or internal access requests. Its central judgment is: choose the repeat count from the product’s real task frequency and consequences, then require the exact release candidate to produce the right final state, no forbidden side effects, and the right user communication across that sequence.

You will leave with a vocabulary for pass@k and pass^k, a task-frequency worksheet, a complete support scenario, an outcome verifier, a repeatability receipt, failure tests, and a ship/limit/hold decision. This is not a claim that Thinkingbox predicts your app. It is a method for replacing one persuasive demo with evidence that matches the promise you plan to make.

Understand what Thinkingbox actually measured

Thinkingbox is both an open framework and a benchmark built on it. The framework starts an isolated, MCP-compatible tool session, runs a simulated user and an agent through a multi-turn task, records the full trace, retrieves state changes, and applies executable checks. Its open repository describes the session lifecycle: create an isolated environment, expose tool schemas, execute calls, retrieve side effects for judging, and destroy the session after the attempt.

Thinkingbox-Bench version 1.0 contains 507 workflows across retail and e-commerce, travel and hospitality, auto insurance, neobank internal IT, and consulting IT/HR. The separate data and tool-server repository publishes the executable tasks, stateful tool environments, and release instructions. Every task checks final backend state and side effects; 30 also apply a binary requirement to the final response.

The benchmark does not ask only whether a tool name and arguments look valid. A task can require the agent to locate the right record, collect missing information, observe a domain policy, perform the allowed mutation, avoid unrelated changes, and correctly complete the conversation. A valid alternative sequence can pass if it reaches the required outcome. A smooth but wrong sequence fails.

The authors evaluated 507 tasks with 20 stochastic attempts per task for each system, or 10,140 trials per system. They report four important facts:

Reported signal	What it means inside the benchmark	What it does not prove
65.36% pass@1 for the strongest tested system	Average success over single trials and benchmark tasks	A 65.36% success rate in your product
91.12% pass@20	At least one of 20 attempts succeeded for many tasks	That retrying a customer-visible mutation 20 times is safe
25.25% reported pass^20	All 20 independently sampled attempts succeed, averaged by task using the paper’s estimator	A universal production reliability level
128 tasks passed all 20 times for that system	Some benchmark tasks were consistently solved in the sampled runs	Those tasks cover your users, policies, tools, or failure costs

These are author-reported results from a newly released benchmark, not an independent reproduction. The value for a founder is not the model ranking. It is the separation of discoverability from repeatability, and the insistence that the verifier inspect the world after the agent acts.

Separate pass@1, pass@k, and pass^k

The notation is easy to misread, and the product decisions are opposites.

pass@1 asks whether one attempt succeeds. It is useful for tracking a basic capability. A single successful demo, however, is not an estimate of pass@1; it is one observation selected from an unknown set of possible inputs and trajectories. pass@k asks whether at least one of k attempts succeeds. It measures discovery through retries. This can be appropriate for private, reversible work where failed candidates are discarded by a trustworthy verifier—for example, generating several draft layouts and selecting one that passes deterministic checks. It is dangerous when every retry can send an email, change a booking, issue a refund, or create another account record. Twenty writes are not a search strategy. pass^k asks whether all k attempts succeed. The original τ-bench paper introduced the metric for repeated reliability in policy-guided tool-and-user interactions. Thinkingbox uses the same core distinction: pass@k represents finding at least one successful route, while pass^k represents dependable repeated execution.

For one hypothetical workflow with a true, stable, independent 95% per-attempt success probability, the probability of 20 clean successes is 0.95^20, about 35.8%. That arithmetic is only an intuition pump. Real failures may be correlated by an outage, ambiguous policy, stale schema, shared prompt, or the same hard customer segment. Benchmark tasks also differ in difficulty, so you must not take Thinkingbox’s aggregate 65.36% pass@1 and raise it to the twentieth power. The paper calculates reliability task by task before averaging and discusses estimator limitations.

For product work, translate the notation into plain questions:

Can this workflow succeed at all under the pinned conditions?
Can a safe verifier reject a failed candidate before any consequence escapes?
How many times will users ask the product to do this during a release window?
What must remain correct on every one of those attempts?
Which failure is tolerable, which needs recovery, and which must never reach the model’s discretion?

That last set is more valuable than copying a leaderboard number.

Choose k from product exposure, not benchmark fashion

There is no universal k=20 launch rule. Twenty was the repeated-trial design in Thinkingbox. Your repeat count should reflect the exposure you are about to authorize.

Start with a named workflow, not “the agent.” Then estimate how often it will occur before the team can detect a problem, stop the feature, and ship a correction. Call that the exposure window. If the agent will attempt eight booking changes during a two-day pilot, a five-run test understates the promise. If a drafting feature produces hundreds of private suggestions that a user must approve, requiring hundreds of flawless drafts may confuse harmless variation with consequential failure.

Use this worksheet:

Field	Founder question	Example entry
Workflow	What exact user job is being delegated?	Refund one eligible order item
Unit of exposure	What counts as one consequential attempt?	One submitted refund request
Expected volume	How many attempts before review or rollback?	20 during the first-week pilot
Maximum consequence	What can one bad attempt change?	Money and an order ledger
Reversibility	Can the team fully undo it, how fast, and with whose approval?	Reversible within one hour by support lead
Required outcome	What state and user result must be true?	Correct item refunded once; accurate receipt sent
Forbidden effects	What must remain unchanged?	Other items, accounts, inventory, credits
Detection delay	How long until the team knows it failed?	Immediate automated state check plus daily review
Test sequence `k`	What repeated sequence represents the exposure window?	20 independent reset cases plus targeted variants

A practical starting policy follows consequence, not model prestige:

Workflow type	Repeatability test	Runtime posture
Private suggestion with mandatory user review	Repeat representative inputs; score usefulness and refusal separately	Keep output non-authoritative; do not claim autonomous completion
Reversible, low-value write	Match `k` to the pilot exposure window; verify state and rollback on every run	Limit volume, surface receipts, monitor continuously
Money, permissions, identity, legal status, regulated records, or safety	Repeated success is necessary but insufficient	Put eligibility, limits, authorization, and invariant enforcement in deterministic controls; require accountable human approval where appropriate

Do not turn pass^k into a safety slogan. A perfect small sample cannot establish zero risk. It can reveal that a workflow is already too inconsistent for the promise, and it can make the release decision auditable.

Verify the terminal state and the absence of collateral effects

An agent trace tells you what the system attempted. The product outcome tells you what happened.

Thinkingbox formalizes this distinction by checking final backend state, side effects, and—where required—dialogue properties. Its documentation for writing effective tests gives a useful design rule: a good check asks whether the agent completed the task, not whether it followed one expected sequence of calls. Counting exactly two tool calls is brittle; verifying the intended state accepts alternative valid trajectories.

The same principle appears in other primary work. AWS-Bench provisions disposable cloud accounts and uses programmatic checks against live AWS state for mutation tasks, then resets the environment. That is materially stronger than accepting a terminal transcript saying a resource was created. Anthropic’s agent-evaluation guidance likewise distinguishes a task, repeated trials, graders, and the full trace, and recommends multiple graders because agent behavior has several dimensions.

For each workflow, define four layers:

Initial-state fixture: the exact records, permissions, policy version, time, feature flags, tool versions, and user facts available before the attempt.
Required state: the records and observable user outcome that must exist after success.
Forbidden state: duplicates, wrong-entity updates, over-limit values, unauthorized access, missing audit records, or misleading claims that must not exist.
Recovery state: the rollback, escalation, and user notification required when the attempt is rejected or its result is uncertain.

Test both required and forbidden state. “Refund created” is not enough if the agent refunded the wrong item, duplicated a credit, or changed the membership tier. Also test communication when it carries an obligation. Thinkingbox itself notes a limitation: 477 of its 507 tasks rely only on backend state and side effects, so a trial could make the correct change but misreport it to the user and still pass. Your product verifier should not inherit that boundary when the message affects consent, money, deadlines, or next steps.

Build a repeatability contract for one workflow

A repeatability contract is a versioned agreement between product, operations, and engineering. It states what is being tested, what counts as success, how many consecutive exposures the release is expected to survive, and what evidence binds the result to the candidate you ship.

Use this reusable artifact:

yaml

repeatability_contract:
  workflow_id: refund_single_item_v3
  product_surface: support_assistant
  release_candidate:
    model: pinned-provider-model-version
    prompt_hash: sha256:...
    tool_schema_hash: sha256:...
    policy_version: refund-policy-2026-08-20
    feature_flags: [agent_refund_pilot]
  exposure_window:
    period: 7_days
    expected_consequential_attempts: 20
    pilot_accounts: 5
  task_suite:
    representative_cases: 12
    repeats_per_case: 20
    reset_between_trials: true
    production_shadow_cases: 0
  required_checks:
    - correct_customer_and_order
    - item_is_eligible
    - explicit_confirmation_recorded
    - exact_amount_refunded_once
    - ledger_and_user_receipt_agree
  forbidden_effects:
    - unrelated_record_changed
    - duplicate_refund
    - limit_bypassed
    - success_claim_without_verified_write
  reliability_views: [pass_at_1, pass_at_k, pass_to_k, slice_failures]
  runtime_controls:
    approval: support_lead
    per_attempt_limit_usd: 100
    idempotency_key_required: true
    post_write_verification: true
    automatic_pause_on_uncertain_state: true
  evidence:
    run_ids: []
    state_diff_uris: []
    rejected_trials: []
    reviewer: null
  decision: hold

The hashes matter because reliability does not attach to a model name in isolation. A new system prompt, tool description, timeout, retry policy, user simulator, context compaction rule, or backend schema can change behavior. OpenAI’s current Agents SDK testing documentation draws a useful boundary: deterministic in-memory tests can cover orchestration the application owns, while external model, network, protocol, and sandbox behavior needs real adapters or integration environments. Your receipt should identify both kinds of evidence.

The contract also prevents a common category error: retrying an uncertain mutation is not the same as running another independent benchmark trial. In evaluation, reset the world completely. In production, use idempotency and reconciliation before retrying, because the first call may have succeeded even when the response was lost.

Walk through a concrete founder scenario

Imagine ParcelPilot, a hypothetical small SaaS product for independent online shops. It adds an AI support agent that may refund one item under $100 when the delivery is late, the item is eligible, and the customer confirms the amount. A support lead approves every refund during the first week.

The team’s demo succeeds. Instead of launching broadly, they create 12 stateful cases drawn from documented support patterns:

one eligible late item in a multi-item order;
two similar orders from the same customer;
an ineligible sale item;
a refund already pending;
a stale tracking event;
a tool timeout after the payment provider accepts the refund;
an amount near the limit;
a customer who changes the requested item;
a record with a legacy currency code;
a user who does not confirm;
a duplicated webhook;
a policy version that changed after the order date.

Each case starts from a pinned fixture and runs 20 isolated attempts against the exact release candidate. The verifier checks customer, order, item, eligibility, confirmation, amount, idempotency key, payment-provider result, internal ledger, user receipt, and untouched records. It also checks that declined or uncertain attempts do not say “refunded.”

Suppose the headline pass@1 looks strong. The rejected-trial view nevertheless shows that the legacy-currency case fails in four of 20 attempts and the timeout case creates an uncertain state in two. That is a hold, even if the overall average clears a target. The failures share a consequence that the aggregate hides: money may move without a matching, truthful product state.

The team does not solve this by telling the model to be more careful. It puts currency normalization before the agent, requires an idempotency key for the payment write, reads the payment state after a timeout, and makes uncertain a first-class outcome that pauses automation. It then reruns the entire suite with new hashes. If the candidate passes, the pilot still remains limited to five accounts, support-lead approval, a $100 deterministic ceiling, immediate post-write checks, and daily receipt review.

This scenario is not a reported customer result. It illustrates the difference between improving a prompt to win more attempts and changing the product so a stochastic mistake cannot cross every boundary.

Run failure tests that a happy-path sequence will miss

Repeatability needs variety as well as repetition. Running the same easy fixture 20 times measures sampling variation on that fixture; it does not cover a realistic task distribution. Pair repeats with targeted perturbations and consequence-focused cases.

1. Wrong-entity collision

Place similar names, adjacent order numbers, or multiple open tickets in the fixture. Fail if the agent touches any entity that the user did not identify and authorize.

2. Missing-information pressure

Omit a fact required by policy. Pass only if the agent asks, refuses, or escalates correctly. Do not reward a lucky guess that happens to match the hidden fixture.

3. Tool error after partial progress

Return a failed lookup, expired authorization, rate limit, malformed response, or timeout after a write may have occurred. Verify recovery from actual state rather than from the text of the error. Thinkingbox’s authors report that tool-usage and unrecovered tool-feedback problems dominate their diagnostic categories, averaging 77.5% of failed trials across the models in that analysis.

4. Successful call, wrong business result

Return HTTP success for an update that targets the wrong record, exceeds a limit, or omits a required linked write. This catches the false belief that a successful final tool response proves completion.

5. Clean ending, incomplete state

Let the agent produce a polished completion message after reads only, or after one of several required mutations. Fail the run if the terminal state is incomplete.

6. Correct state, misleading message

Make the backend outcome correct but the explanation wrong about amount, timing, reversibility, approval, or next step. Apply a deterministic check where possible and an independent human or calibrated rubric where meaning must be judged.

7. Repeated request and duplicate delivery

Submit the same intent twice or replay a webhook. Pass only if idempotency rules prevent duplicate side effects and the user receives an accurate status.

8. Version and segment shift

Change one consequential dimension: policy version, locale, currency, account tier, tool schema, mobile input shape, or permission set. Report the slice separately. A strong mean must not wash out the group that bears the cost.

Interpret failure clusters before changing the model

Thinkingbox’s four diagnostic groups are tool usage, no state-changing action, incomplete user resolution, and wrong state update. The paper correctly calls them observable signatures, not unique root causes. Use the same discipline with your traces.

Observed cluster	Likely product questions	Changes to test before a model upgrade
Tool/precondition failures are not recovered	Is the error structured? Can the agent distinguish retryable, rejected, and uncertain?	Typed errors, state read-back, bounded retry, explicit recovery branch
Reads succeed but no write occurs	Is completion defined? Does the agent know when authority is missing?	State machine, required-field check, escalation as a valid outcome
User issue remains unresolved	Did the conversation capture confirmation and missing facts?	Clarification policy, user-state tests, truthful partial-completion message
Wrong or extra record changes	Are entity identity and limits enforced outside the model?	Resource binding, deterministic policy, idempotency, post-write invariant checks

A larger model may improve results, but it does not replace these controls. NIST’s definition of valid and reliable AI ties reliability to performing as required over a time interval under expected conditions, and calls for realistic test sets plus ongoing testing and monitoring. That definition is closer to a product obligation than “the latest model ranks first.”

Keep the full rejected set. If you keep only successful traces, pass@k can look like progress while recurring failures disappear from review. Record why each run failed, whether the verifier was correct, what consequence escaped, and whether the repair belongs in the model, prompt, tool, policy, UI, or deterministic control plane.

Decide whether to ship, limit, or hold

Use a decision matrix that combines repeatability with consequence and verifier quality:

Evidence	Consequence boundary	Decision
Required and forbidden outcomes pass across the chosen sequence and critical slices; verifier is independent; recovery is tested	Low or reversible, with monitoring and fast rollback	Ship a bounded pilot
Average is strong but one critical slice is unstable; all actions require accountable approval and no state escapes before verification	Consequential but contained	Limit to named users, volume, and actions while fixing the slice
Only happy paths pass; failures make uncertain or wrong writes; completion is judged from text or final tool status	Money, permissions, identity, records, or external communication	Hold autonomous action
Workflow has no objective acceptance state, several legitimate resolutions, or consequences the team cannot detect or reverse	Any material consequence	Do not automate yet; redesign the job or keep a human operator

Do not approve a release only because all sampled trials passed. Confirm that the suite represents expected use, the verifier would catch the failures you care about, the candidate matches the receipt, and runtime controls constrain what the test cannot prove. Also define the production tripwire: a forbidden side effect, an uncertain write, a critical-slice failure, a verifier disagreement, or a drifted tool/policy hash should pause the capability automatically or route it to review.

For a 48-hour start:

Pick one consequential workflow and write its required and forbidden terminal states.
Estimate the first pilot’s exposure window and choose k from that volume.
Build five to twelve representative and adversarial fixtures with complete reset.
Pin the model, prompt, tools, policies, feature flags, timeouts, and retry behavior.
Run repeated trials; report pass@1, pass@k, pass^k, critical slices, and every rejected trace.
Repair the product boundary, not only the prompt, then rerun the whole sequence.
Launch only with volume limits, state verification, receipts, recovery, monitoring, and a named stop owner.

Know where the method does not apply cleanly

Thinkingbox is strong evidence for a measurement habit, not a universal production emulator. The authors state that its tasks are synthetic reconstructions from a non-public source collection and are not a random or statistically representative sample of enterprise work. Each retained task has one golden terminal state, so workflows with several defensible outcomes are excluded. Results depend on the fixed simulator, termination convention, turn limits, token limits, and other harness choices. The paper also uses a plug-in estimator for reported pass^k and explains why it differs from an unbiased estimator on sparse successes.

Your own repeatability sequence has parallel limits:

It estimates behavior only for the candidate, fixtures, environment, and sampling settings tested.
It does not prove future performance after model, prompt, policy, schema, provider, or user-population drift.
It can miss correlated failures that are absent from isolated resets, including outages, shared stale state, congestion, and cascading retries.
It cannot validate an outcome that the team has not defined or cannot observe.
It should not force one golden answer onto ambiguous human decisions.
It is not a substitute for security review, privacy controls, legal advice, accessibility testing, domain expertise, incident response, or human authority in high-impact work.

This laboratory-to-product gap is not unique to agent benchmarks. NIST AI 600-1 recommends empirically validated capability claims, documented test results for release authorities, and testing under conditions similar to deployment; it also warns that benchmark and laboratory conditions may not extrapolate to real-world use. Treat those limits as part of the result, not as a footnote to remove from the launch deck.

Use repeated reliability where the product makes a repeated promise and the outcome is meaningfully verifiable. Use exploratory research, qualitative review, user studies, or human-led operations where value is subjective or the acceptable resolution is negotiated. And when a product cannot tolerate one wrong mutation, the right answer is not an enormous k; it is to prevent the model from authorizing that mutation by itself.

Turn one success into an honest product decision

A successful agent run is valuable. It shows the workflow may be possible. A failed repeatability sequence is also valuable: it reveals that the product promise is ahead of the evidence before more users pay the cost.

Thinkingbox’s most useful contribution for a small team is not a leaderboard winner. It is a sharper receipt for completed work: the right persistent state, no collateral effects, truthful communication, repeated under the conditions the product expects. Tie k to exposure, keep discovery separate from reliability, and bind the result to the exact candidate you ship.

Then make a modest claim. Do not say “our agent is reliable.” Say which workflow, which release, which sequence, which state checks, which limits, and which recovery path earned a bounded pilot. That sentence is less impressive than a flawless demo. It is much closer to a product users can trust.

References