MCP Schema Changes Need Product Regression, Not Just Tool Coverage

Apple Research has published Agent Seer, a method that turns a Model Context Protocol (MCP) tool specification into evaluation scenarios without examples, live tool access, or domain-specific tuning. It enriches tool schemas, proposes workflows, invents mock outputs, and expands suitable cases into multi-turn dialogues. That is a useful answer to a problem every AI app builder eventually meets: connectors change faster than people maintain regression tests.

The headline is not that a schema can now certify an agent. It cannot. Agent Seer's own results show why. Across seven MCP specifications, the authors generated 337 scenarios and 391 evaluation records, but only 54 scenarios expanded into multi-turn cases. All 871 mock calls in the reported experiment had low grounding because the source specifications supplied no example outputs. The main remaining tool-calling error was not choosing the tool name; it was supplying the right argument value.

For a nontechnical founder, this creates a practical product decision. A schema change can cheaply generate test candidates, but someone must still decide whether those candidates describe the customer promise, whether the values are authorized, whether the mock behavior matches the real service, and whether the final external state is correct.

This guide supplies that missing release layer. You will get a six-stage schema-change regression contract, a change-to-test matrix, a complete scheduling-agent scenario, a reusable YAML manifest, failure drills, and ship/limit/hold rules. The method applies whether your tools arrive through MCP, another function-calling format, or a private connector catalog.

This is not a database/API migration guide or a general prompt-regression checklist. A migration guide asks whether old and new producers and consumers remain compatible. A prompt suite asks whether behavior changed after any stack update. The narrower question here is: when the effective tool specification changes, how do you turn that diff into product-valid cases without mistaking generated plausibility for customer truth? If a tool change also breaks wire or data compatibility, run both gates.

What Agent Seer Actually Adds

The Apple Research page describes a cold-start evaluation pipeline that needs only one MCP specification. A tool specification contains names, descriptions, and typed parameter schemas. Agent Seer first interprets that material, then generates simple and complex scenarios, synthesizes mock tool results, and attempts to turn substantial scenarios into grounded multi-turn conversations.

The full paper makes the contribution more precise. Each harness artifact contains a natural-language prompt, an ordered expected tool sequence with arguments, synthetic outputs, a possible multi-turn conversation, and a held-out oracle. A downstream evaluator can present the prompt to an agent, return the mock outputs when it calls tools, and compare the resulting trace with the expected workflow. No live calendar, database, browser, or source-code repository is required to create the initial case.

That changes the economics of test creation. A team adding a new connector no longer needs to start with a blank spreadsheet. A server that adds, removes, or changes tools can produce a fresh set of candidate workflows before a product manager has written every case by hand. The method can also expose less obvious combinations among tools instead of testing each function in isolation.

The reported experiment is substantial enough to take seriously and bounded enough not to overstate. It covers seven public MCP specifications across Illustrator, Selenium, Redis, Git, Elasticsearch, Slack, and filesystem domains. The authors report mean tool-calling quality of 0.911, complete tool coverage for small and medium specifications, and a 7.3 percentage-point average decline from simple to complex scenarios. They also say these are observations over seven specifications and one generation model, not universal performance claims.

The durable idea is therefore specification-to-test bootstrapping. It is not automatic product approval.

Define Four Things That Teams Commonly Call a Test

Before wiring schema changes into a release pipeline, separate four artifacts:

• A generated scenario is a proposed user goal, tool sequence, argument set, and mock response path inferred from a specification. It is a candidate test, not established ground truth.
• A product case is a generated or human-written scenario that a product owner has accepted as representative of a real customer promise, policy, and consequence.
• A trial is one execution of the release candidate against a product case with a pinned model, prompt, tool catalog, permissions, and environment.
• An outcome receipt is independent evidence that the authoritative external state matches the promised result and contains no forbidden side effect.

Anthropic's current agent-evaluation guidance uses a similar separation among tasks, trials, graders, transcripts, and outcomes. It gives a simple example: an agent can say a flight was booked, but the outcome is whether the reservation exists in the environment's database. For product teams, the lesson is direct: schema-generated cases lower authoring cost, while outcome verification establishes product truth.

A Schema Describes Shape, Not the Whole Customer Promise

The current MCP tools specification gives a tool a name, description, inputSchema, optional outputSchema, annotations, and execution metadata. Input and output schemas use JSON Schema. Servers must validate inputs and implement access controls; clients should confirm sensitive operations, show tool inputs, validate results, apply timeouts, and log tool use.

Those rules are important, but a valid schema answers only part of the product question. It can express that start_time is a string, attendees is an array, or amount must be a nonnegative number. It does not inherently prove:

• that the time is in the user's intended time zone;
• that every attendee may legally or appropriately receive the invitation;
• that the selected account belongs to the active tenant;
• that a refund amount matches the current order balance;
• that an approval still covers the changed arguments;
• that an upstream service accepted the action exactly once;
• that the user-visible result persisted after a retry or delayed failure.

This is why Agent Seer's strongest failure signal matters. Argument value accuracy dominated imperfect cases even when name and type accuracy were much better. In real products, a plausible but wrong value is often more dangerous than an obvious parse error. customer_id: "cust_204" is valid JSON and may satisfy the schema; it is still unacceptable if the active customer is cust_240.

Treat a schema as the candidate-generation surface. Product policy, authorization, live semantics, and external truth must come from other sources.

Make Every Schema Diff Produce a Test Obligation

Do not wait for a major version label. Capture the old and new tool catalog, normalize them, and classify every difference by product effect. The official MCP Registry versioning guidance requires a unique version string for each published server version and recommends aligning the server, package, and remote API versions. That helps identify releases, but semantic versioning cannot decide which customer promises changed inside your product.

Use this matrix to compile changes into minimum test obligations:

The MCP protocol can notify a client that the tool list changed, and its recent roadmap documents rapid protocol evolution across discovery, Tasks, authorization, and result types. But notifications/tools/list_changed is an invalidation signal, not a product-quality verdict. It should trigger refresh, diff, candidate generation, and review. It should never silently bless the new catalog.

Give each diff a stable ID and keep the old specification. Otherwise, when a regression appears, the team will know the server changed but not which description, constraint, or result field generated the new behavior.

Build a Six-Stage Regression Contract

The release contract should move from cheap, broad evidence to expensive, realistic evidence. No stage can certify what the next stage exists to measure.

1. Pin and diff the effective catalog

Record server version, package or image digest, remote API version, MCP protocol revision, tool-list hash, model version, agent prompt, client policy, and evaluator version. Diff the catalog the agent actually received, not only the repository source. Cache rules, feature flags, permissions, and tenant configuration can make the effective catalog different from the nominal release.

2. Generate candidate scenarios

Use the changed schemas to propose normal, boundary, misuse, recovery, and multi-tool workflows. Favor scenarios that exercise new required fields, ambiguous values, neighboring tools, and changes in output shape. Agent Seer demonstrates that this stage can begin without live access, but its cases should be labeled with grounding level and generator version.

3. Apply a product-truth veto

A product owner or domain expert must accept, edit, or reject each case. Require an explicit user promise, permitted actor, authoritative data source, forbidden effects, approval rule, and expected final state. Reject cases that are merely plausible. Add cases the schema cannot infer: contractual limits, tenant boundaries, local business rules, accessibility needs, human escalation, and delayed outcomes.

4. Run deterministic simulation

Replay the exact release candidate against controlled mock data. Grade tool necessity, selection, order, argument name, argument value, type, format, and relevance separately. Verify forbidden calls and state transitions, not just the expected happy path. Repeat stochastic trials and preserve full traces. Synthetic outputs must be internally consistent across calls; if a first call returns event ID evt_41, later calls must operate on evt_41 rather than a newly invented object.

5. Run a restricted live shadow

Exercise real authentication, latency, error shapes, pagination, service limits, and response semantics without allowing customer-visible effects. Reads can query a seeded test tenant. Writes should target a disposable sandbox, require a nonproduction credential, or be intercepted before execution. Compare real results with mock assumptions and record where the specification omitted behavior.

6. Verify external outcome and decide exposure

For any write workflow, query the authoritative system independently after execution. Confirm required state, forbidden state, duplication count, identity, timestamp, and recovery status. Then choose ship, limited pilot, hold, or rollback. The agent that issued the command should not be the only component declaring success.

This sequence is intentionally asymmetric. Generated cases may enter cheaply; production authority exits only after reviewed truth and external verification.

A Concrete Scenario: When a Calendar Tool Adds time_zone

Consider CedarMeet, a hypothetical AI scheduling app built by a three-person team. Its MCP server originally exposed:

{
  "name": "create_meeting",
  "inputSchema": {
    "type": "object",
    "properties": {
      "start": {"type": "string"},
      "duration_minutes": {"type": "integer"},
      "attendee_emails": {"type": "array", "items": {"type": "string"}}
    },
    "required": ["start", "duration_minutes", "attendee_emails"]
  }
}

A new version adds required time_zone, optional send_invites, and a structured result containing event_id, status, and invite_delivery. This looks like a clean improvement. It also creates several ways to break the customer promise.

A scenario generator proposes: “Schedule a 30-minute call with Mei tomorrow at 9.” It chooses create_meeting, supplies time_zone: "Asia/Tokyo", and sets send_invites: true. The trace is coherent and the arguments satisfy the schema.

The product-truth review rejects the case as written. The user's time zone is known, but “Mei” maps to two contacts. The product policy requires clarification before sending an external invitation. “Tomorrow at 9” also depends on the user's locale and current date. A schema can suggest plausible values; it cannot infer CedarMeet's authority policy.

The reviewed suite now contains six cases:

1. Known contact and explicit time zone: create one event and send one invitation.
2. Ambiguous contact: ask a question; call no write tool.
3. Missing time zone with reliable profile data: show the interpreted zone before approval.
4. Missing time zone without reliable data: ask; do not default silently.
5. Tool timeout after server acceptance: query by idempotency key before retrying.
6. status: "created" with invite_delivery: "failed": report partial success and offer a bounded recovery path.

Nothing in this scenario requires the founder to implement a benchmark framework. The founder's job is to make the customer promise, ambiguity policy, approval boundary, and final evidence explicit. Engineering can then automate the contract.

Use This Reusable Regression Manifest

The manifest below is a YBuild product artifact, not an Agent Seer or MCP standard. Keep it small enough for product, operations, and engineering to review together.

mcp_change_regression:
  change_id: "calendar-mcp-2.4.0-to-2.5.0"
  owner: "product-reliability"
  effective_stack:
    server_version: "2.5.0"
    server_artifact_digest: "sha256:..."
    remote_api_version: "record-exact-version"
    protocol_revision: "2026-07-28"
    tool_catalog_hash_before: "sha256:..."
    tool_catalog_hash_after: "sha256:..."
    model_and_settings: "provider/model/version/settings"
    agent_prompt_version: "scheduler-17"
    client_policy_version: "writes-9"

  schema_diff:
    tools_changed: ["create_meeting"]
    required_added: ["time_zone"]
    optional_added: ["send_invites"]
    output_added: ["event_id", "status", "invite_delivery"]

  generated_candidates:
    generator: "name/version"
    generator_input_hash: "sha256:..."
    total: 24
    grounding: "spec-only"
    accepted_by_product_owner: 11
    edited: 8
    rejected: 5

  product_truth:
    user_promise: "create the approved event once at the interpreted time"
    authority_source: "user approval bound to previewed arguments"
    authoritative_system: "calendar provider event record"
    forbidden_effects: ["wrong contact", "silent timezone default", "duplicate event"]

  gates:
    schema_validation: pass
    deterministic_simulation: pass
    repeated_trials: "5 per critical case"
    restricted_live_shadow: pass
    external_outcome_check: pass
    unresolved_argument_value_failures: 0

  decision:
    posture: "limited-pilot"
    eligible_tenants: ["internal-dogfood"]
    expires_at: "2026-09-05T00:00:00Z"
    rollback_to: "calendar-mcp-2.4.0"
    human_approver: "named-person"

Do not replace unknown values with estimates that look authoritative. A missing remote API version or unverified final state should remain unknown and may block the release. Record how many generated cases were rejected; rejection rate is useful evidence about schema completeness and generator fit, not wasted work.

Grade Arguments and Effects More Strictly Than Tool Names

Agent Seer's failure analysis is the strongest reason not to use tool coverage as the release metric. The authors decompose tool calling into necessity, selection, ordering, and arguments, then split arguments into completeness, name, value, type, format, and relevance. Argument value accuracy was the dominant sub-failure under both the primary and out-of-family judges.

Build the product dashboard in the same spirit:

OpenAI's official trace-grading guidance describes structured scoring of an end-to-end agent trace to find regressions and understand where behavior failed. Use that visibility, but do not let a trace grader substitute for a state check. A trace is evidence of the path; the provider record, database row, delivered message, or changed file is evidence of the effect.

For high-consequence values, use deterministic assertions before model judgment. Exact tenant ID, currency, amount ceiling, timestamp conversion, recipient, resource ID, and approval hash should not depend on whether a judge finds them plausible. Use a model grader for qualities that require interpretation, such as whether a clarification was understandable or whether the agent chose an unnecessarily powerful workflow. Calibrate those judgments with humans.

Run Eight Failure Drills Before Promotion

Every changed write-capable tool should survive at least these drills:

1. Valid shape, wrong identity: substitute another tenant's schema-valid resource ID. The client or server must reject it before effect.
2. New required value missing: verify that the agent asks or uses an explicitly authorized source rather than inventing a default.
3. Description-only authority expansion: change wording from “draft” to “send.” Confirm that client policy still requires the correct approval.
4. Partial success: return an authoritative object plus a failed secondary effect. The product must show a truthful partial state.
5. Timeout after acceptance: hide the initial response. The agent must reconcile via idempotency or lookup before retrying.
6. Output field removed or null: verify safe parsing, visible degradation, and no false success statement.
7. Tool-list drift mid-session: invalidate the cached catalog and ensure old arguments are not sent blindly to the new tool.
8. Mock/live disagreement: make the real sandbox return an undocumented status or pagination path. The release must hold until the contract covers it.

Know When Spec-Generated Testing Fits, and When It Does Not

This approach is a strong fit when tools have reasonably descriptive schemas, the team lacks an initial test set, tool catalogs change often, live calls are expensive or risky, and product outcomes can later be verified. It is especially useful for generating breadth: forgotten tools, changed fields, multi-tool combinations, and boundary candidates.

It is a weak fit as the sole evaluator when meaning lives outside the specification. Sparse descriptions, undocumented remote behavior, dynamic authorization, visual interfaces, physical devices, long-delayed outcomes, subjective quality, and cross-organization policies all need additional evidence. Agent Seer's experiment did not use live tools, had no example outputs for its seven specifications, and has no systematic human-evaluation anchor. The paper explicitly calls LLM-generated ground truth its most significant limitation.

Multi-turn confidence should also stay bounded. Only 54 of 337 generated scenarios expanded into multi-turn cases, with expansion heavily concentrated in complex scenarios. That is useful research evidence, not proof that a generated suite covers real customer repair, interruption, negotiation, or recovery conversations.

The NIST AI Risk Management Framework treats testing, evaluation, verification, and validation as work across the full lifecycle, including production integration, user experience, monitoring, incident handling, and subject-matter-expert recalibration. A spec-generated suite fits inside that lifecycle. It does not replace it.

Choose Ship, Limit, Hold, or Roll Back

Use four release postures:

Never average away an identity or authorization failure. A 98% scenario pass rate does not compensate for one cross-tenant write. Set hard gates for forbidden effects and report the remaining suite as supporting evidence.

Also preserve the previous effective catalog and client behavior. The August 2026 MCP roadmap shows how quickly protocol features, discovery, Tasks, authorization, and result types continue to evolve. A rollback that restores only the server package while leaving a new cache, prompt, policy, or remote API in place is not a true rollback.

A 48-Hour Plan for a Small Product Team

In the first four hours, export the effective tool catalog from production and the proposed release. Pin all stack versions, normalize both files, produce a semantic diff, and mark every write-capable tool. Do not start by asking a model to invent tests without preserving the exact inputs.

By hour 12, generate candidate cases for each changed surface. Label them normal, boundary, misuse, recovery, or multi-tool. Ask the product owner to accept, edit, or reject them and to add the customer promise, approval source, forbidden effects, and authoritative outcome for every retained write case.

By hour 24, run deterministic simulations with seeded identities and state. Repeat critical cases, inspect argument-value failures, and compare full traces. Convert every discovered ambiguity into either a clarification rule, deterministic resolver, narrower permission, or release blocker.

By hour 36, run restricted live shadow against a disposable tenant. Exercise real authentication, errors, timeouts, pagination, output variants, and service limits. Update mocks where they disagree with the real integration; never edit the real observation to match the convenient mock.

By hour 48, verify external outcomes, complete the manifest, choose a release posture, name the approver, set an expiry for limited exposure, and test rollback. Schedule a post-release review using real failures and support cases to improve the permanent suite.

The Founder Decision

Agent Seer is valuable because it makes the empty-test-suite problem smaller. A changing MCP specification can become a stream of structured scenario candidates instead of a vague request for someone to “retest the agent.” That is a real improvement for small teams.

The product boundary remains human-owned. A schema does not know which customer promise matters, which ambiguity requires a question, which value carries authority, which external record is final, or which failure is unacceptable. Generated coverage becomes release evidence only after product review, controlled trials, live semantic checks, and independent outcome verification.

Adopt the method when it accelerates the first draft of regression work. Do not let it collapse candidate generation, ground truth, execution, and product approval into one green score. The safest useful rule is simple: let schema changes generate questions automatically; make the product earn its answers before the changed tool reaches a customer.

References

1. Apple Machine Learning Research — Agent Seer
2. Agent Seer paper — arXiv:2608.26133
3. Model Context Protocol — Tools specification, 2026-07-28
4. Model Context Protocol Registry — Versioning published servers
5. Model Context Protocol — August 2026 roadmap
6. JSON Schema — Draft 2020-12 release notes
7. Anthropic — Demystifying evals for AI agents
8. OpenAI — Trace grading
9. NIST — Artificial Intelligence Risk Management Framework 1.0