Compact-Bench: Can Your AI Builder Resume Without Losing Product Progress?

An AI builder can remember your app's name, describe the right feature, and still restart the wrong piece of work. Compact-Bench, a research paper first submitted October 22 and revised October 28, gives this familiar frustration a sharper test: compare what an agent finishes with its full conversation against what it finishes after that history is replaced by a summary.

This matters if you are a nontechnical founder building an app over several sessions. A saved project and a readable recap do not establish that tomorrow's work will start where today's verified progress ended. You may spend another session explaining a decision, watching a rejected approach return, or discovering that a feature described as finished was never checked.

Our recommendation is to buy and manage continuity as an observable delivery capability. Before committing a larger milestone to an AI builder, ask it to resume one unfinished feature from a small, evidence-backed handoff. This article supplies the handoff sheet, a practical demonstration, and a way to separate necessary discovery from avoidable rework. These are proposed product-management methods, not results from our own builder trial. We reviewed the research and official documentation; we have not measured a commercial platform's failure rate.

1. What the study changes about the question you ask

Compact-Bench's paired design holds the repository state constant while comparing full-history and summary-only continuations. Its task track covers 35 repairs; separate tracks examine recurring details in 146 sessions (143 real and three simulated) and successive compactions in 106 production chains. The authors find that retaining relevant strings does not necessarily preserve the progress needed to finish. These are author-run coding experiments, not a measured reliability rate for your app builder.

That distinction gives a founder a useful purchasing question: “Show me that an interrupted project resumes toward the same accepted outcome.” Asking “How much context does your model have?” is less decisive. A larger window may delay compression, but it does not tell you whether a resumed session sees the right files, understands an unfinished decision, or notices that evidence belongs to an older build.

The opportunity is not to demand an impossible promise that an agent will never repeat a step. Real development includes rereading code and checking assumptions. Instead, make the starting point visible. Which parts of the feature are implemented? Which are demonstrated? What remains unverified? What action should happen next, and what observation would change that action?

This is also a better way to interpret a polished progress message. “The booking workflow is mostly complete” bundles many different states into a reassuring phrase. The screen may exist while availability checks still use sample data. An email may be drafted while delivery remains untested. A founder can ask for these distinctions without reading every line of code.

Treat the paper as a reason to inspect continuity, not as a ranking you can paste into a vendor comparison. A coding repair under a research budget and a hosted builder maintaining a full project are different systems. The useful transfer is the question and comparison discipline; the product conclusion still needs evidence from your own workflow.

2. Separate saved files, remembered context, and verified progress

A context window is the material a model can consider during a call. Compaction replaces a longer working history with a shorter representation. A checkpoint is a saved state from which a system may resume or restore. Those terms describe mechanisms; none automatically means that a product milestone has been accepted.

A founder should distinguish three layers. The first is the project itself: files, data, settings, and integrations. The second is the agent's working explanation of that project. The third is your record of what has actually been demonstrated. If the project changes but the explanation stays old, the agent can confidently continue from a false premise. If the explanation is accurate but your acceptance record is absent, you cannot tell whether “done” means implemented or verified.

LangGraph's persistence documentation distinguishes thread-scoped graph checkpoints from stores used across threads. That is a concrete example of saved state having a defined scope. It does not establish that every platform persists every aspect of a project. Ask your builder what its saved project includes and what a new session must reconstruct.

For a booking app, the files might contain a cancellation screen, the recap might say cancellation is implemented, and the last demonstration might only show a button opening a dialog. These statements can all be true while the customer outcome remains incomplete. The missing evidence is whether a confirmed cancellation releases the slot and produces the intended user-visible result.

Keep stable project conventions separate from the changing handoff. Claude Code's memory documentation describes persistent instructions and automatically accumulated notes as context, rather than enforced configuration. In a similar spirit, “use the existing design system” belongs in stable project guidance. “The cancellation dialog opens, but slot release is unverified in preview revision B” belongs in the current progress record. Combining everything into one growing memory document makes stale status harder to spot.

3. Follow one unfinished feature across a realistic interruption

Consider a fictional founder building a small workshop-booking service. Customers choose a session, reserve a place, and receive a confirmation. The founder now wants cancellation to reopen capacity. The builder has added the dialog and connected an endpoint, but the preview has not yet demonstrated that capacity changes correctly. Work stops because the founder needs to leave.

A weak handoff says: “Cancellation implemented; polish and deploy next.” The next session may reasonably focus on labels, spacing, and deployment. The trouble began before the restart: the status statement promoted implementation into completion. Compression can preserve that inaccurate statement perfectly and still steer the work away from the missing check.

A useful handoff says: “The dialog and endpoint exist in preview revision B. Opening and dismissing the dialog were demonstrated. Confirmed cancellation has not been checked against capacity. Next, use a test booking, cancel it, and inspect the session's available places before any deployment decision.” It names an observable gap and a next action tied to that gap.

Now introduce a complication: during the break, a teammate changes capacity handling. The old handoff is no longer sufficient, even if every sentence was once correct. The resumed builder must compare the project revision and refresh relevant evidence. The handoff should prompt inspection, not force an obsolete plan to continue unchanged.

A second complication is an uncertain external result. Suppose the cancellation request timed out. The handoff must say that its outcome is unknown and point to the test booking. Repeating the request without checking could confuse the experiment or create duplicate follow-up messages. This is a different state from “not started,” and it deserves a different next step.

Nothing in this scenario is a customer result or a platform test. It illustrates the kind of unfinished work to choose for a demonstration. Use a feature with a visible screen, a stored state change, and one unresolved check. A purely cosmetic task will not reveal whether the builder can preserve progress through a meaningful dependency.

4. Use this progress handoff sheet before ending a session

The reusable artifact below is a founder-facing record, not a runtime schema or a guarantee that a model will follow instructions. Keep it with the project and link to evidence that you can actually reopen. One completed sheet should describe one feature or tightly connected slice of work. Avoid copying the entire conversation into it.

Write the rejected approach carefully. “Never use client-side state” is an unjustified general rule. “The earlier client-only count did not survive refresh in revision A” preserves a useful observation without forbidding a future design that solves the actual problem. Negative knowledge should prevent pointless repetition while remaining open to new evidence.

Evidence can be a short recording, a preview URL plus a reproducible instruction, or a test report that names its revision. A screenshot alone might show the dialog but cannot establish that capacity was released. Use evidence suited to the claim. A technical collaborator can attach automated checks; the founder still needs a visible explanation of the customer outcome.

Do not put secrets, private customer examples, or raw access tokens into this sheet. Use test records and references to access-controlled systems. Give the record a clear last-updated time, then replace stale status rather than stacking contradictory updates beneath it. The aim is a small current account of progress with a traceable history, not another document the founder must summarize every morning.

5. Ask for a resumption demonstration before expanding the milestone

Start in a test environment with a recoverable project copy. Choose the unfinished feature and save its current revision. Show what already works, identify the unresolved check, and write the sheet before ending the session. Do not let the resumed session invent a retrospective account of what the earlier session supposedly verified.

Then resume using the platform's ordinary workflow. It may retain a transcript, generate a recap, or open a new session linked to the project. Ask the builder to identify the current revision, state the next unresolved check, and explain whether anything has changed since the handoff. Only then let it continue the feature. This tests the experience you will actually use rather than an artificial prompt that supplies every answer at the last moment.

Observe behavior, not just the opening explanation. Does it inspect the relevant current state? Does it preserve working parts? Does it investigate the unfinished check? Does it ask you to repeat a product decision already recorded? Does it return to a discarded approach without explaining new evidence? A good recap followed by the wrong work is a failed demonstration of continuity.

For a fair comparison, ask a technical collaborator whether the platform supports two isolated copies from the same saved state: one with the ordinary retained history and one with the supported handoff or compressed-history path. Keep the task and available working time comparable. This borrows the study's comparison principle without claiming to reproduce its benchmark. If you cannot isolate the starting state or control history, report a workflow observation, not a causal estimate of compaction damage.

The public benchmark repository provides a runner and separates task completion from detail-retention tracks. The dataset card now describes frozen histories and matching repository snapshots. A technical team can inspect those materials for research reproduction. A founder does not need to run them to demand a credible product demonstration, and passing our proposed demonstration does not imply passing the research benchmark.

6. Measure avoidable rework without punishing sensible verification

Reopening a file is not automatically wasted work. The file may have changed, the earlier observation may be incomplete, or a fresh check may be the safest next step. Count rework by its reason and consequence. Otherwise a team may discourage exactly the verification that makes the project trustworthy.

Use four practical categories. Freshness work checks whether prior evidence still applies. Necessary discovery investigates a new uncertainty. Recovery work reconstructs information that should have been available. Regression work repairs previously verified behavior that the continuation broke. These categories are our proposed management vocabulary, not labels measured in Compact-Bench.

For each session, record elapsed time, actual platform usage where available, founder explanation time, and the accepted outcome. Keep a separate note of recovery and regression work. A session that consumes fewer credits while requiring the founder to rebuild the task history may be more expensive overall. A session that spends longer checking a changed integration may be worth the cost.

Avoid false precision. Do not put an hourly value on founder time unless the team has chosen a defensible planning assumption. You can compare minutes, charges, and completed checks separately. If you later combine them, show the assumption and the uncombined values so a different reader can make a different tradeoff.

Use all attempted continuations as the denominator when reporting completion. Include abandoned and blocked sessions with their reasons; do not remove them because the builder never reached its final answer. Also distinguish “builder says finished,” “founder observed the outcome,” and “ready for release.” Those are separate events. A successful continuation can produce a verified test result while still uncovering a reason to delay deployment.

A small demonstration is a purchasing signal, not a reliable estimate of rare failures. Repeat it on different kinds of work before delegating larger milestones: a stateful feature, a changed requirement, and an interruption near an uncertain tool result are useful candidates. Choose the cases from your product rather than claiming that an arbitrary number of trials certifies reliability.

7. Know what a checkpoint can restore before relying on it

A checkpoint button can mean different things. It might restore the conversation, the code, or both. It might omit database changes, configuration outside the project, or actions in another service. Ask for a plain-language inventory of what comes back and what remains as it is.

Claude Code's checkpoint documentation offers a concrete example: conversation and code restoration are distinct options, and file tracking has limits, including files changed only through shell commands. This is a documented boundary for that tool, not evidence about every hosted builder. Your platform's recovery boundary needs its own answer.

For code-based projects, Git status helps reveal tracked differences and untracked files. A clean-looking preview is not a substitute for knowing which changes are saved. Git revert records changes that reverse earlier commits; it does not reverse customer emails or restore every external system. A founder should ask a collaborator to explain the recovery plan rather than run destructive cleanup commands from a generic guide.

Runtime resumption has related limits. LangGraph's interrupt documentation explains that a resumed node can run again and that side effects need appropriate handling. Idempotency means repeating an operation does not create an additional effect beyond the intended one. This is why “resume where you left off” must distinguish rebuilding a screen from repeating a notification, payment request, or data write.

Return to the fictional booking app. Restoring yesterday's files does not tell you whether a cancellation request completed during a timeout. Check the test booking's authoritative record before repeating the request. For a real product, define who can inspect external effects and who can repair them. A founder's handoff sheet points to that evidence; it cannot replace the underlying recovery mechanism.

This section is not a reason to demand a custom orchestration stack for every prototype. It is a reason to keep the first demonstration reversible and to avoid treating a code restore as a complete business rollback.

8. Choose the right delivery agreement for the builder you have

If a builder resumes the feature with current evidence, completes the unresolved check, and preserves the working parts, expand responsibility gradually. Give it the next bounded slice of the customer journey. Keep the handoff attached to the milestone so progress remains understandable when another person or session takes over.

If files persist but the agent needs repeated explanation, use shorter work packages and a founder-owned progress sheet. That may still be an acceptable tool for a small product. The tradeoff is that you are buying implementation assistance while retaining more coordination work. Price and schedule the project accordingly instead of assuming autonomous continuity.

If the platform cannot identify the project revision or distinguish implemented from verified work, restrict it to drafts and isolated experiments until that gap is resolved. Ask for evidence of the missing capability in a test project. Avoid migrating a live workflow merely because the sales demo resumed a visually simple page.

If the continuation repeatedly breaks accepted behavior, narrow the feature boundary and improve acceptance evidence before increasing the model budget. A more expensive model could help, but it cannot establish what “done” means when the milestone itself is ambiguous. Conversely, a well-written sheet cannot repair a platform that restores the wrong project state.

There are workflows where the full method is excessive. A one-shot landing-page draft with no stored state may need only a saved version and a visual review. A long-running app with integrations and several unfinished dependencies needs stronger continuity evidence. The relevant difference is the cost of losing the position in the work, not whether the founder knows how to code.

Make ownership explicit when working with an agency or a teammate. Who updates the sheet? Who confirms the revision? Who accepts the resumed result? The builder can prepare the record, but the person accountable for delivery should verify the claims that unlock the next milestone. This prevents a self-written progress message from becoming its own acceptance certificate.

9. Put continuity into your next product milestone

For your next build session, choose one feature whose completion you can observe. Write its customer outcome in ordinary language. Ask the builder to show the current revision and separate changes already present from behavior already demonstrated. Capture the smallest useful evidence, then name one unresolved check.

Before stopping, complete the handoff sheet and save it somewhere the next session can access. Leave uncertain outcomes explicitly uncertain. Preserve a recoverable project state using the platform's supported mechanism, with help if necessary. When you return, ask for an inspection of current state before new changes begin.

After the resumed session, review the result against the same customer outcome. Record necessary verification separately from reconstruction and regression. If the evidence is stale, refresh it. If the requirement changed, update the milestone openly. If the builder cannot continue, keep that failed attempt in your account of delivery rather than replacing it with a polished recap.

The decision to extend a builder's responsibility should follow these observed results. You are looking for a tool that can carry work forward with an understandable record of its position. Compact-Bench makes that question timely; your own project supplies the evidence that determines the next commitment.

References

1. Compact-Bench, paper v2
2. Compact-Bench public code
3. Compact-Bench dataset and release notes
4. LangGraph: Persistence
5. LangGraph: Interrupts
6. Claude Code: Checkpointing
7. Claude Code: Project memory
8. Git: status
9. Git: revert