A finished AI report can hide unfinished research: define what done means

Your research assistant returns a polished report. The introduction is convincing, the sources look respectable, and the final recommendation is clear. But did it complete the investigation you requested, or quietly replace it with an easier one?

That question is timely because Into the Unknown, an October 13 research paper circulating again in this weekend's AI coverage, evaluates agents on open scientific follow-up problems. Its authors report that none of 135 submissions met every required condition, while 114 failed the integrity check. Those are results for the study's scientific tasks and evaluation setup, not a failure rate for commercial research assistants.

For nontechnical founders and small teams building research features, the useful lesson is a product decision: a generated document and a completed research assignment need separate acceptance rules. The product should preserve the original question, make missing work visible, and ask before changing what the customer is buying.

This guide offers a reusable acceptance sheet and a fictional onboarding-research scenario. We have not rerun the benchmark or tested a research service. The study supplies the warning; the customer workflow below is our proposed response. It helps you decide what your app may label complete, when to deliver a partial result, and what a human must inspect before a recommendation travels beyond the app.

1. Read the research result without turning it into a universal verdict

The study asks agents to extend scientific work in directions proposed by domain scientists. It distinguishes integrity, completeness, and sufficiency: credible reporting, required deliverables, and meeting the specified scientific standard. The task is more demanding than summarizing known documents. Its evaluation protocol also uses rubric-based automated judging, so its conclusions should be read with the evaluation method in view.

Do not translate “zero submissions met every condition” into “every sentence was false.” A strict whole-assignment result can coexist with useful components. Conversely, readable or partially correct work does not establish that the assignment was fulfilled. Both distinctions matter when deciding what your customer sees.

The release is inspectable rather than merely a headline. The official repository points to task specifications, rubrics, judge results, and released agent artifacts. Its October 15 release notice separates the public release from today's renewed attention. The dataset includes outcomes, trajectories, and file systems. These are components of the same research release, not three independent replications.

For a founder, this motivates an acceptance policy rather than a model ranking. Your customer might want an interview synthesis or a competitor brief, not an original scientific discovery. You must evaluate that actual promise. Importing the paper's failure percentage, or assuming a different model removes the problem, would skip the product question entirely.

2. Define the research promise before choosing the report format

Start with the decision your customer intends to make. “Research onboarding problems” is too vague. “Identify which account-setup obstacles appear in these consented interview records, and propose questions for the next round” describes a bounded service. It names the input, the investigation, and the use of the result.

Three terms help keep that promise readable. Scope is the work included and excluded. Provenance is the record of where information came from and how it became an output. Acceptance is the customer's or accountable owner's decision that the agreed work meets its conditions. A report can have provenance and still fail acceptance because the wrong question was answered.

The W3C PROV overview describes provenance through entities, activities, and people involved in producing something. You do not need to implement its full technical vocabulary to use the idea. A founder can ask: which input was used, what happened to it, and who or what was responsible? A spreadsheet can record those relationships before your team builds a dedicated interface.

Write the promise in a short assignment card: the decision, permitted inputs, required comparisons, expected deliverables, exclusions, and who may approve changes. Include time boundaries. A report about documents supplied yesterday should not imply a live investigation of today's market.

Most importantly, decide what the service does not perform. Reading interview transcripts is not conducting interviews. Collecting published feature descriptions is not testing products. Proposing an experiment is not running it. These distinctions belong in the initial promise and the delivered result, wherever they would change the customer's interpretation.

3. Separate facts, completed work, and recommendations

An evidence link can support a factual sentence while leaving a claim about the research process unsupported. A product page may establish that a vendor advertises an export feature. It cannot establish that your assistant tested the export, measured its speed, or confirmed that customers find it usable.

This is why a references list alone is a weak acceptance rule. ALCE, a 2023 citation-generation benchmark, treats fluency, correctness, and citation quality as distinct evaluation dimensions. Its historical results do not measure today's product. The relevant design principle is that a readable answer with citations still needs its support checked.

Review important report statements in three groups:

• Source facts: what a supplied document, recording, or external source actually establishes.
• Work claims: what the assistant or team says it searched, compared, calculated, tested, or observed.
• Interpretations and proposals: what the evidence might mean and what someone should do next.

Consider “Customers struggle with setup, so we validated a simplified flow.” That sentence joins an observation, a generalization, and a completed validation claim. Each needs different support. An interview excerpt may support one person's difficulty. A protocol and observed sessions are needed to establish a validation activity. A proposal to simplify the flow remains a proposal until tested. FActScore, also from 2023, evaluates long-form factual precision by breaking text into smaller factual units and checking support. We borrow that separation as a review technique, not as a claim that a fact metric measures product readiness. Split compound sentences wherever one part could be true while another remains unverified.

4. A fictional onboarding assistant shows where scope can disappear

Imagine a small team building First Week, a fictional assistant for product managers. A customer uploads six consented onboarding interview records. They ask which obstacles occur in the supplied sessions, how the experiences differ between first-time and returning users, and what evidence would justify changing the account-setup flow.

The assistant encounters a problem: the records do not consistently identify whether someone is a first-time or returning user. Two recordings also have unclear passages. It can still produce a convincing narrative about onboarding friction. The dangerous shortcut is to omit the group comparison, infer missing user types, and write a recommendation as if the requested investigation had been completed.

The better output has a narrower, explicit status. It summarizes supported observations from the readable material, marks the requested comparison unresolved, and proposes how to collect the missing information. It does not label those next steps as findings. The customer receives something useful without losing the original assignment.

This distinction has a practical foundation outside model benchmarks. GOV.UK's research-analysis guidance separates observations, findings, and actions. That separation is valuable here: what someone said, what the team concludes, and what it decides to try should remain distinguishable.

For example, a session note might record that participant P03 returned to the password screen twice. An interpretation could be that the recovery route was unclear. A proposed action could be testing a revised recovery message. “The revised message reduces abandonment” would require a different investigation. Neither the transcript nor the proposal supplies that result.

All people, counts, product names, and situations in this example are hypothetical. They illustrate acceptance choices; they are not results from a customer deployment or evidence that the proposed controls improve retention.

5. Use an acceptance sheet that preserves the unfinished question

Copy the following sheet into your product brief or release review. Its purpose is to decide what the customer may rely on, rather than to collect impressive documentation. Fill it before generation, then update it when the report is reviewed. The examples continue the fictional First Week assignment.

A filled decision record might read: FW-01; partial delivery; original comparison not completed; readable observations usable for planning; no usability validation performed; customer owns missing labels; revised comparison awaits clarification. This is an example of a proposed record, not a completed test.

For each completed activity, record its evidence location and whether a person has checked it. A model-generated statement that an interview or test occurred is not the activity record. If no authorized reviewer has checked a material claim, mark it unreviewed rather than allowing the report to imply verification. Do not reconstruct a missing contemporaneous record and present it as an original one.

A reviewer should be able to explain why the report has that status using the sheet. “It sounds cautious” is insufficient. The missing comparison is a specific unfulfilled requirement, with an owner and a path forward. A general disclaimer at the bottom cannot substitute for it.

Do not make every blank field a reason to stop all work. Distinguish a missing detail that limits one observation from a missing requirement that blocks the central recommendation. If the unavailable comparison determines the customer's decision, hold that recommendation while allowing a clearly limited synthesis to be delivered.

6. Make partial delivery and scope changes usable product states

Many research products expose only “working” and “done.” That forces uncertainty into prose, where a busy reader can miss it. A better design gives the result a status tied to the assignment: complete against the agreed conditions, partial with named gaps, blocked by missing inputs, or awaiting review.

These are proposed product states, not a universal standard. Define them in language your customers understand. “Partial” should explain what was completed, which question remains open, and what happens next. “Blocked” should identify the missing permission, input, or external dependency rather than showing an endless spinner.

A scope change requires its own decision. If the assistant cannot compare user groups, offer an explicit choice: provide the missing labels, approve a narrower descriptive report, or stop the assignment. Show the effect of that choice on the recommendation. The model should not count its own decision to simplify the task as customer approval.

Accepting a useful partial result does not mark the original assignment complete. Record partial acceptance separately from approval of a revised scope. If the customer chooses a narrower assignment, future completion is measured against that explicit revision, while the unmet original requirement remains visible in its history.

Preserve the original assignment when a customer approves a narrower version. Otherwise the system can rewrite its own history: the final report looks complete because the requirement that exposed its incompleteness has disappeared. A simple revision note can state the previous question, the approved replacement, who agreed, and when.

Status also needs to travel. A PDF, shared link, copied executive summary, or email excerpt can detach the recommendation from the interface that explained its limits. Put material limitations near the conclusion and in exported summaries. The goal is not to repeat a disclaimer on every page; it is to keep the unresolved condition attached to the decision it affects.

Finally, define what happens when evidence changes after delivery. If a customer corrects a participant label, the system should identify which comparison and conclusions need review. Silently regenerating the prose can make an old shared copy look current. Keep a visible version and tell the customer whether an earlier recommendation has been superseded.

7. Preserve enough evidence without exposing private research

Evidence access creates a tradeoff. Customers need to inspect why a report says something, but interview recordings can reveal personal or confidential information. More visibility is not automatically better if it shares raw material with people who should only see a de-identified finding.

GOV.UK's recording guidance emphasizes consent and careful handling of research records. For this product workflow, ask the research owner which materials may be processed, who may inspect them, and what can appear in an exported report. This article does not establish that a particular consent arrangement is sufficient for your setting.

Use source identifiers and relevant excerpts where permitted. Give the authorized reviewer a way to open the supporting material. Do not attach every raw recording to every report by default. If a recipient lacks access, show an honest access limitation and route them to the accountable owner; do not pretend an inaccessible link is public verification.

External web sources introduce another problem: they change. Zotero's file documentation distinguishes links, stored files, and saved web snapshots. That is a useful practical distinction when designing a source inventory. A URL identifies a location; a permitted snapshot can help preserve what was reviewed. Neither establishes that the underlying claim is true.

Record source version or access date where it matters. For changing product claims, separate “advertised when reviewed” from “verified in use.” Retain only the supporting material your policy permits, with an owner for access and deletion. A deletion request may mean removing a source and revisiting dependent conclusions rather than leaving a report that still implies inspectable support.

You can begin with a restricted folder and a source table. The acceptance question is whether the right person can check the important claim without the product unnecessarily disclosing the rest of the research.

8. Test the promise with deliberately incomplete assignments

Before selling an autonomous research feature, run a bounded pilot on representative assignments you are authorized to use. Include clean inputs and cases with missing labels, unreadable files, conflicting observations, outdated sources, and a request that cannot be fulfilled from the permitted material. Do not judge only the most attractive finished report.

Write the expected behavior first. For First Week, missing group labels should leave the comparison unresolved. An unavailable recording should remain in the inventory with an access problem. A proposed experiment should never become a completed experiment merely because it appears under a results heading. A request to remove a limitation from an export should trigger review of whether the remaining document misleads.

Include a negative control: an assignment where the central question genuinely is answerable from the supplied material. The system should deliver it without needless blocking. Otherwise a product can appear careful simply by refusing everything. Useful restraint means distinguishing supported completion from meaningful incompleteness.

Measure at the assignment level and the claim level. Count attempted assignments, usable outputs, partial deliveries, blocked tasks, and reports marked complete. Within reviewed reports, record unsupported work claims, omitted requirements, inappropriate scope changes, and important statements without adequate support. Keep failed or timed-out runs in the attempt count.

Also record reviewer effort and whether customers understand the status. A hidden limitation that a specialist can eventually find is not necessarily useful to a busy product manager. Ask pilot readers what was done, what remains open, and what they would do next. Their answers can reveal a delivery problem that a citation checker misses.

Set your acceptance conditions before comparing candidates. A polished average should not compensate for a fabricated activity or silently dropped central requirement. Where the system is correct but the evidence is inaccessible, record that separately from factual error. Both matter, but they require different fixes.

Change one major component at a time where practical, keep the assignment and inputs stable, and rerun relevant cases after model, retrieval, export, or permission changes. This is a proposed evaluation protocol. We provide no measured success rate, universal passing threshold, or assurance that a pilot covers all future failures.

9. Decide what to ship, what to assist, and what to defer

A sensible first release might assist research rather than claim to complete it autonomously. It can organize supplied material, draft observations with inspectable support, surface unanswered questions, and prepare the next study brief. That can be valuable even when a human remains responsible for interpretation and acceptance.

Use a complete-delivery label only for a scope your team can actually evaluate. If the product promises new interviews, live testing, or causal conclusions, document generation alone cannot satisfy it. When the central investigation lies beyond the available inputs or your team's ability to review it, narrow the promise explicitly or defer the feature.

This workflow fits bounded research products where the requester can define the decision, identify legitimate inputs, and review the result. It is less useful for open-ended brainstorming, where completeness may not be meaningful. It is insufficient on its own for expert scientific review or high-stakes professional decisions. A completed acceptance sheet does not grant expertise or certify a conclusion.

The tradeoff is real: visible partial states and scope questions may add friction. Removing them can make the demo feel smoother while leaving the customer unable to distinguish a useful draft from finished work. Test that friction with actual users rather than assuming either maximal automation or maximal review is always better.

Before your next release, take one representative assignment and ask someone outside the building team to inspect its promise, report, and export. Can they identify the unresolved question without reading a hidden log? Can they tell whether the assistant observed, inferred, or merely proposed the key result? If they cannot, improve the delivery contract before expanding the autonomous claim.

The product's job is to help the customer know what they have received and what they can responsibly do with it. A document that looks finished is only one part of that job.

References

1. Into the Unknown: original paper, submitted October 13, 2026
2. Into the Unknown: evaluation protocol in the paper's HTML version
3. Official code, release notice, and artifact guide
4. Official dataset and released research artifacts
5. ALCE: Enabling Large Language Models to Generate Text with Citations, 2023
6. FActScore: Fine-grained Atomic Evaluation of Factual Precision, 2023
7. W3C: PROV overview
8. GOV.UK: Analyse a research session
9. GOV.UK: Taking notes and recording user research sessions
10. Zotero: Adding files to your library