DataSpace Exposes the Data Agent Demo Trap: A Complete-Result Gate for Founders

A new benchmark called DataSpace asks data agents to answer questions from messy workspaces containing CSV, JSON, SQLite, Markdown, PDF, and video. The task is deliberately ordinary: find the relevant material, combine it, and return the complete requested table. Yet the strongest controlled result reported by the authors was 66.34% task accuracy across 410 tasks. Changing the agent harness while holding the model fixed produced a 15.36-point spread.

Those figures are interesting, but the more useful finding for a founder is where the best system failed. In the authors' audit of 136 failed runs, 56.6% were traced to misunderstanding the requested result or materializing the wrong columns, even when useful work had already happened upstream. The agent could find evidence and perform calculations, then still deliver the wrong business object.

This is the data-agent demo trap. A fluent explanation, a plausible chart, or five correct rows can look successful while a renewal, account, region, exception, or required field is missing. If a user acts on the output, partial correctness can be more dangerous than a visible error.

This guide is for nontechnical founders, AI app builder users, and small teams shipping agents over documents, spreadsheets, databases, dashboards, or mixed workspaces. You will get a complete-result contract, a worked customer-renewal scenario, six acceptance tests, a decision matrix, and a 48-hour rollout. The central judgment is: grade the business deliverable, not the persuasiveness of the answer.

What DataSpace actually measured

The DataSpace paper was submitted on August 4, 2026. Its 410 tasks contain 7,439 artifacts totaling 15.01 GB. Questions and evidence can cross Chinese and English, and the available material can include structured files, databases, long documents, and videos. Each agent sees one question and one task-local workspace, then must produce a complete table.

The benchmark is unusual in three useful ways.

First, the relevant source is not handed to the agent. The workspace contains both useful and irrelevant artifacts, so the system must discover what matters. Second, answers must be directly consumable tables rather than open-ended reports. Third, evaluation is deterministic. The evaluator accepts equivalent column names, ordering, numeric precision, units, nulls, and row-order semantics according to a frozen configuration, but it rejects incomplete or incorrect results.

The official repository makes all 410 inputs public and provides gold answers plus frozen evaluation configurations for 60 representative tasks. The other 350 reference answers are withheld for official full-benchmark evaluation. The dataset card documents the release, file inventory, public-reference selection, and MIT license. This matters because teams can inspect real task shapes and run the evaluator locally rather than treating a paper score as a marketing claim.

DataSpace also served as the evaluation benchmark for the KDD Cup 2026 Data Agents competition. That history increases the practical relevance of the task format, but it does not make the benchmark identical to your production workload.

Define the result before evaluating the agent

Five terms prevent most evaluation conversations from drifting.

The distinction between a complete result and a good explanation is fundamental. Microsoft’s RAG evaluation guidance defines completeness as answering all parts of a query and treats it separately from groundedness and relevance. For data-agent products, go one step further: define what “all parts” means as rows, columns, types, units, exclusions, and exception states before the agent runs.

Read the benchmark as a product warning, not a model ranking

DataSpace reported a best controlled backbone score of 66.34%. It also reported that 76 of 410 tasks were missed by all six tested backbones. With one backbone held fixed, the five harnesses ranged from 30.98% to 46.34%. These are benchmark results under the authors' configuration, not universal model reliability rates.

The product warning is broader than the leaderboard. The model, tools, context management, prompts, output adapter, and stopping behavior work as one system. Buying the highest-scoring model cannot compensate for a harness that loses columns, changes row grain, truncates output, or stops before saving the final artifact.

The task characteristics also matter. Across the six backbones, multimodal tasks scored 1.8 to 14.0 percentage points lower than single-modal tasks, while tasks requiring joins fell by 9.7 to 19.8 points. The authors label these comparisons descriptive rather than causal. A founder should not copy the deltas into a forecast, but should recognize the risk pattern: every extra representation and relationship creates another place to drop, misalign, duplicate, or misinterpret evidence.

This is why a model bake-off using ten clean spreadsheet questions is not enough for a product that promises analysis across invoices, contracts, CRM exports, support notes, and recorded calls. Your evaluation set should resemble the business path, including distractors, inconsistent labels, missing values, stale versions, and joins.

The most important failure happened at the finish line

The paper's trace audit is more actionable than the aggregate score. Of 136 failures from the strongest controlled backbone, answer materialization accounted for 71. Another 31 began with task specification or intent. Sixty materialization failures added or omitted columns after the needed internal result was available. Seventeen intent failures misunderstood the requested output or row grain.

Together, those two routes accounted for 77 of 136 audited failures. By comparison, only three primary failures were attributed to choosing the wrong evidence source. Finding a relevant file was not the same as recovering, aligning, computing, and delivering the required result.

For a product team, “materialization” means the last conversion from internal work into the artifact the user receives. Typical failures include:

• the analysis contains the right customer IDs, but the exported CSV omits them;
• the agent computes one row per subscription, but the user asked for one row per customer;
• a Markdown preview shows 20 rows while the saved file contains 17;
• a null is silently turned into zero;
• currency symbols disappear, making USD and JPY indistinguishable;
• the agent returns the top ten items even though the user requested every item above a threshold;
• a chart looks correct while its underlying table has duplicate join rows.

Why polished partial answers pass weak reviews

Most prototype reviews are biased toward visible success. The founder asks a question, the agent searches several files, explains its reasoning, and returns a table that looks credible. Reviewers spot-check two familiar rows and approve the demo.

That process measures plausibility. It does not measure completeness.

A partial result often avoids obvious hallucinations. Every displayed row may be correct. The failure is what is absent: the cancelled account whose status uses a different spelling, the second invoice from a legacy system, the customer mentioned only in a PDF appendix, or the zero-result region that should appear explicitly. A confident paragraph can make omission harder to notice because it gives reviewers a coherent story.

NIST's work on evaluation probes for agentic AI separates faithfulness, completeness, and sufficiency, and proposes machine-readable audit trails that map claims to evidence. That is a useful pattern, but a business result also needs population-level checks. Evidence for each displayed row does not prove that every required row was displayed.

The U.S. Government Accountability Office's data reliability guidance treats accuracy, completeness, and applicability to the intended purpose as separate questions. Adopt the same discipline. “The values we checked are accurate” is not a substitute for “the result covers the full in-scope population.”

Use a complete-result contract

Before evaluating a data agent, write one small contract for the output. A founder should be able to review it without reading code. An engineer or builder should be able to turn it into assertions.

result_contract:
  job: "List subscriptions that need renewal intervention"
  as_of: "2026-08-08T00:00:00Z"
  source_snapshot: "billing-v184 + crm-export-20260808 + contracts-index-v12"
  population:
    include: "active annual subscriptions renewing in the next 45 days"
    exclude: "test tenants, already-cancelled subscriptions, approved non-renewals"
  row_grain: "one row per subscription"
  required_columns:
    - subscription_id
    - customer_id
    - renewal_date
    - contract_value_usd
    - risk_reason
    - evidence_ids
  type_and_unit_rules:
    renewal_date: "ISO 8601 date"
    contract_value_usd: "decimal USD, no currency conversion"
    evidence_ids: "one or more resolvable source references"
  uniqueness_key: [subscription_id]
  ordering: [renewal_date_ascending, subscription_id_ascending]
  precedence: "signed amendment > signed order form > current billing record > CRM note"
  empty_result: "return a valid header-only CSV plus a no_matches receipt"
  exceptions: "emit conflicting, missing, or unreadable records separately; never drop them"
  reconciliation:
    source_population_count: required
    included_count: required
    excluded_count_by_reason: required
    unresolved_count: required
  delivery:
    artifact: "renewal_intervention.csv"
    preview_must_match_artifact: true

The contract has six jobs. It fixes scope and source versions at a point in time. It defines the row grain and schema. It declares which source wins when records conflict. It prevents convenient coercions of nulls, dates, currencies, and units. It makes empty and unresolved cases visible. Finally, it requires reconciliation so the team can account for the full source population.

You do not need this much structure for “summarize this note.” You do need it when omissions could change outreach, money, inventory, staffing, compliance, or a customer decision.

Walk through a customer-renewal agent

Imagine a five-person SaaS company, Northstar Metrics. Its founder wants an agent to identify annual subscriptions that renew within 45 days and need intervention. Relevant evidence is scattered across a billing CSV, a CRM JSON export, signed order-form PDFs, a Markdown pricing policy, and support notes.

The demo question is: “Which renewals are at risk, why, and what should we do?” An attractive prototype returns twelve customers with concise explanations. Three are real risks, and every quoted support note is accurate. The team is ready to connect the result to an outreach workflow.

The complete-result contract changes the review.

First, the team defines one row per subscription, not one row per customer. That matters because a customer can have two products with different terms. Second, the source population is every active annual subscription renewing in the next 45 days, not every account found in the CRM search. Third, order-form terms outrank default pricing policy when they conflict. Fourth, “risk unknown” is a valid result state; it must not disappear because a PDF could not be parsed.

The team creates 24 synthetic but realistic subscriptions. It includes one duplicate customer name, two currencies, an amended PDF, a missing support record, a cancelled subscription still present in the CRM, and one account with two subscriptions. The agent returns 22 rows. Every returned row looks correct.

Reconciliation reveals the problem. One subscription vanished in a customer-level grouping. Another was removed because its amended PDF used a different contract identifier. The preview also rounded contract value while the CSV preserved cents. None of these failures would have been obvious from the narrative.

Northstar does not need to abandon the product. It narrows the initial promise. The agent prepares a renewal-review table and an unresolved queue; it does not automatically trigger outreach. A deterministic validator checks schema, uniqueness, dates, counts, exclusions, totals, and preview/file equality. A human reviews unresolved contracts and the highest-value renewals. The product becomes useful because the team defined the boundary of trust.

Run six acceptance tests before launch

Use a small, versioned test workspace. Keep expected results outside the agent's accessible context.

1. Population closure

Create records that should be included, excluded for different reasons, and marked unresolved. Require the counts to reconcile:

Fail if a record disappears between stages, even when every returned row is correct.

2. Row-grain collision

Include two objects that share a customer, name, date, or category but must remain separate. This catches accidental grouping and bad uniqueness assumptions. Then include one deliberate duplicate that should collapse under the declared key.

3. Cross-source join

Put the identifier in one file, a status in another, and an overriding term in a document. Add a distractor with a similar label. Pass only if the final row uses the correct entity, precedence rule, and evidence references.

4. Type, unit, null, and ordering preservation

Mix dates, decimals, percentages, currencies, nulls, and values close to thresholds. Require exact rules for conversion and rounding. Verify the saved artifact, not just the interface preview.

5. Empty and unresolved outcomes

Test a legitimate empty result and a workspace with unreadable or contradictory evidence. The first should return a valid empty artifact. The second should return explicit unresolved records. Neither should become a fabricated row or a generic success message.

6. Repeat, preview, and export consistency

Run the same frozen task several times. Nondeterministic reasoning may take different paths, but the accepted business artifact must satisfy the same contract. Compare the UI preview, downloaded file, API response, and any downstream handoff. Fail if one surface truncates, reformats, or reorders the result in a consequential way.

Score the product in layers

One pass/fail metric can hide the intervention you need. Keep a layered scorecard.

This layering borrows the paper's distinction among intent, discovery, extraction, grounding, computation, materialization, and termination, then adds provenance and delivery for production use. It helps a small team choose the right fix. A retrieval problem may need better source metadata; a correct internal table with a broken export needs an output fix, not a larger model.

The W3C PROV overview defines provenance around the entities, activities, and people involved in producing data, with support for derivation, validation, versioning, and reproducibility. You do not need to implement the full standard to benefit from the idea. At minimum, keep source ID and version, transformation or query version, agent/harness version, run ID, time, and reviewer or approval state.

Set launch decisions by consequence

Use consequence, not benchmark enthusiasm, to choose the operating mode.

The UK government's AI-ready dataset guidance emphasizes that readiness depends on the specific dataset and use case, not a universal label. It also calls for schema/version tracking, provenance, known quality issues, stable identifiers, source traceability, and explicit ownership. Apply the same context-specific standard to output readiness.

Know what DataSpace does not prove

DataSpace is a valuable benchmark, not a production certification.

Its tasks are constructed from clinical and financial Text-to-SQL sources, then transformed into heterogeneous workspaces. That gives the authors executable reference logic and supports deterministic evaluation, but it does not reproduce every messy organizational history, permission model, live API, or adversarial document. Most tasks come from financial domains, with a smaller healthcare subset.

Only 60 complete reference packages are public for local end-to-end evaluation; full scoring over the remaining references is controlled. The reported model and harness results use specific snapshots, tools, limits, and July 2026 endpoints. They should not be treated as permanent rankings or probabilities for your users.

Task accuracy is intentionally strict: one incomplete table makes the task incorrect. That is appropriate when the product promises a complete result, but your product may also need row-level precision and recall, calibrated uncertainty, latency, cost, privacy, permission isolation, accessibility, and human recovery. A strict table metric is one gate, not the whole product evaluation.

Finally, a gold table exists only when the question is sufficiently specified. Real users ask ambiguous questions. When scope, row grain, unit, time, or precedence is unclear, the right behavior is to ask a clarifying question or produce a draft contract. Deterministic evaluation begins after the product defines what success means.

Avoid five common reactions

A 48-hour founder rollout

The durable lesson from DataSpace is not that data agents are only 66.34% good. It is that a system can do impressive work and still fail while forming the final deliverable. Founders can control that risk today: define the result, reconcile the population, verify the artifact, preserve provenance, and give the agent no more authority than the evidence supports.

References

• DataSpace authors, DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces, August 2026.
• HKUSTDial, DataSpace official repository and evaluator.
• HKUSTDial, DataSpace dataset card and public release.
• KDD Cup 2026, Data Agents for Complex Data Analysis.
• NIST, Building Evaluation Probes into Agentic AI.
• U.S. Government Accountability Office, Assessing Data Reliability.
• Microsoft Azure Architecture Center, Large Language Model End-to-End Evaluation Phase.
• UK Government, Guidelines and best practices for making government datasets ready for AI.
• W3C, PROV Overview.