Fabric Data Agents: What a 20-Question Baseline Proves

Picture a team that connects a data agent to a lakehouse on a Friday, asks it ten questions they already know the answers to, and gets ten right. On Monday they hand it to the business. On Tuesday someone asks "who is our biggest customer?", gets a confident, well formatted name, and puts it in a slide. Nobody asked which "biggest" they meant, and nobody would notice that the agent never asked either.

I wanted to know what a baseline actually catches before any tuning, so I ran one. This post is the method, the numbers, and what I would change before trusting the result.

Figure 1. The pass rate is the least interesting output. Steps 2 and 3 are where a baseline earns its keep.

Context and constraint

What this is and is not. This is a first-hand test on synthetic data: a TPC-H-derived schema at roughly scale factor 1, generated with a seeded Spark generator rather than the official dbgen tool, and loaded into a lakehouse as seven Delta tables. It is not official TPC-H data and nothing here is an audited TPC result. It is one run per question, on the smallest paid capacity, with English questions against a simple, well named schema. It is not a production accuracy claim, and 20 questions is a smoke test, not a benchmark. Treat every number below as "what one careful afternoon shows", not as "what a data agent scores".

The setup. One lakehouse, one data agent, all seven tables selected, no agent instructions, no data source instructions, no example queries, published. The API responses reported model gpt-5.1. The agent was on the standard runtime (read from the agent settings after the run): the preview runtime switch is not set. I called the published agent from a Python script with the Fabric data agent Python SDK, with a fresh conversation per question, so no answer could lean on a previous one. Every question got a gold answer computed with DuckDB against the same Delta snapshot, not with the agent and not with the SQL endpoint the agent uses.

The question set. Twenty questions: single table counts and sums, joins of two to five tables, date filters, a top N, a month with the most orders, and two questions planted to be ambiguous on purpose.

How the questions are worded. This matters for every number below. Eighteen of the twenty use the schema's own vocabulary. Four of them spell out the revenue formula ("extended price times one minus discount"), five use literal column values such as a ship mode, a return flag or an order priority, and five say how to round. Only the two ambiguous questions are phrased the way a business user would ask. That makes the 18 easier than real questions, which is part of the point: a high pass rate on questions like these says more about the wording than about the agent.

The headline numbers.

Measured on my F2 capacity, 30 Oct 2026, one run per question.

The slowest call, at 15.7 s, was the first request after publishing. It started 22 seconds after I republished the agent and had nothing in the prompt cache, so read it as a first request after publishing, not as a general cold start. Without it the median is 7.55 s and the p90 is 10.8 s. One run per question cannot show whether it happens every time.

Design

1. Gold answers come from outside the agent

The gold set is the whole test, so it cannot share a code path with the thing under test. I wrote one SQL statement per question and ran it with DuckDB on the same Delta files. If the agent and the gold disagree, you want the disagreement to mean something about the agent, not about two copies of the same mistake.

Two scoring details matter more than they look:

• Rounding. One revenue total came back as 8,683,700,250.8999 against a gold of 8,683,700,250.90. Same number, different precision. A strict string match calls that wrong, so my scorer compares numbers with a small tolerance, and it passed this one on its own.
• Formatting. The "month with the most orders" answer said July with 20,773 orders. My scorer looked for the month as a number, so it marked the answer partial. A human read marked it correct.

The second one is why every partial and every wrong gets a human read before it counts. The scorer erred on the strict side, which is the safe direction, but it still erred.

2. Plant ambiguity, and write every reading down first

Two questions were ambiguous by design:

• "What were our total sales in 1995?" I wrote down five defensible readings: net of discount by order date (my gold), net of discount by ship date, gross by order date, gross by ship date, and net including tax by order date. On this data those five span from 35.15 billion to 37.01 billion, about 5 percent apart. Every one of them is a plausible looking number with no error message.
• "Who is our biggest customer?" can mean most net revenue, most gross revenue, most orders, or highest account balance. Net and gross revenue name the same customer. Most orders and highest account balance each name a different one.

I wrote all of those readings, with their numbers, into the test before the run. That is the step people skip, and it is the step that turns "the answer looked fine" into a score.

On the sales question the agent chose net of discount, by order date, which matched my gold reading. I counted it correct, but be honest about what that means: it matched my convention. A finance team that books sales by ship date would have got a number that was wrong for them, and the answer did not say which definition it used.

On the customer question the agent chose highest account balance. The answer did say "by account balance", which is better than nothing, but it did not ask. I counted it partial: a different reasonable reading, and one I had written down, but not the one the business most likely meant.

3. Read the rewrite and the SQL, not only the answer

Figure 2. The SQL was correct for the question it was handed. The guess happened one step earlier, in the rewrite.

When you call a published data agent from code, the response carries its tool steps: the agent's rewrite of your question in natural language, the SQL it generated, and the result it ran. For the customer question the rewrite already said "the single customer with the highest account balance". The SQL step then did exactly what it was told. If you only score the final answer, you see a wrong customer. If you read the rewrite, you see where the decision was made, which tells you where to fix it: an instruction that defines "biggest customer", or an example query that shows it.

Reading the SQL also surfaces things that are right today and fragile tomorrow:

• A top customers query grouped by customer name rather than customer key. On this sample every name is unique, so the result was exact. On real data with two customers of the same name, it would silently merge them.
• A quarter filter used BETWEEN '1994-01-01' AND '1994-03-31'. On a date column that is correct. On a datetime column it would drop most of the last day.

Neither of those costs a point in a pass rate. Both are worth an instruction or an example query before real users arrive.

One more thing showed up in the tool inputs. In eight of the nine multi-table questions, the trace shows the agent handing its SQL step a list of join keys labelled "Verified ontology join path". I had written no instructions. In the ninth, that list was empty and the SQL joined the tables anyway. No ontology was attached to this agent, and Microsoft's docs don't describe this step, so I am only reporting what the trace shows. If I re-run the set after the November update, I will check whether it is still there. You only see it if you read the steps.

4. Treat returned token usage as an estimate

The response also returns token usage. Per request it showed roughly 5,800 input tokens, of which 5,632 were cached on 18 of the 20 calls, and between 13 and 137 output tokens. The Fabric operations page lists data agent usage as the AI query operation and links its rates to the Copilot consumption page: 100 CU seconds per 1,000 input tokens, 10 per 1,000 cached input tokens and 400 per 1,000 output tokens. Applied to the API's own token counts, that puts the AI cost at about 2,900 CU seconds for the 20 questions, roughly 145 each. That is probably a lower bound. The Capacity Metrics figure follows in an update on 1 November or later, because the app lags, and I am not giving a currency figure until then.

Two cautions before anyone copies that figure:

• Caching swings it by four to eight times. The two calls with no cached tokens cost about 580 CU seconds each. The cached ones cost between 75 and 140. One of the uncached calls was the first request. The other was the fifth of 20, early in the run, so caching is likely, not guaranteed.
• It is probably not the full bill. Each response carries a single usage object, while the step that writes the SQL shows its own generation with no token counts, so the returned usage looks like it covers the orchestrator call only. The SQL the agent runs is billed separately, as a SQL Endpoint Query. In the Capacity Metrics app the agent itself shows up as AI query under the LlmPlugin item kind, a background operation, and that is the number to quote. Until you have it, call the token figure an estimate and probably a lower bound.

5. Re-run after every change

A baseline is only useful as the "before" in a comparison. Keep the question set, the gold answers and the readings in source control, and re-run them after any change to instructions, example queries, selected tables, the model, or the runtime.

That last one is not hypothetical. A Fabric blog post on 19 October 2026 announced advanced NL2SQL and advanced DAX generation as generally available and said the core improvements roll out to the standard runtime from 2 November, expected to finish by 10 November, with no opt-in. The standard runtime also moves to GPT-5.6 Terra. On the SQL side, the list of what moves names improved example query following, filter value substitution and data exploration. Asking clarifying questions is not on that list: it is one of the advanced NL2SQL modes, and the documentation places it on the preview runtime. Learn's SQL sources page still lists advanced NL2SQL as a preview tool on the preview runtime today, while the blog post calls it generally available, so check both before you rely on either. I do not expect the rollout alone to fix the customer question. A re-run after the rollout is planned but not scheduled yet, and with one run per question it could show a change but not prove one. The same post recommends rerunning a ground truth set and comparing it with your previous baseline, which is exactly what this set is for.

For repeated runs at scale, Microsoft's documented SDK route (preview) is evaluate_data_agent in the fabric-data-agent-sdk package. It takes question and expected answer pairs, needs a paid F2 or higher (or P1), and scores each answer with an LLM judging prompt that you can replace with your own. That makes the judge a model too, so hand-check its partials and failures the same way. A hand-built harness like mine is fine for 20 questions. It does not scale to the few hundred a real domain needs.

Common mistakes

Testing only questions you already know are clear. A clean schema with good column names passes easy questions with no instructions at all. That proves the schema is clean, not that the agent is ready.

Using the agent, or its SQL endpoint, to build the gold set. Then a shared mistake looks like agreement. Compute expected answers through a different path.

Scoring only the final answer. The answer tells you that something went wrong. The rewrite and the SQL tell you where, and so what to change.

Counting a matched ambiguous question as a pass without saying so. If the agent's reading matched yours, it matched your convention. Record which reading it chose, so a later run that picks a different one shows up as a change.

Running once and calling it a baseline. One run per question tells you nothing about variance. Plan at least three runs per question before you compare a before and after. I ran one here, and I would not compare runtimes on it.

Quoting token CU as cost. It likely leaves out the inner calls, and it certainly leaves out the query engine. Use the Capacity Metrics app.

Use as-is or adapt

Use as-is:

• The four steps: gold from outside the agent, planted ambiguity with readings written down, scoring the rewrite and the SQL, re-running after every change.
• A fresh conversation per question, so answers cannot lean on each other.
• A human read of every partial and every wrong before it counts.
• Recording the rewritten question and the SQL next to each answer, so two runs can be diffed.

Adapt before you use:

• Question set. Twenty is a smoke test. Use the questions your users actually ask, and grow the set every time a real question surprises you.
• Ambiguity. Plant the ambiguities your business really has: fiscal or calendar year, booked or shipped, gross or net, which customer hierarchy. Sample data will not tell you which ones matter.
• Runs. Run each question several times on the same configuration before you compare configurations.
• Cost. Replace the token estimate with the Capacity Metrics app figure for the same window.
• Scale. Move from a hand harness to evaluate_data_agent once the set passes a few dozen questions.

Try it yourself

The data agent baseline demo in my public tech-demos repo has the 20 questions, the gold answers and every alternative reading, the scorer, and the results with the generated SQL, so you can run the same baseline against your own agent and diff it after the November update.

Related

• Fabric IQ: Is Your Semantic Model Ready? (the modelling work that decides how many of these questions an agent can answer at all)
• Agent Evals: Can a Score Block a Release? (turning a baseline like this into a gate that can stop a release)
• Data Agent Readiness: Is Genie (and Fabric Agents) Really a Modeling Problem? (the same lesson on another platform: the agent is only as clear as the data model under it)

If you only change one habit, write down every reading of your ambiguous questions before you run them. A baseline without that is a pass rate, and a pass rate on clear questions says very little.