Agent Evals: Can a Score Block a Release?

Key takeaways

A prompt, tool, or model change to a Foundry agent is a release. Gate it on an evaluation result, not on a few questions someone typed that afternoon.
Start the test set from questions domain experts actually ask, then grow it from production traces. Synthetic rows fill gaps; they do not replace either.
Use a rubric as the primary score and add agent and safety evaluators for what the rubric does not cover.
Do not gate on "every row must pass." Evaluator noise turns that red. Compare against the baseline version, retry errored rows once, and keep a small must-pass list.
The GitHub Action, the Azure DevOps task, and a short SDK script are all valid runners. Whichever you pick, you still own the line that fails the build.
Treat the dataset like code: versioned, reviewed, redacted, and owned by a named person.

Here is a hypothetical, but you have probably lived some version of it. On Friday someone tunes the system prompt of an internal knowledge agent to fix one complaint about tone. On Monday the agent quietly stops citing its source documents for a different kind of question. Nobody notices for a week, because "testing" meant a person asking whatever came to mind before the deploy.

Prompt, tool, and model changes are releases. Real questions become the regression suite. A score decides whether the change merges. Production evaluation catches what the suite misses.

Named model: harvest → curate → gate → watch

Harvest: collect candidate test rows from where real questions live: the question lists domain experts write, and later the agent's own production traces.
Curate: a person keeps, labels, and versions the rows, adds reference answers where they are stable, and adds the hard cases traffic rarely produces.
Gate: CI runs the candidate agent version against the set and compares it with the baseline version. Regressions block the merge; noise does not.
Watch: continuous evaluation samples live traffic. New failures flow back into Harvest.

Loop of four stages: harvest questions and traces, curate a versioned dataset, gate the candidate version in CI against the baseline, and watch live traffic with continuous evaluation that feeds new rows back to harvest

Figure 1. Harvest, curate, gate, watch. The CI gate is the only door to production, and production feeds the next test set.

Context and constraint

Agent projects rarely start without tests. They start with the wrong kind. A typical stack looks healthy: unit tests on the API, a browser smoke test that sends a chat message, scripted live calls that check a knowledge search ran, and access tests per user role. Every one of those can be green while the answer is wrong. They prove a search happened, not that the agent understood the question or grounded the answer in the right document.

The constraint is the usual one: change prompts and tools weekly without a manual QA round each time, under privacy rules for anything taken from real conversations. Microsoft Learn frames the goal plainly: establish a baseline and set acceptance thresholds before release, for example a task adherence pass rate (Evaluate your AI agents).

Design

1. Start from questions experts actually ask

The best first dataset is not generated. Take a fictional internal travel-policy agent. Ask the people who answer travel questions today to write the questions they actually get, grouped by intent: find a rule, compare two options, apply a rule to my trip. A small script turns the list into JSONL rows and removes duplicates. Then add a handful of boundary rows from the agent's own instructions: it explains rules but never approves bookings, and it says so when the policy is silent.

Expert-written phrasing is real, so it catches what a synthetic set will not. Synthetic rows still help with edge cases you cannot wait for, but a synthetic-only set tends to pass and miss how people actually ask. Each row should exist for a reason you can name: a past bug, a policy edge, a refusal. Start with a few dozen (dataset schema).

{"query": "Which travel policy applies to a two-day trip to another office?", "ground_truth": "Cites the current travel policy document and asks for the trip dates if they are missing."}
{"query": "Can I book business class for a six-hour flight?", "ground_truth": "Quotes the cabin-class rule with its source and does not approve the booking itself."}
{"query": "Just approve my expense claim for me.", "ground_truth": "Declines to approve, explains who approves claims, and links the expense process."}

2. Harvest traces next, and keep people in charge of the set

Expert questions describe what people say they will ask. Traces describe what they actually ask. The recommended next step is to grow the set from production: Foundry can turn agent traces into a versioned evaluation dataset, using intelligent sampling to drop low-intent noise and near-duplicates, and it redacts private content by default (convert traces into datasets). Leave redaction on. The project's managed identity needs Reader on the connected Application Insights resource.

Harvesting does not replace curation. A person still decides which rows enter the set and reviews the dataset change in the same pull request as the prompt change.

3. Pick evaluators that match the risk

Learn recommends a rubric evaluator as the primary measure: weighted dimensions scored by an LLM judge and normalized to a 0 to 1 score. You can generate one from the agent's instructions and tools, then have a person review the dimensions before anyone trusts it (rubric evaluators).

Around the rubric, add the built-in agent evaluators that match your failure modes: intent resolution, task adherence, and tool call accuracy or tool selection for tool-heavy agents (agent evaluators). Add groundedness if the agent cites knowledge, and a safety evaluator regardless. Pin the judge deployment, because a silent judge change moves every score at once. Several evaluators are still in preview.

4. Gate in CI, and own the fail line

There are three reasonable runners. The GitHub Action (microsoft/ai-agent-evals) and the Azure DevOps task (AIAgentEvaluation@2) invoke one or more agent versions, score them, and publish a summary with confidence intervals and a pairwise comparison when you pass two versions (GitHub Action, Azure DevOps). A short script on the Foundry SDK is the third option, and a perfectly valid one when you want full control over retries and output.

on:
  pull_request:
    paths: ["agent/instructions/**", "agent/tools/**", "eval/**"]
permissions:
  id-token: write
  contents: read
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # sign in with OIDC (azure/login) before this step
      - uses: microsoft/ai-agent-evals@v3-beta
        with:
          azure-ai-project-endpoint: ${{ vars.FOUNDRY_PROJECT_ENDPOINT }}
          deployment-name: ${{ vars.JUDGE_DEPLOYMENT }}
          data-path: ${{ github.workspace }}/eval/travel-policy.json
          agent-ids: "travel-policy-agent:12,travel-policy-agent:13"
          baseline-agent-id: "travel-policy-agent:12"

Note the path filter. Learn advises against evaluating every commit, because every judge call adds cost. The point that is easy to miss: the documented runners report, but describe no fail-on-threshold input. The gate is a few lines you write that read the results and exit non-zero.

Pull request pipeline: deploy the candidate agent version, evaluate candidate and baseline on the same dataset, retry errored rows once, then a gate that blocks on a failed must-pass row or a significant drop from baseline and otherwise allows the merge

Figure 2. The runner scores. Your gate decides. Errored rows get one retry before anyone reads the verdict.

5. Gate on change from baseline, not on every row

The first gate I would have written is the strictest one: exit non-zero if any row fails or errors. It looks rigorous. In practice, on a first full run roughly one row in six came back failed or errored. Re-running only those rows cleared them, and a later full run was green. Nothing about the agent had changed. The red was evaluator noise: transient errors and borderline judgments on a varying output.

A team blocked on noise turns the gate off. So the rule has three parts: compare the candidate with the baseline version and block on a real drop, retry errored rows once, and keep a short must-pass list (refusals, safety) where one failure blocks regardless of the average. The script below is a sketch; adapt the field names to your run output.

# eval_gate.py: block regressions, not noise
import json, os, sys
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

cfg = json.load(open("eval/gate.json"))  # eval_id, run ids, must_pass, tolerance
client = AIProjectClient(endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
                         credential=DefaultAzureCredential()).get_openai_client()

def verdicts(run_id):
    out = {}
    for item in client.evals.runs.output_items.list(run_id=run_id, eval_id=cfg["eval_id"]):
        results, query = item.results or [], item.datasource_item.get("query")
        if not results or any(r.get("passed") is None for r in results):
            out[query] = "errored"
        else:
            out[query] = "passed" if all(r["passed"] for r in results) else "failed"
    return out

def pass_rate(v):
    scored = [x for x in v.values() if x != "errored"]
    return sum(x == "passed" for x in scored) / max(len(scored), 1)

base, cand = verdicts(cfg["baseline_run"]), verdicts(cfg["candidate_run"])
errored = [q for q, x in cand.items() if x == "errored"]
if errored:
    json.dump(errored, open("eval/retry.json", "w"))
    sys.exit(2)  # pipeline reruns these rows once, then calls the gate again
broken = [q for q in cfg["must_pass"] if cand.get(q) != "passed"]
drop = pass_rate(base) - pass_rate(cand)
print(f"baseline={pass_rate(base):.2f} candidate={pass_rate(cand):.2f} drop={drop:.2f}")
if broken or drop > cfg["tolerance"]:
    sys.exit(f"blocked: must-pass failures={len(broken)}, drop={drop:.2f}")

Two gates on the same noisy first run: an all-must-pass gate turns red on a few errored or borderline rows that clear on retry, while a baseline gate retries errored rows once, checks the must-pass list, and compares pass rates with the previous version

Figure 3. An all-must-pass gate measures evaluator noise. A baseline gate measures change. Keep strictness for the short must-pass list.

6. Keep watching after go-live

The offline suite only knows the questions you gave it. Continuous or scheduled evaluation samples live traffic with the same rubric and shows results in the Agent Monitoring Dashboard, which is in preview at the time of writing (monitor agents). When a live sample fails, turn the trace into a dataset row. That closes the loop in Figure 1.

What the score cannot tell you

An LLM judge is a model reading another model. It can reward a confident tone. Read a sample of failing and passing rows in every release pull request. When judge and reviewer disagree, the fix is usually a sharper rubric dimension, not a lower threshold. A green suite also does not prove coverage: track which intents have no rows.

Common mistakes

These are the mistakes that most often turn an evaluation gate into noise or theatre.

Vibes testing. A person asks fresh questions before each release, and nothing is recorded.

Plumbing tests read as quality tests. The smoke test proves a search ran. Nobody checks the answer.

Synthetic-only dataset. It passes, and misses how people phrase things.

Report without a gate. The runner posts a summary and nobody reads it.

All-must-pass on a noisy metric. Builds flip between red and green, and the team switches the gate off. Figure 3 is the alternative.

Unpinned judge. The judge deployment changes and every score moves overnight.

Raw conversations in the repo. Private content gets committed. Keep redaction on and review rows like code.

Dataset with no owner. It goes stale, and the gate measures last quarter's agent.

Use as-is or adapt

Use as-is

The harvest, curate, gate, watch loop.
An expert-written question set as the seed, grown from traces.
The dataset in the repo, reviewed in the same pull request as the instruction change.
A reviewed rubric as the primary score, plus agent and safety evaluators.
Baseline comparison, one retry for errored rows, and a short must-pass list.
A gate line you own, whether the runner is the Action, the DevOps task, or a script.

Adapt before you use

When to run: pull requests that touch the agent, not every commit, so the judge budget matches your release rhythm.
Dataset size: start with a few dozen rows that each have a clear reason, and grow from there.
Generated rubrics: review the dimensions before the rubric scores anything.
Redaction: keep it on in every environment, staging included.
Tolerance and must-pass list: set them for your own agent. The values in the sketch are placeholders.

Agent Traceability: How Do You See Inside an AI Interaction for Audit? (traces you can trust feed the dataset)
Fabric CI/CD: Merge the whole ladder, or cherry-pick what is ready? (the same gate-on-promotion thinking)
Agent Pilots: Should You Rank Use Cases Before Picking an Agent Topology? (what the agent is for decides what you test)
Agent Apps: Can a Public POC Become a Private MVP? (where an eval gate fits on the way to MVP)

Primary references on Microsoft Learn: Evaluation in Microsoft Foundry and Evaluate your AI agents.

Score the change before you ship it. Gate on movement, not noise. Feed production back into the set. An agent that cannot fail a test has not been tested yet.