If you cannot explain the tool calls, you do not have an agent product
Support opens a session. The UI says two searches and six tools. Foundry traces show something else. App Insights has a third story. Nobody can tell whether the agent searched knowledge, hallucinated a citation path, or the counter lied. That is not a cosmetic bug. That is a missing product surface.
Activity timelines must reconcile platform traces, MCP knowledge searches, and UI counts. Persist normalized events, not only chat bubbles. Count knowledge searches separately from generic tool invocations. Traceability is part of the customer contract for regulated knowledge agents.
Figure 1. The shell shows activity chips. Correlated traces must tell the same story, or support has three truths. Source: Microsoft Learn: Application Insights correlation.
Figure 2. Persist normalized activity events and reconcile them with platform traces. UI counts alone are not an audit trail. Source: Microsoft Learn: Application Insights overview.
I learned this while debugging agent shells where the debug drawer and the marketing-friendly activity chips disagreed. Engineers trusted traces. Users trusted the chips. Support trusted neither. Incident time ballooned because the product could not explain itself.
The core idea
If you cannot explain the tool calls, you do not have an agent product. You have a chat skin over an opaque runtime.
Foundry (or equivalent) traces, MCP knowledge searches, and UI summaries must share a normalized event model. The UI is allowed to simplify. It is not allowed to invent. A debug drawer backed by Foundry plus App Insights shortens incidents only when the same identifiers stitch chat, tools, and platform spans.
A model that stays explainable
1. Treat UI counts as projections of stored events
"2 searches / 6 tools" must be computed from persisted activity events, not incremented in React because a spinner ran. If the platform rejects a tool call, the UI must not keep the optimistic count. Projection bugs are modeling bugs.
2. Persist normalized activity events
Store a structured timeline: tool name, knowledge search versus generic tool, status, latency, correlation ids, and redacted arguments as policy allows. Chat bubbles are a render. Events are the system of record for support. Without persistence, you cannot reconstruct a session after the websocket dies.
3. Count MCP knowledge searches separately
Knowledge search is a different risk and a different support question than "ran calculator" or "exported CSV." Separate counters and filters. Regulated users ask what corpus was hit. Bundling everything as "tools" hides the answer.
4. Ship a debug drawer on Foundry + App Insights
Give internal users (and carefully gated support roles) a drawer that shows platform spans next to normalized events. End-to-end transaction views in App Insights become usable when you propagate operation ids through the shell and the API. This is not optional polish for enterprise agents.
5. Make traceability a customer contract item
Write what you retain, for how long, who can see raw traces, and how redaction works. If the sales story is "grounded with citations," the ops story must include "we can show which search ran." Otherwise audits become theater.
6. Reconcile on purpose in tests
Automated tests should assert that a scripted tool session produces matching UI aggregates and persisted event counts. Snapshot the timeline shape. Catch drift when someone adds a tool path that forgets to emit events.
Failure modes I design against
Optimistic UI counters. Spinners increment; failures do not decrement.
Bubbles only. No event store; support reads screenshots.
One "tools" bucket. Knowledge access questions become unanswerable.
Traces without correlation ids. App Insights islands; Foundry islands; UI islands.
Debug drawer as afterthought. Built during an outage, never maintained.
Promising auditability in decks without retention design. Legal asks; engineering shrugs.
Redaction versus usefulness
Persisted tool arguments help debugging and create leakage risk. Default to redacting free text and secrets, keep tool names, statuses, timings, and corpus identifiers that policy allows. Document the redaction profile next to the event schema. Support should know what they will not see before they open a ticket expecting full prompt replay.
Trade-offs
Normalized event persistence costs storage and redaction design. It buys incident speed and trust. Separate knowledge counters add UX complexity. They match how buyers think about risk. Debug drawers need access control so they do not become a data leak. Test reconciliation slows PR cycles slightly and prevents silent counter lies. You will argue about argument redaction verbosity versus debuggability. Write the policy once.
What I would put on an ADR
- Normalized activity event schema persisted per session turn.
- UI aggregates are projections of that schema only.
- MCP knowledge searches counted and labeled separately from generic tools.
- Correlation ids stitch shell, API, Foundry, and App Insights.
- Support-facing debug drawer with role gates.
- Contract text for retention, redaction, and audit access.
Primary references: Microsoft Learn on Application Insights correlation and Microsoft Foundry tracing/observability docs for your stack version. Pair them with a written event schema in the repo.
What support should be able to answer in five minutes
Which knowledge searches ran, against which corpus hint, with what status? Which generic tools ran? Do UI chips match persisted events for that turn? Can we open the correlated App Insights transaction? If any answer requires reading three disconnected systems and guessing, the product is not operable yet. Build the drawer and the schema until those answers are boring.
When adding a new tool, the PR checklist includes event emission, UI projection, and a reconciliation test. Tools that skip the checklist recreate the original lie in a new costume.
Product managers should be able to read the event schema without learning the Foundry console first.
Closing
Chat UIs forgive a lot. Enterprise support forgives almost nothing when tool counts lie.
Persist the timeline. Reconcile the counters. Separate knowledge searches. If you cannot explain the tool calls, pause the launch and build the observability product you already sold.