The short answer: trace every run, score a sample, alert on behavior
Published: Oct 8, 2026 · Last updated: Oct 8, 2026
To monitor AI agents in production, record every run as a structured trace that captures each model call and tool call. Then score a steady sample of those runs with automated evaluations, and alert on changes in behavior, not just on errors. Uptime and latency dashboards alone will stay green while an agent quietly gives wrong answers, loops on a tool, or skips a step it used to do.
This guide is for teams that already shipped an agent and now feel blind. It covers what to log, how to trace tool calls, how to keep evals running after launch, and which alerts are worth waking someone up for. If you are still at the design stage, start with our AI agent development guide and come back here before launch.
Why normal application monitoring misses agent failures
A regular API either returns data in the right shape or throws an error. An agent can return a well-formatted, confident, wrong answer with a 200 status code. That is the core problem.
Agents also fail in ways a request log cannot show:
• Silent wrong path: the agent calls the search tool instead of the order lookup tool and answers from stale content.
• Loops: it retries the same tool with slightly different arguments until it hits the step limit.
• Partial completion: it updates the CRM record but never sends the confirmation email, then tells the user everything is done.
• Drift: nothing in your code changed, but the model provider updated a model, a tool API changed its response format, or your users started asking different questions.
So agent monitoring needs three layers working together: traces to see what happened, evals to judge whether it was good, and alerts to tell you when the pattern changes.
Which signals to log for every agent run
Give each agent run its own record with a unique run ID. Every model call, tool call and decision inside that run should link back to it. At minimum, log these fields:
Run level
• Run ID, user or tenant ID (hashed if needed), session ID and timestamp
• Agent version, prompt version, model name and model version string
• Final outcome: completed, failed, handed off to a human, abandoned by user
• Total steps, total latency, total input and output tokens
• Stop reason: answered, step limit reached, timeout, guardrail triggered
Step level
• Step number and type (model call, tool call, retrieval, guardrail check)
• Full prompt and response for model calls, or a redacted version where privacy rules require it
• Tool name, arguments, raw result, status and duration
• Retrieved document IDs and their relevance scores for RAG steps
• Any fallback taken, such as switching to a backup model
Outcome level
• Explicit user feedback: thumbs up or down, edits to the agent output, a repeated question
• Implicit signals: user abandons the session, opens a support ticket, or redoes the task by hand
• Business result where you can link it: ticket resolved, booking created, refund issued
A simple decision rule: if you would need a field to explain a bad answer to a customer, log it. Storing a few extra fields is far less painful than spending a week guessing why the agent misbehaved.
How to trace tool calls end to end
Most agent failures hide in tool calls, because that is where the agent touches real systems. Model each run as a trace and each step as a span, the same pattern distributed tracing uses for microservices. The OpenTelemetry GenAI semantic conventions give you standard attribute names for model and tool spans, which keeps you portable across observability vendors.
Structure it like this:
• Root span: the agent run, with run level attributes.
• Child spans: each model call and each tool call, in order, with parent links so nested sub agents show up as a tree.
• Downstream spans: when a tool calls your own API or database, pass the trace context along so the database query appears under the tool call that caused it.
A few details that save hours during an incident:
• Log tool arguments exactly as the model produced them, before any validation or cleanup. Malformed arguments are often the first sign that a prompt or model change broke something.
• Record retries as separate spans, not as one longer span. Three failed calls and one success tells a different story than one slow call.
• Tag side effects. Mark spans that write data, send messages or trigger payments so you can filter for every run that changed something in the real world.
• Capture the tool result the model actually saw, including any truncation. If you cut a long result to fit the context window, the agent may have missed the line that mattered.
If your agent uses MCP servers, put the server name and version on every tool span. Our guide to running an MCP server in production covers the server side of this.
Redact before you store. Run personal data, secrets and payment details through a scrubber at the logging layer, not later in a dashboard. Traces get shared widely during debugging, so treat them like production data.
How to keep evaluations running after launch
Most teams build an eval set before launch, get a good score, and never run it again. That is how drift sneaks in. After launch, evals should run in the background against live traffic and against every change you ship.
1. Offline regression suite. Keep a fixed set of test cases with known good outcomes. Run it on every prompt change, model version change, tool change and dependency upgrade. Block the release if key scores drop.
2. Online sampled evals. Pick a sample of real production runs every day and score them automatically. Useful checks include:
• Did the agent call the right tools in a sensible order?
• Is the final answer grounded in the tool results and retrieved documents?
• Did it follow policy rules, such as never approving a refund above its allowed limit?
• Did it complete the whole task, not just the first part?
3. LLM as judge, with calibration. Using a model to grade outputs scales well, but judges drift too. Have humans label a small batch each week and compare their verdicts with the judge. If agreement drops, fix the judge prompt before you trust its scores.
4. Human review queue. Route low-scoring runs, negative feedback and high-risk actions to a queue that a domain expert reviews. Their labels become new test cases.
5. Turn every incident into a test. When a real failure reaches a user, add that exact input to the regression suite. Over time your eval set starts to match how users actually behave, not how you imagined they would.
Use deterministic checks where you can. Schema validation, required tool calls and forbidden phrases are cheaper and more reliable than a model grader. Save judge models for questions of quality and correctness that rules cannot express.
Alerts that tell you an agent is drifting
Good agent alerts compare current behavior with a recent baseline. Fixed thresholds work for hard failures, but drift shows up as a slow shift, so compare against the same window last week. These are the alerts worth setting up first:
Hard failure alerts (page someone)
• The error rate for any single tool climbs above its normal range
• Runs hitting the step limit or timeout spike
• Guardrail blocks jump, which can signal abuse or a prompt injection attempt
• Any write action fails validation after the agent reported success
Drift alerts (review within a day)
• Average steps per run rises: the agent is working harder for the same tasks
• Tokens per run rise without a feature change, which drives your model bill up directly
• Online eval scores fall below the baseline for two or more days in a row
• Tool selection mix shifts, such as a sudden drop in calls to a key tool
• Handoff-to-human rate or refusal rate moves sharply in either direction
User signal alerts
• Negative feedback rate rises
• More users repeat the same question within a session
• Support tickets mentioning the agent increase
Annotate your dashboards with every deploy, prompt change and model version change. When an alert fires, the first question is always what changed, and that annotation often answers it in seconds.
Watch spikes in blocked or unusual tool calls closely. They can be an attack rather than a bug. Our article on AI agent prompt injection explains what to look for.
What to do when an alert fires
Monitoring only pays off if the team can act fast. Write a short runbook before you need it:
• Kill switch: a flag that turns off the agent or a single risky tool without a deploy, falling back to a human queue or a simpler flow.
• Version rollback: store prompts and agent config as versioned artifacts so you can roll back a prompt as easily as code.
• Pinned models: call a specific model version, not a floating alias, so provider updates happen when you choose.
• Replay: rerun failed traces against the previous version to confirm whether the change caused the failure.
• Blast radius query: find every run with a side effect since the problem started, so you know which customers to contact.
Decide in advance who owns agent alerts. In many teams nobody does, because the agent sits between product, data and engineering. Give it one named owner with the authority to flip the kill switch.
A practical two-week plan to get visibility
You do not need a perfect platform to start. A focused team can get useful visibility in about two weeks:
Days 1 to 3: add run IDs, version tags and structured trace logging for every model and tool call. Set up redaction.
Days 4 to 6: build a dashboard showing outcome, steps, tokens, latency and tool errors per run. Annotate deploys.
Days 7 to 9: build a regression suite from real traces, including known failures. Wire it into your release pipeline.
Days 10 to 12: start daily sampled online evals and a weekly human calibration batch.
Days 13 to 14: set up the hard failure and drift alerts, write the runbook and test the kill switch.
One trade-off to keep in mind: logging full prompts and responses gives you the best debugging data but raises storage and privacy concerns. A common middle path is to keep full traces for a sample of runs plus every failed or flagged run, with lightweight metrics for the rest.
If your agent went live faster than its monitoring did, you are not alone. Our guide on moving an AI pilot to production covers the wider hardening checklist. Geminate Solutions builds and hardens production AI agents, with 50+ products shipped and 10M+ requests per minute handled in production (exam platform). See our AI agent development services if you want a team to set this up with you.
