Amplitude Agent Analytics: what it measures and how to instrument it

What it is

Agent Analytics is Amplitude's feature for measuring the AI agents your own team builds and ships - a support assistant, an in-product copilot, an internal tool with a chat interface - not Amplitude's own AI agents that analyze your instance for you (that's a separate capability covered in our guide to Amplitude's AI agents). It decomposes each agent conversation into a three-level hierarchy Amplitude calls sessions, turns, and spans: a session is one unit of work end to end, a turn is one exchange within it, and a span is a single operation inside a turn - a tool call, a retrieval step, a model call. Each level is written out as an ordinary Amplitude event carrying the user's user_id and an agent ID, living in the same project as the rest of your product data rather than a separate observability silo.

Amplitude runs three layers of quality measurement on top of that event stream, per its own documentation. Signals are automatic, built-in evaluations run on every closed session with no configuration required - task completion, response quality, user intent, session safety, user friction, negative feedback, and data quality - written to a `[Agent] Session Record` event that carries the rationale behind each score, not just the number. Evaluators are criteria you define yourself (policy compliance, groundedness, tone) for Amplitude to score using either a code-based check or an LLM-as-judge, written to `[Agent] Evaluator Result` events. Scores are real user feedback - a thumbs rating, a CSAT prompt - that you send in yourself; per Amplitude's own framing, the platform never generates a Score, only stores and surfaces the ones you provide. [Verify the exact evaluator list and event names against current docs - this is one of the areas Amplitude is actively shipping new evaluators into.]

On top of the embeddings it already computes to run those evaluations, Agent Analytics automatically clusters sessions by intent into a topic and subtopic hierarchy - surfacing what people are actually asking the agent for, and which of those intents the agent handles well versus where it consistently falls apart. It's the same job a support lead does by hand reading transcripts, run continuously and at the full volume of traffic instead of a weekly sample.

When you need it

• You've shipped, or are about to ship, an AI agent or assistant inside your own product, and the question that matters is not "did it respond" but "was the response good, and does that connect to whether the user converts, retains, or churns."
• Your current AI observability is infrastructure-level - latency, token cost, error rate - and can't tell you whether the agent actually resolved what the user asked, which is a product question, not an infrastructure one.
• You want agent quality sitting in the same funnels, cohorts, and retention analyses the rest of your product team already trusts, instead of a separate dashboard only the AI team ever opens.

Implementation, done properly

1. Pick the instrumentation path that matches how your agent is built. Amplitude documents the Node or Python AI SDK, with CLI setup assistance, as the fastest path for an agent you're building directly; OpenTelemetry GenAI-format traces and a direct HTTP API are documented paths for an agent already emitting traces elsewhere or built on a framework that speaks OTel; a warehouse-import path exists for agent logs that already land in your own data warehouse. Verify current SDK language support and OTel trace-format compatibility against Amplitude's docs before committing to a path, since this integration surface is new and expanding.
2. Decide the privacy and content-capture mode deliberately before turning this on for real traffic. Amplitude documents multiple levels, from full conversation content with PII redaction down to metadata-only capture with no message content stored at all. Conversations are among the most sensitive data most teams will ever pipe into an analytics tool - treat the mode choice as an owned privacy decision, not a default left at whatever the SDK ships with, and loop in whoever owns your privacy posture before a wide rollout.
3. Confirm the user_id an agent event carries matches the identity your existing product events already use, not a separate agent-session identifier invented for this integration - the entire value of the feature is that agent and product events join on one identity graph, and a mismatched identifier silently breaks that join.
4. Build the failure-first view before the celebration view: a saved view for sessions where quality signals failed, letting you drill into the trace - what was asked, what tools were called, where it went sideways - before building the adoption dashboard leadership will actually ask for first. Failure debugging is the harder problem, and the one worth getting the workflow right for while the instance is still small.
5. Wire quality into the metrics leadership already reads. Build one funnel or cohort that answers the KPI question directly - does conversion, upgrade, or retention differ between users whose agent sessions scored well versus poorly - rather than leaving Agent Analytics as an isolated quality dashboard nobody connects to the numbers the business tracks.

How to verify it worked

1. In the Agent Analytics session view, confirm signals are populating on real production traffic, not just a test conversation - task completion, response quality, and friction scores should show real variance across sessions, not a flat, uniform number that suggests the evaluator isn't actually differentiating anything.
2. Pick one session you know went badly - a real complaint, a support escalation - and confirm you can find it through the quality signals rather than by manually reading transcripts, and that drilling into its trace surfaces the actual point of failure: the prompt, the tool call, or the retrieved context.
3. Build the funnel from first agent session to your real conversion or retention event, and confirm the numbers move in a sane direction as agent quality changes - a quality signal that never correlates with any downstream behavior is either mismeasuring quality or measuring something users don't actually care about.
4. Confirm the intent-cluster view stays legible at your actual traffic volume. A clustering feature that produces dozens of near-duplicate micro-clusters at low session volume needs either more data or a coarser view before it's useful for a weekly review.

Related: Amplitude AI agents: what they do and when to use them · Guide: AI consumption KPIs · Pendo Agent Analytics vs Amplitude Agent Analytics · Measuring AI-agent usage inside your product · Building behavioral cohorts in Amplitude

Elsewhere in the agentic-traffic research: What is agent analytics? The three senses, disambiguated · LLM observability, explained · From LLM evals to product metrics · The cross-tool agent-analytics landscape

Sources

• Amplitude: Agent Analytics (product page)
• Amplitude Docs: Agent Analytics overview
• Amplitude Docs: Agent Analytics taxonomy
• Amplitude: "Why We Created Agent Analytics"

This space moves fast - every landscape figure above links to the primary source it came from. Platform-specific claims are flagged in-line to verify against current vendor documentation.