How to measure whether your AI feature is actually being used

Count completed tasks rather than messages. Five numbers hold up in a board review: task completion rate, repeat use by the same person inside a fixed window, escalation rate to a human, deflection measured against a pre-launch baseline, and movement in one account metric you already report. Each needs a record written by whatever ran the agent, and a baseline captured before launch.

The six steps below define the unit of use, put instrumentation where the outcome actually is, split adoption into three separate questions, break every number down by tenant, tie one of them to an account outcome, and describe the shape of a healthy funnel.

Messages sent goes up whether the feature works or not

The dashboards available on day one count messages sent, sessions started, and monthly active users of the feature. A person who asks a question, reads a confidently wrong answer, rephrases and asks again produces two messages where a working assistant produced one. Confusion traffic and engagement traffic are the same line on that chart.

Leadership is asking a different question anyway. Renewal, support cost and expansion are the terms of the conversation, and message volume speaks to none of them, so the meeting turns into an argument about whether the chart means anything.

The averaging problem is worse when your own customers use the agent. Twelve thousand messages can come from four enthusiastic accounts while three hundred others opened the assistant once and never returned. How many accounts formed a habit is the number under discussion, and no global counter reports it.

How do I measure if our AI feature is working

Five metrics, each defined precisely enough that two people computing it independently get the same answer. Satisfaction scores are absent from the list on purpose, for the reason given further down.

Step 1: define the unit of use

The unit is a task attempt, not a message and not a session. Write the definition down before any instrumentation: an attempt begins when a user submits a request matching an intent you support, and ends in exactly one of four states, which are completed, abandoned, escalated, or refused as out of scope.

Name the tasks individually. For a billing assistant, "explain this charge" and "issue a refund" are separate tasks with separate completion definitions, and merging them produces a rate that moves when the mix shifts rather than when quality shifts. Out-of-scope requests get their own bucket, since counting them as failures makes a well-behaved refusal look like a defect.

Some tasks end in a verifiable action and some end in advice. Where a terminal tool call exists (refund issued, ticket created, record updated), completion is a fact you read off the trace. Where it does not, completion has to be graded, and the methods for that are covered in evaluation metrics without ground truth.

Step 2: instrument at the execution, not at the click

Front-end analytics knows a button was pressed and a panel was opened. It does not know whether the search tool returned an empty array, whether the refund call came back 200, or whether the model refused. Every fact worth reporting lives in the run, so the record has to be emitted by the runtime that executed it, whether you built that layer yourself or bought an AI agent platform.

One row per execution, written server side, carrying enough to recompute all five metrics later:

{
  "execution_id": "exec_9f2c41d8",
  "tenant_id": "acct_1042",
  "end_user_id": "u_88131",
  "surface": "web_chat",
  "task": "explain_charge",
  "terminal_state": "completed",
  "completion_evidence": "tool:get_invoice_lines returned 200 with 3 rows",
  "escalated": false,
  "turns": 3,
  "cost_usd": 0.0412,
  "config_version": "v14",
  "started_at": "2026-09-01T14:02:11Z"
}

Client-side instrumentation also decays quietly as surfaces multiply. The day the same agent answers in Slack, over a REST endpoint a customer's backend calls, and on a nightly schedule, the web analytics tag covers a shrinking fraction of real usage while the chart it feeds looks unchanged.

Step 3: separate tried, retained and relied on

One adoption percentage compresses three unrelated questions into a number nobody can act on. Report them separately:

1. Tried. Distinct end users with at least one task attempt in the period. This measures discovery and placement in the product, and it responds to a banner or a launch email.
2. Retained. Users with attempts in two consecutive windows, measured within a cohort so that new exposure does not inflate it. This measures whether the first experience was good enough to come back to.
3. Relied on. Completed attempts inside a workflow that has a business outcome attached, or the share of a task type that now goes through the assistant instead of the old path. This is the number that argues for renewal.

High tried with near-zero retained is a first-run quality problem, and the conversations that caused it are recoverable from your traces. The detection loop for that is in finding silently failing conversations.

Satisfaction thumbs measure who clicks, not who was helped

Thumbs up and down capture the small, self-selected group willing to rate an interaction, and that group skews toward people who were either delighted or angry enough to act. A 2025 Google study of an internal developer assistant, Reading Between the Lines, measured how small: across 372 developers over nine weeks, ratings arrived on about 0.6 percent of the roughly 36,000 turns and touched 2 percent of conversations. That is one product with one population, so treat it as an order of magnitude rather than a benchmark. The user who got a plausible wrong answer, believed it, and left satisfied is the most expensive case on the list, and they will rate you positively.

Keep the control if you have it, because a downvote is a cheap pointer into a trace worth reading. Treat it as a sampling mechanism for review rather than as a metric with a target.

Step 4: break every number down by tenant

Compute all five metrics grouped by tenant and look at the distribution rather than the mean. Report the median account, the count of accounts below a floor you set, and the count with zero attempts in the period. A tenant whose knowledge base was never indexed, or whose tool credentials expired in March, fails every attempt while a global success rate moves by a fraction of a percent.

Per-tenant configuration makes accounts non-comparable unless you record which configuration ran. Different tool sets, different prompts and different enabled features produce different completion rates for reasons that have nothing to do with the users, which is why config_version sits on the row above. Cost belongs in the same breakdown, since cost per completed task varies more across tenants than anyone expects, and the pricing consequences are covered in pricing AI features.

Step 5: tie one number to an account outcome

Pick a single metric your company already reports, ideally one somebody is compensated on: support contacts per account, time to first value, seat expansion, or renewal rate. Record its current value for a defined cohort before the feature ships. That measurement cannot be reconstructed afterward, and its absence is the reason most launches end in an unwinnable attribution argument six months later.

A staged rollout gives you the comparison group for free. Ship to a slice of accounts, hold the rest for a defined period, and compare the exposed and unexposed cohorts on the same metric with the same window. Deflection specifically needs the baseline expressed as a rate, such as contacts per hundred active users over the eight weeks before launch, because raw ticket counts move with seasonality and account growth.

Claim less than the data supports. A cohort comparison over a few weeks is directional evidence, and saying so protects the number when someone senior pushes on it.

Step 6: what a healthy funnel looks like

Shape is readable even where thresholds are not. The published benchmarks are aggregates over product analytics vendors' own installed bases, and they report general feature adoption rather than AI feature adoption, across products with different placements, different pricing and different user populations. Importing a threshold from one gives you a target that means nothing about your own feature.

A funnel that is working shows tried rising with exposure and then flattening as the addressable population saturates. Retained holds as a roughly stable fraction of tried within each cohort rather than decaying week over week. Completion stays flat or rises as volume grows, which is the signal that new usage is not lower quality than early usage. Escalation falls after the first weeks of fixes and then settles.

The per-tenant distribution should tighten over time as configuration problems get found and fixed. Two failing signals deserve attention ahead of the others: a completion rate that falls as volume rises means the feature only handled the easy early cases, and a retained fraction that decays inside a fixed cohort means people tried it and decided against it.

Where this gets easier

Most of this work is instrumentation you have to own somewhere, and the question is only whether the runtime hands it to you or you assemble it from logs. Runtype records each execution as a trace with per-step inputs, outputs and tool calls including handoffs, plus cost per execution, under the tenant and end-user identity the run was authorized with. Terminal state, completion evidence, repeat use and per-tenant breakdowns are then queries over one store rather than a separate analytics pipeline to keep in sync as surfaces are added.