Product Screenshot
Add heroImage in frontmatter to render a product screenshot here.
Agent Experience Eval
Point it at your docs domain with the questions your users ask. It fans each one out to research-enabled models, browsing agents and coding harnesses, grades every answer against your docs, and reports which agents found the right content.
Tags
Your users now ask an AI assistant or a coding agent about your product before they open your docs. If that agent cannot find your official pages, cites a stale blog post, or invents a flag, the user leaves with the wrong answer and you never see it happen. This product puts a number on that experience: give it your domain and the questions your users actually ask, and it shows which agents got them right and what to change on your site.
How it works
You start a run from the ax_agent chat or the Agent Experience Eval API and MCP surfaces. ax_start takes your domain and key prompts and sends each prompt as written to every enabled lane, with the domain as a hint rather than a restriction. It submits one batch eval per lane, three attempts per prompt by default, so a single lucky answer cannot carry a score.
A lane is a model plus the way it reaches the web. Six web lanes cover provider-native research on OpenAI, Anthropic, Gemini and xAI models, an Exa search lane, and a browse-only lane. The claude_managed lane is a Claude Managed Agent with web search, fetch, bash and file tools; two harness lanes run the opencode and pi coding agents in a sandbox.
ax_evaluate_case runs each case: it collects the lane's answer, applies deterministic checks, researches your official sources, and has a judge grade the answer on task success, coverage, accuracy, currency and citation quality. ax_build_report then aggregates the run into a leaderboard by lane, a lane-by-prompt matrix, consistency across attempts, and a ranked list of site and docs fixes written for the people who own them.
Write the key prompts the way your users write them, not the way your docs are titled. The eval sends each one unchanged, so a prompt phrased in your internal vocabulary measures how well agents read your docs, not how well they serve your users.
Measure what agents actually do
9
lanes per prompt: research models, browsing agents and coding harnesses
5
graded dimensions per answer, plus deterministic checks
3
attempts per prompt and lane by default, reported for consistency
Before and after
| Spot-checking assistants by hand | With the agent experience eval |
|---|---|
| Someone pastes a question into one chatbot and screenshots the answer | Every key prompt runs on nine lanes, three times each, with the same grading |
| Nobody knows whether agents read your docs or a third-party summary | Each answer is scored for citation quality and the share of official sources |
| A wrong answer is a vague complaint about AI | The report names the missing or outdated page and the change that fixes it |
| A docs change ships and nobody checks whether agents noticed | Rerun the same prompts and compare scores by lane against the previous run |
Deploy it
This page ships as a Runtype template. Deploying it asks for two values: your product's domain, such as docs.example.com, and your key prompts, one per line. Both become the defaults for every run, and you can override the domain, the prompts, the lanes or the number of attempts when you start a run.
The template also asks for a Runtype API key, which goes straight into the secrets vault. The run flow uses it to submit one batch eval per lane, and the per-case flow uses it to reach the Claude Managed lane agent.
Every run, case result and report is a record you can read, rerun or compare. That is the pattern Runtype is built for: fan work out to many models and agents, keep the scoring deterministic where it can be, and turn the results into changes your team can make.
Agent Experience Eval
Point it at your docs domain with the questions your users ask. It fans each one out to research-enabled models, browsing agents and coding harnesses, grades every answer against your docs, and reports which agents found the right content.
Tags
Product Screenshot
Add heroImage in frontmatter to render a product screenshot here.
Your users now ask an AI assistant or a coding agent about your product before they open your docs. If that agent cannot find your official pages, cites a stale blog post, or invents a flag, the user leaves with the wrong answer and you never see it happen. This product puts a number on that experience: give it your domain and the questions your users actually ask, and it shows which agents got them right and what to change on your site.
How it works
You start a run from the ax_agent chat or the Agent Experience Eval API and MCP surfaces. ax_start takes your domain and key prompts and sends each prompt as written to every enabled lane, with the domain as a hint rather than a restriction. It submits one batch eval per lane, three attempts per prompt by default, so a single lucky answer cannot carry a score.
A lane is a model plus the way it reaches the web. Six web lanes cover provider-native research on OpenAI, Anthropic, Gemini and xAI models, an Exa search lane, and a browse-only lane. The claude_managed lane is a Claude Managed Agent with web search, fetch, bash and file tools; two harness lanes run the opencode and pi coding agents in a sandbox.
ax_evaluate_case runs each case: it collects the lane's answer, applies deterministic checks, researches your official sources, and has a judge grade the answer on task success, coverage, accuracy, currency and citation quality. ax_build_report then aggregates the run into a leaderboard by lane, a lane-by-prompt matrix, consistency across attempts, and a ranked list of site and docs fixes written for the people who own them.
Write the key prompts the way your users write them, not the way your docs are titled. The eval sends each one unchanged, so a prompt phrased in your internal vocabulary measures how well agents read your docs, not how well they serve your users.
Measure what agents actually do
9
lanes per prompt: research models, browsing agents and coding harnesses
5
graded dimensions per answer, plus deterministic checks
3
attempts per prompt and lane by default, reported for consistency
Before and after
| Spot-checking assistants by hand | With the agent experience eval |
|---|---|
| Someone pastes a question into one chatbot and screenshots the answer | Every key prompt runs on nine lanes, three times each, with the same grading |
| Nobody knows whether agents read your docs or a third-party summary | Each answer is scored for citation quality and the share of official sources |
| A wrong answer is a vague complaint about AI | The report names the missing or outdated page and the change that fixes it |
| A docs change ships and nobody checks whether agents noticed | Rerun the same prompts and compare scores by lane against the previous run |
Deploy it
This page ships as a Runtype template. Deploying it asks for two values: your product's domain, such as docs.example.com, and your key prompts, one per line. Both become the defaults for every run, and you can override the domain, the prompts, the lanes or the number of attempts when you start a run.
The template also asks for a Runtype API key, which goes straight into the secrets vault. The run flow uses it to submit one batch eval per lane, and the per-case flow uses it to reach the Claude Managed lane agent.
Every run, case result and report is a record you can read, rerun or compare. That is the pattern Runtype is built for: fan work out to many models and agents, keep the scoring deterministic where it can be, and turn the results into changes your team can make.