Guides
How to ship agents your customers use
Procedural guides written for the team that owns an AI feature inside a multi-tenant product. Each one answers a question you can act on today.
Agent architecture
- GuideHow to design agents for work that takes minutes, not secondsRunning an AI agent task that takes minutes: resumable steps, a run handle instead of a held connection, idempotent side effects, budgets and cancellation.
- GuideHow to get JSON out of a model that you can actually put in a databaseLayer the guarantees: schema design, provider structured-output modes, validation at the boundary, retrying with the error, and the record that still fails.
- GuideHow to run agent work on a schedule without building a queueWhat scheduled agent work needs beyond a cron line: idempotency, overlap prevention, per-tenant fan-out, per-run budgets, run history and alerting.
- GuideWhat to do when the conversation outgrows the context windowWhy agents drift and forget after thirty turns, how to budget context across system, tools, history and output, and when to end a session instead.
Cost & unit economics
- GuideHow to cut LLM spend without making the product worseRank LLM cost optimization by risk to output: measure first, trim context, cache the prefix, cap loops, route by task class, then downgrade the model.
- GuideHow to stop one customer's usage from destroying your marginHow to limit AI usage per customer: rate limits versus spend limits, per-tenant and per-user budgets, run-level caps, what to do at the cap, and alerting.
- GuideHow to work out what one AI conversation actually costs youHow to turn one provider invoice into cost per turn, per conversation, per customer and per resolved task, with the retries, tools and evals included.
Multi-tenant & platform
- GuideHow to deploy AI into a customer's environment when they will not send data outCustomer VPC, on-prem and private model endpoints compared: what each costs in capability, latency, telemetry and support, and how to split the planes.
- GuideHow to let each customer configure their own agent behaviorHow to let customers configure your AI agent: a shared base, a typed set of per-tenant overrides, write-time validation, versioning and safe rollout.
- GuideHow to measure whether your AI feature is actually being usedMessage counts rise whether an AI feature helps or not. Six steps to a metric set that survives scrutiny: completion, repeat use, escalation and outcome.
- GuideHow to run one agent for many customers without leaking data between themA numbered tenant isolation checklist for a multi-tenant AI application: identity, retrieval, caches, memory, credentials, logs and evals, each with a test.
Observability & debugging
- GuideHow to debug an AI agent that did the wrong thing three steps agoA repeatable way to debug an AI agent from its trace: find the run, walk the spans backwards to the last correct output, classify the step, fix it.
- GuideHow to find the small share of conversations that are silently failingError rates stay near zero while users quietly give up. The behavioural signals that mark a silently failed AI conversation, and a review loop that ranks them.
- GuideHow to instrument an agent you built yourselfAdd OpenTelemetry tracing to a custom, LangChain or Vercel AI SDK agent loop: span layout, GenAI attributes, tenant IDs, token usage and sampling.
- GuideHow to trace an agent run across models, tools and retriesModel an agent run as one span tree: a root span per run, spans per turn, model call and tool call, plus attribute naming, retries, streaming and sampling.
- GuideWhat to log for an LLM feature, and what you will regret not loggingA capture list for LLM features grouped by the question each field answers later: version, execution, tokens and cost, outcome, retention and redaction.
- GuideWhy your agent gets stuck in a loopWhy an AI agent calls the same tool over and over, how to spot the loop in a trace, the fix for each cause, and where to set the hard call budget.
Reliability & evals
- GuideA QA process for AI features that a team without ML engineers can runA four-stage QA process for AI features: an exploratory pass, a frozen regression set, a weekly sampled review with a rubric, and one rollback metric.
- GuideHow to build your first eval set from real production conversationsA repeatable loop for mining eval cases from production traces: stratified sampling, PII and consent handling, tenant isolation, labeling, and set size.
- GuideHow to catch a model upgrade that quietly degrades your productHow to test a model upgrade before it ships: pin or route the model id, hold the suite constant, read per-case diffs, shadow live traffic, then gate.
- GuideHow to evaluate an AI agent when there is no single right answerAssertions fail when an agent has many acceptable outputs. Four layers to grade separately, rubric design, pairwise scoring, and grader validation.
- GuideHow to tell if a prompt change broke something before your users doBuild a 30 to 50 case regression suite for prompt edits: where cases come from, criteria that survive rewording, thresholds, a CI gate and red-run triage.
- GuideHow to test tool calling, not just the final answerGrading the final answer hides wrong tools, bad arguments and skipped lookups. Four levels of tool-call assertion, recorded results, live calls versus mocks.
- GuideWhat to measure when your AI feature has no ground truthA working set of LLM evaluation metrics you can compute without labels, how each one is calculated from traces and product events, and what each misses.
- GuideWhy your agent gives a different answer every time, and what to do about itWhere LLM variance comes from: sampling, batched inference, model updates, retrieval drift and context order. How to measure it and bound what is left.
Security & governance
- GuideHow to add a human approval step before an agent does something irreversibleHow to gate an AI agent's irreversible actions behind a human decision: classify tools by blast radius, pause the run, show the exact call, and record it.
- GuideHow to keep PII out of your LLM logsRedact customer PII on the write path into agent traces: what counts as PII in a chat payload, detector trade-offs, stable tokens, retention tiers and tests.
- GuidePrompt injection when your agent has real toolsHow indirect prompt injection reaches an agent through tickets, pages, tool results, and six controls ranked by whether the runtime or the model enforces them.
- GuideWhat to tell your security reviewer about your AI featureThe six artifacts an AI agent security review asks for, the questions a reviewer puts verbatim, the form of an answer that closes each, and a packet template.
Shipping to production
- GuideHow to add an AI assistant to a product people already pay forA sequenced plan for adding an AI assistant to an existing SaaS product: one job, the data boundary and auth pattern, the surface, failure behavior, one metric.
- GuideHow to expose your agent as an API your customers can callDesigning an AI agent API customers call from their own code: request and response shape, sync or async run handles, streaming, idempotency, keys, versioning.
- GuideHow to get from an impressive demo to something on-call can supportThe gap list between an AI demo and a supportable feature: identity scoping, secrets, budgets, logs, evals, rollback, escalation, alerts and a runbook.
- GuideHow to put your agent in Slack without building a Slack app from scratchMinimum scopes for a Slack bot, the 3-second acknowledgement, threading rules, approval buttons, identity mapping to your users, and multi-workspace installs.
- GuideHow to ship the same agent to your app, Slack and email without three codebasesHow to run one AI agent in your app, Slack, email and an API without forking it per channel: the agent/surface boundary, per-channel differences, and testing.
Tools, MCP & integrations
- GuideHow many tools is too many, and what to do when you pass the limitWhy tool selection degrades past twenty tools, and how to fix it: measuring accuracy, merging duplicates, namespacing, tool search and per-task pools.
- GuideHow to turn an internal workflow into a tool an agent can callTurn a CRM lookup or an internal service into an agent tool: task-shaped granularity, typed inputs, server-side credentials, timeouts and a read-only rollout.
- GuideWhy your agent calls the wrong tool, and how to fix it in the tool layerTwo tools that sound alike are a schema problem, not a prompt problem. Names, descriptions, argument enums, result shapes and a selection eval set.