Vercel AI SDK for the interface, what runs behind it

Keep the Vercel AI SDK. For the interface it is the strongest option available: useChat on the client, streamText behind a route handler, one model interface across every provider. What it does not carry is a backend agent runtime, so the durable turn, the tenant model, the eval loop and every non-web surface stay yours.

This page is written for a team that has already shipped a chat feature with the SDK and is now putting it in front of paying customers, where several tenants share one deployment. Nothing below asks you to remove the SDK.

What the Vercel AI SDK does well

The client half is the part with the least competition. useChat from @ai-sdk/react holds an array of UIMessage values, each split into typed parts for text, reasoning, files and tool calls, and re-renders as tokens arrive. Its transport (DefaultChatTransport) points at any URL, so the endpoint it posts to is a decision you make rather than a shape the hook imposes. A tool call renders while its arguments are still arriving, because the part is in the message array before it is complete.

The server half is a small set of functions that hide a large amount of provider variance. streamText takes messages, a model and a tool set and returns a result you hand back to the browser:

// app/api/chat/route.ts, AI SDK 7
import {
  convertToModelMessages,
  createUIMessageStreamResponse,
  isStepCount,
  streamText,
  toUIMessageStream,
  type UIMessage,
} from 'ai'
import { anthropic } from '@ai-sdk/anthropic'
import { tools } from '@/lib/tools'

export async function POST(req: Request) {
  const { messages }: { messages: UIMessage[] } = await req.json()

  const result = streamText({
    model: anthropic('claude-sonnet-5'),
    messages: await convertToModelMessages(messages),
    tools,
    stopWhen: isStepCount(8),
    telemetry: { functionId: 'support-chat' },
  })

  return createUIMessageStreamResponse({
    stream: toUIMessageStream({ stream: result.stream }),
  })
}

Four things in that file are worth naming individually. The provider abstraction means anthropic(...) swaps for openai(...) on one line, and createProviderRegistry or customProvider gives you aliases such as chat-default that resolve to a real model in configuration rather than in code. Tool calling is defined with the tool() helper and a schema, and the SDK validates arguments against that schema before your function runs, so a malformed call from the model surfaces as an error rather than as a crash inside your handler. Structured output has its own entry points, generateObject and streamObject, which parse and validate the model's JSON and give you a typed value.

The fourth is the loop itself. stopWhen: isStepCount(8) bounds a multi-step turn, prepareStep lets you change the model, the instructions or the available tools between steps, activeTools narrows the set the model can see on a given call, and onStepEnd fires after each one with the tool calls and results from that step. Version 7 also packages a model, a tool set, stop conditions and an approval policy into a reusable ToolLoopAgent. Check that name against your lockfile: it was Experimental_Agent through version 5 and took its current name in version 6, alongside system becoming instructions and a stopWhen default of twenty steps.

Underneath all of it sits a middleware layer most teams find later than they should. wrapLanguageModel composes behavior around any model instance, and the shipped middlewares cover the cases people otherwise hand-roll: defaultSettingsMiddleware for baseline call settings, extractReasoningMiddleware for providers that return reasoning inside the text stream, and simulateStreamingMiddleware for models that only answer in one shot. Writing your own is a small object with a wrapGenerate hook and its streaming equivalent, which is a reasonable place to put redaction or a request log that has to run regardless of call site.

The seam

Everything above happens inside one HTTP request, in your application process, for the lifetime of one response. That is the correct scope for a library of this kind, and it is also where the seam is. Four things sit on the far side of it.

The turn is bound to the connection. If the browser tab closes, the deploy rolls, or the function reaches its maximum duration, the in-flight turn is gone with no record of how far it got. Duration is the least of it now: as of September 2026 a Vercel function defaults to 300 seconds on every plan, and Pro and Enterprise teams can raise a single function to 800 seconds, or to 1,800 seconds in the extended-duration beta. The gap is the run handle, so a turn that dies at second 700 leaves the client nothing to ask about and nothing to resume.

Tenant is a variable in your code, not a concept the SDK holds. functionId groups telemetry by function, and any tenant attribute you attach is one you remembered to attach at that call site. Nothing rejects a request whose caller identity was never established, because the SDK was never told what a caller is. The background job that re-summarizes a thread and the webhook that replays a failed turn are two more call sites that can forget.

There is no eval loop. The SDK gives you the pieces to run a model against an input, and it has no dataset, no judge, no run-to-run comparison and no record of whether last week's prompt edit made the agent worse. Teams usually discover this the first time a model version changes underneath them.

Two consequences of the first three are worth stating on their own, because they are what customers notice. Cost arrives as one provider invoice for the whole deployment, and splitting it by customer afterward depends on every call site having tagged the turn correctly, including the ones that ran on a cron. Human approval is the other, and here the SDK gives you more than it used to: marking a tool user-approval through the toolApproval option on generateText or streamText, which supersedes the deprecated needsApproval on tool(), returns an approval request instead of executing. Resuming is a second model call carrying a tool-approval-response message, so the transcript, the tenant and the pending call still have to be stored somewhere that outlives the function, forty minutes later, after the reviewer comes back from lunch.

Surfaces multiply faster than the loop does. useChat is React in your own app, so the same agent reached from Slack, from SMS, from an MCP client, or from a customer's own backend over an API key means a new transport and a second copy of the loop configuration for each one. Those copies drift, and the drift shows up as an agent that behaves differently in Slack than on the web for reasons nobody can reconstruct.

How to close it

The arrangement that holds up keeps the SDK on the interface and moves the loop behind the route. Five steps, in the order they cause the least disruption.

1. Leave the client alone. useChat, the message parts, the streaming render and your own composer stay exactly as they are. The hook only needs the endpoint to keep returning a UI message stream.

2. Make the route handler a proxy. Instead of calling streamText in the handler, forward the turn to the service that owns the loop and pipe its stream straight back. The tool set, the model choice, the step budget and the system prompt move with it, into configuration that one deployment reads rather than into a file the app bundles.

3. Inject tenant context server-side, in that handler. Read the organization and end user from your session, never from the request body, and send them as named fields alongside the message:

// app/api/chat/route.ts
const session = await auth()
if (!session) return new Response('Unauthorized', { status: 401 })

const upstream = await fetch(`${process.env.AGENT_BASE_URL}/runs`, {
  method: 'POST',
  headers: {
    authorization: `Bearer ${process.env.AGENT_API_KEY}`,
    'content-type': 'application/json',
  },
  body: JSON.stringify({
    message: latestUserText(messages), // your own helper
    tenantId: session.orgId,
    endUserId: session.userId,
  }),
})

return new Response(upstream.body, {
  headers: { 'x-vercel-ai-ui-message-stream': 'v1' },
})

The two identity fields travel as request metadata, so the model never sees them in a prompt and cannot be talked into changing them. Piping upstream.body through unchanged is only correct when the service already speaks the UI message stream format the hook parses; if it emits its own event names, translate the frames in the handler and keep that translation in one file.

1. Export OpenTelemetry from whatever still calls a model in the app. A title generator, a classifier on a background job or a one-shot generateObject will stay in your codebase, and each one should emit spans. In AI SDK 7 that is one registration at startup, registerTelemetry(new OpenTelemetry()) from @ai-sdk/otel, after which every call emits telemetry and a per-call telemetry option only adds a functionId or opts back out. Configure a standard exporter through environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT="https://collector.example.com/v1/otel"
OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ${OTEL_TOKEN}"

Set recordInputs: false on any call that handles regulated data, since prompt and tool arguments are recorded by default. The mechanics of pointing an externally built agent at a trace backend are covered in instrumenting an external agent.

1. Keep the eval set beside the loop, not beside the UI. Cases belong where the tool calls actually execute, because a judge that cannot see the tool result is grading prose. Promote real failures into cases as they happen.

What to keep from the SDK after all of that: the client hook and its transport, the provider registry for in-app calls, generateObject anywhere a single validated call is the entire job, and the tool schemas themselves, which usually port to a backend tool definition with only the wrapper changing. If you are still deciding which loop to run behind the route, agent framework comparison covers the trade-offs, and exposing an agent as an API covers the case where your customers call it directly rather than through your UI.

Where Runtype fits

Runtype is a runtime and a record for the layer behind the route, and it changes nothing on the interface. The lightest way in leaves streamText where it is: register the SDK's OpenTelemetry integration, point a standard OTLP exporter at https://api.runtype.com/v1/otel, and each call becomes a run with a trace tree, structured logs, tool calls with arguments and results, token usage and a display-only cost estimate. The endpoint accepts traces from Vercel AI SDK apps, LangChain and hand-written loops, so the background classifier and the title generator that stay in your codebase land beside the chat route. Point exactly one instrumentation at it, since two doubles the tokens and the cost, and your provider keeps invoicing you as before (reporting external telemetry).

The second way in registers the loop as an external agent, which fits a route handler that is already a proxy. The endpoint speaks Runtype's unified stream or A2A, Runtype calls it, and useChat keeps talking to your route while your route talks to Runtype. The same agent can then be embedded in the open-source Persona chat widget, added to a product as a capability, reached from Slack, SMS, a REST API, an MCP server and A2A, and put on a schedule, with tool calls and cost recorded per run. The copies of the loop configuration that drift between surfaces collapse into one definition.

The third way points the tools the other direction. An MCP surface publishes a product's flows, agents, records, skills and tools as MCP tools, so your streamText loop stays the orchestrator and calls Runtype for the parts worth centralizing, with the tool schemas in your repo untouched. The fourth moves one capability off the AI SDK route and into a native flow or hosted agent, taken after a suite harvested from real runs can prove parity, then gated by runtype eval run in CI, which returns a non-zero exit code on a regression.

Tenancy is declared per resource as internal, tenant-isolated or end-user-isolated, with an assurance floor of asserted or verified evaluated before execution, so the request your handler forwards is rejected when its identity scope falls short. Every trace and every cost figure is filed under the tenant and end user it ran for, which is the split a provider invoice never gives you. Long-term memory, when enabled, is keyed per agent, tenant or end user (end-user identity).

Behind the route, a turn returns a run handle, keeps running past a dropped connection, and can be polled, with a wall-clock budget per turn. The bounds you set in stopWhen have counterparts in maxToolCalls per turn and loopConfig.maxTurns per run, and an approval gate holds a refund tool for a human answer with a timeout that defaults to 5 minutes, then resumes the same turn. Runtype runs as managed cloud or self-hosted on your own infrastructure.