AI governance for a product team shipping one feature

AI governance is the set of decisions, records and controls that let an organization say what an AI system is for, who is accountable for it, how its risks were measured, and what happens when it misbehaves. For a product team shipping one feature, it reduces to a short list of artifacts the team already produces and only needs to keep in the right shape.

The two references that matter are the NIST AI Risk Management Framework, voluntary and organized into four functions (Govern, Map, Measure, Manage), and the EU AI Act, law that sorts systems into risk tiers. A customer-facing assistant on a third-party model usually lands in the Act's transparency tier, and the evidence both ask for is mostly execution records you already keep. The version for a team with no compliance function is AI governance for product teams.

Why AI governance is hard

Both frameworks address an organization or a provider, so a team with one assistant to ship finds forty pages about programs and two paragraphs about the thing it is building. Scoped to one feature, the hard parts are these four.

The risk tier is decided by use, and use drifts

Under the EU AI Act the same model, prompt and tool set can sit in different tiers depending on use. An assistant that answers billing questions falls under the transparency obligations in Article 50. Wire the same assistant to decide credit limit increases or screen job applicants and it becomes a high-risk system under Annex III. Features drift that way one request at a time, and nothing in the codebase flags the crossing.

Policy documents rot faster than the system changes

A governance document describes the agent as it was on the day of writing: this model, this prompt, these eleven tools. Two weeks later the prompt has been edited three times, the model alias resolves to a new version, and a fourteenth tool has appeared, yet a reviewer reads the document as current. Any framework asking for "documentation" is asking for a versioned record generated from the running configuration.

Logs exist but do not carry what an auditor asks

An auditor's question is narrow: for this conversation on this date, which model version ran, which prompt version, which tool calls with which arguments, under which tenant and end user, and who approved the refund. A request log with a latency and a status code answers none of those. A log carrying the full context holds customer personal data with weaker access controls than the production database; see PII redaction for LLM logs.

Human oversight that exists only in the diagram

Article 14 of the EU AI Act and the Manage function of the RMF both want a person who can understand what the system is doing, override it and stop it. Many designs satisfy this with a box labeled "human review" that resolves to an approval nobody is on call for, a timeout that fails open, or an approver who reads the agent's explanation and clicks approve. That explanation is text the agent produced, and under prompt injection it is text an attacker produced. A gate a human can actually operate is the subject of human-in-the-loop AI.

How it works: two frameworks, one feature-level checklist

NIST AI RMF, applied to one feature

NIST published AI RMF 1.0 in January 2023 and, in July 2024, a Generative AI Profile (NIST AI 600-1) listing twelve risks specific to generative models, among them confabulation, data privacy and value chain and component integration. The framework is voluntary, and its unit of work is the function.

EU AI Act: find the tier, then the dates

The EU AI Act (Regulation (EU) 2024/1689) entered into force on 1 August 2024 and sorts systems into four tiers by use rather than by the technology inside. Prohibited practices under Article 5 (social scoring, certain manipulative systems, untargeted scraping of facial images) are banned outright. Anything not in a higher tier is minimal-risk with no specific obligation, though the Article 4 AI literacy duty applies to providers and deployers of any system.

High-risk systems, listed in Annex III (employment, education, credit and insurance, law enforcement, biometrics, among others), carry the heaviest obligations: a risk management system (Article 9), data governance (Article 10), technical documentation (Article 11), automatic logging (Article 12), human oversight (Article 14), accuracy and robustness (Article 15), a conformity assessment and EU database registration. Limited-risk systems, including anything that talks to people, carry the Article 50 transparency obligations.

Article 50(1) requires a system that interacts directly with people to be designed so that they are informed they are dealing with an AI, unless that is obvious from context; Article 50(2) requires machine-readable marking of synthetic content. Annex III applies by use: a support assistant is outside it, a system that ranks candidates or scores creditworthiness is inside. Article 6(3) carves out an Annex III system that poses no significant risk of harm and only performs a narrow procedural or preparatory task, though profiling of natural persons is always high-risk.

The general-purpose AI model obligations in Chapter V (technical documentation, downstream information, a copyright policy and a training-content summary under Article 53, plus evaluation, adversarial testing and incident reporting for systemic-risk models under Article 55) fall on the model provider. A team on a hosted model receives that documentation. Under the Commission's guidelines of July 2025, a downstream modifier becomes the provider of the modified model only on a significant change, indicatively a third of the original's training compute.

Article 113 as enacted set prohibitions and AI literacy from 2 February 2025, general-purpose model obligations and penalties from 2 August 2025, the rest from 2 August 2026, and Annex I safety components from 2 August 2027. Regulation (EU) 2026/1744, the Digital Omnibus on AI, in force since 27 July 2026, moved the high-risk requirements to 2 December 2027 for Annex III systems and 2 August 2028 for Annex I. Article 50 transparency still applies from 2 August 2026, and from 2 December 2026 the new prohibition on non-consensual sexual deepfakes applies and generative systems already on the market must meet the Article 50(2) marking duty.

The artifacts that satisfy both

Both frameworks ask for a small set of artifacts, and ISO/IEC 42001 draws on the same set if certification is ever on the table.

A team that runs evals and keeps traces already produces most of the right-hand column. The work is one file per feature pointing at the live sources rather than copies.

feature: billing-assistant
owner: platform-team@example.com
intended_purpose: >
  Answers billing and account questions for signed-in customers.
  Cannot change a plan or issue a refund without human approval.
eu_ai_act:
  tier: limited
  basis: "Interacts with natural persons (Article 50); not an Annex III use"
  disclosure: "You are chatting with an AI assistant."
nist_ai_rmf:
  govern:
    review_cadence: quarterly
    change_log: version-history
  map:
    data_flow: docs/billing-assistant/data-flow.md
    tool_inventory: docs/billing-assistant/tools.md
  measure:
    eval_suite: billing-assistant-regression
    pass_threshold: 0.9
    rerun_on: [prompt, model, tools]
  manage:
    approval_required: [issue_refund, change_plan]
    max_tool_calls_per_turn: 10
    incident_runbook: runbooks/billing-assistant.md
records:
  keep: [model_id, prompt_version, tool_calls, tenant_id, end_user_id, approver, outcome]
  redact: [EMAIL_ADDRESS, PHONE_NUMBER, CREDIT_CARD]
  retention_days: 180

The eval half is the subject of AI agent evals, and the controls in the manage block are the enforcement layers ranked in LLM guardrails.

What changes when the agent is customer-facing and multi-tenant

An internal assistant has one deployer, you, and one population of users. Exposing the same agent to your customers' own users changes who holds which obligation and multiplies the evidence you must produce.

You become a provider to many deployers. Under the EU AI Act your customers are deployers, with duties of their own: using the system per its instructions, human oversight by competent people, keeping logs for at least six months, and, for public bodies and credit or insurance deployers, a fundamental rights impact assessment under Article 27. They can meet those duties only with what you give them, so the intended-purpose statement, instructions for use and access to their own logs become part of the product.

The tier can differ per customer. Your intended purpose says billing questions; one customer wires the same agent into an eligibility decision. Your terms and the tool set each tenant may enable bound "reasonably foreseeable misuse", and a per-tenant change there needs its own version history.

Disclosure is per surface. Article 50 attaches to the interaction, so the "you are talking to an AI" notice has to appear in the customer's widget, the Slack bot and the SMS thread, and a customer must not be able to strip it.

Evidence is per tenant. A customer's auditor wants that customer's records under that customer's retention and redaction settings, which requires tenant and end-user identity stamped on each execution record and a store that isolates tenants rather than a dashboard filter hiding rows. That identity also scopes each tool call to the requesting tenant, which is where governance meets AI agent security.

Where Runtype fits

Runtype is where an agent your customers use gets a declared identity, an enforced oversight gate and a record of every run.

Registration comes first, and the agent code is unchanged. An external agent whose endpoint speaks Runtype's unified stream or A2A is called by Runtype, so it reaches product surfaces (web chat, Slack, REST, MCP), schedules and approval gates, with tool calls and cost recorded per run.

Identity is a declared property of each resource: a tenancy strategy of internal, tenant-isolated or end-user-isolated with an assurance floor of asserted or verified, evaluated before execution starts (end-user identity). Every trace and cost figure is filed under the tenant and end user it ran for.

An agent that stays where it is can still export OpenTelemetry traces to https://api.runtype.com/v1/otel for the same run records, and an MCP surface points the other way, exposing flows, agents, records and tools to your existing loop. Rebuild one capability natively when a harvested suite can prove parity, and its published versions become the change log.

Execution records are the Measure and Manage evidence: per-step input and output, each tool call with its arguments and result, latency and cost. Logging verbosity and PII redaction are set per product, surface or agent and resolved at dispatch, the more specific layer winning; redact mode masks email addresses, phone numbers, checksum-validated card numbers, US and Canadian social insurance numbers and IBANs by default.

Human oversight is an enforced gate: approval per tool or for every tool, a five-minute default timeout, and a run that resumes only on an approval. The reason the approver reads is the agent's own _approvalReason, shown as its claim and never used to decide anything. A turn is bounded by maxToolCalls (default 10, at most 100) and a run by maxTurns (1 to 100).

Agents and flows carry draft and published versions, so "which prompt was live on this date" is a lookup, and the runtime can be self-hosted (self-hosting). None of this is a certification: the obligations stay with you as provider or deployer, and what changes is how much evidence exists by default.