How to get JSON out of a model that you can actually put in a database

Reliable JSON comes from four layers applied in order: a schema shaped so the model can follow it, the strongest enforcement mode the provider offers, validation at the boundary before anything downstream reads the object, and a decided path for the record that fails anyway. Prompt instructions are the weakest of the four and should never be the only one.

What a malformed response actually looks like

The interesting failures are the ones that survive a naive parser. A trailing comma and a "Here is the JSON you asked for:" preamble both throw immediately, which is the good case. The expensive ones pass JSON.parse and fail later: a due_date of "next Tuesday", a category of "refund" when your enum has five other values, a total field the model invented because the prompt mentioned money, or a customer email silently omitted rather than set to null, so your insert writes a row with a missing column instead of a null one.

Rate matters less than blast radius. An extraction step running inside a nightly pipeline over ten thousand tickets, or inside an agent that then calls a write tool, has no human between the bad object and the table. By the time anyone notices, the malformed rows are mixed with good ones and there is nothing in the record that says which pass produced them. This is one of the reasons a data-shaped step belongs in a deterministic path rather than in free model output, a distinction covered on the agent orchestration platform hub.

How do I make an LLM return reliable structured JSON

Five steps, in the order they should be built. Each one narrows what the next has to catch.

1. Design a schema the model can follow

Schema design does more for reliability than any prompt sentence. Four rules carry most of the weight.

Keep it flat, because every nesting level is another structure the model has to hold open, and providers state their caps in different units. As of September 2026, OpenAI's Structured Outputs allows up to 5,000 object properties total across 10 levels of nesting. Anthropic counts grammar compilation cost instead, capping one request at 24 optional parameters and 16 parameters with union types across all strict schemas. Google publishes no numbers for Gemini and says only that a very large or deeply nested schema may be rejected.

Use enums instead of free text wherever a downstream WHERE clause or switch exists. "category": "refund" is a bug; "category" constrained to five literals cannot produce it under a constrained-decoding mode, and fails validation loudly under every other mode.

Make absence a value. A field the model may not find should be ["string", "null"] and still listed in required, so "I looked and there was nothing" arrives as null rather than as a missing key. That also removes the ambiguity between a model that found nothing and a model that forgot the field.

Constrain formats the consumer is strict about. A date column will not take "next Tuesday", so put the pattern in the schema and let validation reject it.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": ["category", "severity", "customer_email", "due_date", "summary"],
  "properties": {
    "category": {
      "type": "string",
      "enum": ["billing", "bug", "feature_request", "account_access", "other"]
    },
    "severity": { "type": "string", "enum": ["low", "medium", "high"] },
    "customer_email": {
      "type": ["string", "null"],
      "description": "Email address appearing in the ticket body, or null if none appears."
    },
    "due_date": {
      "type": ["string", "null"],
      "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}$",
      "description": "Calendar date only, no time component."
    },
    "summary": { "type": "string", "maxLength": 300 }
  }
}

additionalProperties: false is what stops the invented total. Field names and description strings are read by the model, so they are prompt surface: name the field due_date rather than date2, and say in the description what null means.

2. Turn on the strongest enforcement mode your provider offers

There are three mechanisms, and they are not equally strong.

All three major providers now offer the first mechanism, under different parameter names. OpenAI calls it Structured Outputs: text.format of type json_schema with strict: true on the Responses API, or response_format on Chat Completions. Anthropic calls it JSON outputs, an output_config.format of type json_schema, generally available with no beta header. Google's current Gemini route is a response_format object carrying mime_type of application/json and the schema in schema; the older responseSchema field is marked deprecated in the API reference.

Anthropic reaches the second mechanism with strict: true on a tool definition, which applies the same grammar to tool inputs. Forcing a specific call with tool_choice still works on most models and is refused with a 400 on the newest ones, so treat a forced call as a per-model capability rather than a universal fallback.

Two properties of strict modes are worth knowing before you rely on one. Each accepts a different subset of JSON Schema, and a keyword outside the subset is usually rejected rather than quietly ignored: OpenAI enforces pattern, a fixed list of format values and numeric and array bounds, and errors on anything else; Anthropic returns a 400 for minLength, maxLength and numeric constraints, and several of its SDKs strip those into the field description instead; Gemini's documented subset covers format, minimum, maximum, minItems and maxItems with no pattern at all. The schema you send is therefore sometimes a trimmed version of the one your validator holds.

The second property is the token limit. A response that hits it stops mid-object; the finish reason says length and the text is invalid JSON even though the mode was on. Check the finish reason before you check the body.

If the extraction is already a tool your agent calls, the tool path is also where you get test coverage, since a forced call is the same code path as a chosen one. Testing agent tool calls covers how to assert on arguments rather than on prose.

3. Validate at the boundary

One function owns the transition from model text to typed object, and nothing downstream accepts a raw string. Validate against the schema the request was generated from, so the two cannot drift even when the provider needed a trimmed copy.

import json
from jsonschema import Draft202012Validator

validator = Draft202012Validator(TICKET_SCHEMA)


class SchemaMismatch(Exception):
    def __init__(self, problems, payload):
        self.problems = problems
        self.payload = payload
        super().__init__("; ".join(problems))


def parse_ticket(raw: str) -> dict:
    obj = json.loads(raw)  # trailing comma or preamble raises here
    problems = [
        f"{e.json_path}: {e.message}"
        for e in sorted(validator.iter_errors(obj), key=lambda e: e.json_path)
    ]
    if problems:
        raise SchemaMismatch(problems, obj)
    return obj

Collect every error rather than stopping at the first. A retry that fixes one field and then fails on the next costs two round trips instead of one, and the full list is what step 4 sends back.

4. Retry once, with the validation error in the message

A bare retry re-rolls the dice. A retry that shows the model the exact validator output usually succeeds, because the failure is specific and local. Append the previous assistant message and a user message built from the error strings:

{
  "role": "user",
  "content": "Your previous response failed validation:\n$.due_date: 'next Tuesday' does not match '^[0-9]{4}-[0-9]{2}-[0-9]{2}$'\n$.category: 'refund' is not one of ['billing','bug','feature_request','account_access','other']\nReturn the corrected object only."
}

Cap this at one or two attempts and count them. A retry rate that climbs from a fraction of a percent to several percent after a model version change is a regression signal you would otherwise learn about from a support ticket, and the same run-to-run variation is described in why LLM output is non-deterministic. Do not retry a length finish reason with the same prompt: raise the token limit or shrink the schema instead, because the model will produce the same too-long object again.

5. Decide in advance what happens to the record that still fails

Every extraction pipeline needs a written answer to this, chosen before the first 2am failure.

• Dead-letter the record. Store the raw text, the schema version, the validator errors and the model id, mark the record failed, and continue the batch. Best when partial results are useful and a human can requeue.
• Fail the whole run. Correct when a downstream consumer treats the batch as complete, since a batch that is quietly missing some of its rows is worse than one that did not arrive.
• Write a typed failure row. A row whose status column says extraction_failed keeps the join intact without pretending the fields are known. Use it when the record must exist for referential reasons.

The option that is never correct is coercion. Mapping an unrecognized category to other, or a bad date to today, converts a loud failure into a wrong row that no alert will ever fire on. If the extraction feeds something that writes, put the write behind the validator rather than beside it; exposing an internal workflow as an agent tool covers the same boundary from the tool side.

Why prompt instructions are the weakest layer

"Respond only with valid JSON, no markdown fences, no explanation" is worth including, and it is worth nothing on its own. The instruction competes with every other token in the context, and it loses more often as the conversation grows, as few-shot examples drift, or as a new model version reweights how strongly it follows formatting directives. Nothing about it is enforced anywhere in the stack.

Keep the sentence, put the schema in the request, and treat the prompt as the layer that improves the odds rather than the layer that makes the guarantee. The measurable version of this is a validation failure rate you track per model and per schema version, so a change in either shows up as a number instead of as an anecdote.

Where this gets easier

Runtype validates a flow step's output against a schema as part of the flow itself, so a malformed response stops at the boundary instead of reaching the next step, and the failure appears in the execution trace with the step's input, the raw output and the validation error rather than as a stack trace in a log somewhere downstream. Extracted objects land in records and collections as structured data, which means the schema that governs the write and the schema attached to the model request are configured in one place, and a record that fails validation stays visible as a failed record instead of becoming a silently wrong row.