← Back to Blog
Oct 28, 2026 · 10 min read

Stop Prompting for JSON: Constrain the Decoder with Outlines

AI AgentsStructured OutputOutlinesvLLMLocal LLM

You put "Return ONLY valid JSON" in the system prompt, set temperature to 0, and add an example. It works most of the time. Then one request comes back with "Sure, here is the JSON:" in front of the object, your agent calls json.loads, and the run dies.

Retrying covers the gap until it doesn't. The better fix is not another prompt sentence. It is to stop the sampler from ever choosing a token that breaks the structure. Outlines does that: it compiles your schema into a finite state machine and masks the logits at every step, so tokens that would produce invalid output get zero probability. The model can still pick the wrong category, but it cannot pick a category that is not in your enum.

The setup below runs on a laptop with a 1.7B model, then shows where the same idea lives in a production serving stack.

Prerequisites

  • Python 3.10 or newer
  • Around 4GB of free RAM for a 1.7B model in fp16. A GPU speeds this up but is not required.
  • Basic Pydantic v2

Step 1: install and load a model

python3 -m venv .venv
source .venv/bin/activate
pip install "outlines[transformers]" "pydantic>=2"

The transformers extra pulls in Hugging Face transformers, accelerate, datasets, and sentencepiece. outlines alone does not install them.

import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "HuggingFaceTB/SmolLM2-1.7B-Instruct"

hf_model = AutoModelForCausalLM.from_pretrained(MODEL)
hf_tokenizer = AutoTokenizer.from_pretrained(MODEL)

model = outlines.from_transformers(hf_model, hf_tokenizer)

One thing to remember about the Outlines 1.x API: it always returns a string, even when the output type is a Pydantic class. Parsing is your job with model_validate_json. The older pre-1.0 API returned parsed objects, so a lot of blog code you find will not run as-is.

Step 2: define the shape you actually need

A support agent could call this ticket triage step. The only fields the model may fill are the four below.

from typing import Literal
from pydantic import BaseModel

class Ticket(BaseModel):
    category: Literal["billing", "auth", "outage", "other"]
    severity: Literal["low", "medium", "high"]
    summary: str
    order_id: str | None

MESSAGE = "I was charged twice for order ORD-88213 and the second charge is still pending."

def prompt_for(msg: str) -> str:
    return (
        "You triage support messages.

"
        f"Message: {msg}

"
        "Fill the ticket fields. category must describe the main problem."
    )

raw = model(prompt_for(MESSAGE), Ticket, max_new_tokens=200)
ticket = Ticket.model_validate_json(raw)

print(ticket.category, ticket.severity, ticket.order_id)

Literals become the constraint. At the moment the decoder reaches the category value, only tokens that continue one of the four allowed strings survive the mask. Alternatives like "refund" or "Billing" are not "discouraged", they are removed from the vocabulary for that step.

The str | None for order_id matters too. Without it, the model has to invent an order number when the message does not contain one. With it, null is a legal output. A schema that allows null is usually better than a prompt that says "use null if unknown", because the prompt is a request and the schema is a rule.

Step 3: measure the difference instead of trusting it

Format compliance is the one thing here you can measure cheaply. Run the same messages through both paths and count valid parses.

from pydantic import ValidationError

MESSAGES = [
    "I was charged twice for order ORD-88213 and the second charge is still pending.",
    "login keeps failing with invalid token after I reset my password",
    "the dashboard has been down for 20 minutes, all our users are affected",
    "can you send last month invoice to our finance email",
]

def constrained(msg: str) -> Ticket:
    return Ticket.model_validate_json(
        model(prompt_for(msg), Ticket, max_new_tokens=200)
    )

def unconstrained(msg: str) -> Ticket:
    return Ticket.model_validate_json(model(prompt_for(msg), max_new_tokens=200))

def score(fn) -> str:
    good = 0
    for msg in MESSAGES:
        try:
            fn(msg)
            good += 1
        except (ValidationError, ValueError):
            pass
    return f"{good}/{len(MESSAGES)}"

print("constrained:", score(constrained))
print("unconstrained:", score(unconstrained))

On a 1.7B model, the unconstrained path usually loses a few of these to preamble text, markdown fences, or a severity value like "urgent". The constrained path should return the full count. Run it on your own messages before you believe either number.

That gap is your retry budget. Every malformed response is a retry, and in an agent loop it is worse than a retry: a tool call that receives a string where it expected a dict often fails in a confusing place, several steps away from the cause.

Step 4: constrain things that are not JSON

Not every output is an object. Order IDs and classification labels are two cases where a regex or a fixed choice list beats a full schema.

from outlines.types import Choice, Regex

order_id = model(
    "Extract the order id. Messages use the format ORD- followed by six digits.
"
    "Text: charged twice for ORD-88213",
    Regex(r"ORD-[0-9]{6}"),
)
print(order_id)  # ORD-88213

label = model(
    "Classify the topic: my card was charged twice",
    Choice(["billing", "auth", "outage", "other"]),
)
print(label)  # billing

Choice is the type to use when the list comes from your database at runtime, since Literal needs the values at import time.

Step 5: keep the failure handling

Constrained decoding removes format errors. It does not remove truncation. If max_new_tokens runs out halfway through the object, the sampler stops mid-structure and you get a string that ends with {"category": "billing", "sever. No exception is raised, because nothing went wrong from the decoder's point of view.

So keep the try/except. Treat ValidationError as a real path with a fallback, not as something that cannot happen. Two habits that help:

  • Set max_new_tokens from the schema size, not from a guess. A four field object with short strings does not need 1000 tokens.
  • Log the raw string on failure. If you only log the exception, you lose the evidence.

There is a second failure mode worth naming: a schema-valid answer that is still wrong. If the message is about a duplicate charge and the model labels it auth, the JSON is perfect and the routing is broken. Constrained decoding buys you structure, not judgment. Evaluate the field values separately, the same way you would evaluate any classifier.

Step 6: the same constraint at serving time

Running transformers inside your API process works for low traffic. Once you have concurrent requests, the usual move is vLLM, which supports the same idea through guided decoding. Since vLLM 0.12 the old guided_json, guided_choice, and guided_regex fields are gone, so code written against 0.8 or 0.9 will get rejected. The current name is structured_outputs.

vllm serve HuggingFaceTB/SmolLM2-1.7B-Instruct

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")

resp = client.chat.completions.create(
    model="HuggingFaceTB/SmolLM2-1.7B-Instruct",
    messages=[{"role": "user", "content": MESSAGE}],
    extra_body={"structured_outputs": {"json": Ticket.model_json_schema()}},
)
print(resp.choices[0].message.content)

Offline batching uses the same parameters through StructuredOutputsParams:

from vllm import LLM, SamplingParams
from vllm.sampling_params import StructuredOutputsParams

llm = LLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct")
params = SamplingParams(
    structured_outputs=StructuredOutputsParams(json=Ticket.model_json_schema())
)
out = llm.generate(["Triage: my card was charged twice"], params)
print(out[0].outputs[0].text)

vLLM picks a backend for you (auto), or you can force xgrammar or guidance with --structured-outputs-config.backend. If you hit a schema that compiles slowly or a model whose tokenizer behaves oddly, switching the backend is the first thing to try.

When to use what

  • Outlines with transformers suits offline work, batch scripts, a laptop, and a single process. Easy to unit test, no server to run. This is what step 2 through 5 use.
  • vLLM structured outputs is for many concurrent requests, or when you already expose an OpenAI-compatible endpoint. It needs a GPU to be useful, and it is the right place for structural_tag if you also constrain tool call boundaries.
  • Hosted provider structured outputs means no infrastructure, and the schema is enforced on their side. You send data out, and the supported JSON Schema subset differs between providers, so test the schema you actually plan to ship.
  • llama.cpp GBNF grammars make sense when you already serve a GGUF model and want grammar level control, including non-JSON formats. More work to write than a Pydantic class.

Wrapping up

The point of constrained decoding is to move one class of bugs out of your retry logic and into the sampler. Your prompt stops being a contract and becomes a hint, which is all a prompt ever was.

Reasonable next steps: put Ticket behind a FastAPI endpoint, run the score function from step 3 in CI against a fixed set of messages, and only then look at whether the labels are right. Structure first, accuracy second. Mixing the two makes both harder to debug.

References

Need Help Implementing This?

I help teams design and build scalable cloud infrastructure, DevOps pipelines, and production-grade systems.

Book a Free Consultation