← Back to Blog
Sep 2, 2026 · 7 min read

Stream LLM Responses Token by Token with Server-Sent Events

LLMPythonFastAPISSEAI AgentsTutorial

You send a prompt. Then you stare at a spinner for eight seconds. The model is already generating, but the user sees nothing until the whole answer is done. That is the difference between a chat that feels alive and one that feels like a fax machine.

LLM answers are slow by web standards. A long reply can take many seconds to finish, and a user will not wait for all of it before giving up. Streaming fixes this by pushing tokens to the client the moment each one is produced. The first token lands fast, the rest trickle in, and the reader follows along instead of waiting.

Server-Sent Events (SSE) is the simplest tool for that job. This post walks through a complete working setup: a FastAPI backend that streams from an OpenAI-compatible API, plus a browser client that renders tokens as they arrive. Copy the code, tweak the model name, and you are done.

Prerequisites

  • Python 3.10+
  • pip install "fastapi[standard]" (0.135.0+, which bundles native SSE), openai, uvicorn
  • A key for any OpenAI-compatible endpoint. A local Ollama server works too, because it speaks the same stream=True protocol

How SSE works

SSE is a plain HTTP response with the media type text/event-stream. The server keeps the connection open and pushes small text blocks, each with fields like data and event, separated by blank lines.

data: {"delta": "Hello"}
event: token

data: {"delta": " world"}
event: token

data: [DONE]
event: done

FastAPI added native SSE in version 0.135.0, through EventSourceResponse and ServerSentEvent imported from fastapi.sse. Before that you had to reach for the third-party sse-starlette package. Browsers have supported this format for years, and the native EventSource API reads it out of the box.

Backend: a streaming chat endpoint

This is the whole server. The two changes that matter are response_class=EventSourceResponse and an async generator that yields each chunk.

import json
import os
from collections.abc import AsyncIterable

from fastapi import FastAPI, Request
from fastapi.sse import EventSourceResponse, ServerSentEvent
from openai import AsyncOpenAI
from pydantic import BaseModel

app = FastAPI()
client = AsyncOpenAI(api_key=os.environ.get("OPENAI_API_KEY"))


class ChatRequest(BaseModel):
    message: str
    history: list = []


async def tokens(req: ChatRequest, request: Request) -> AsyncIterable[ServerSentEvent]:
    messages = req.history + [{"role": "user", "content": req.message}]
    stream = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        stream=True,
    )
    async for chunk in stream:
        if await request.is_disconnected():
            await stream.aclose()
            return
        delta = chunk.choices[0].delta.content if chunk.choices else None
        if delta:
            yield ServerSentEvent(data=json.dumps({"delta": delta}), event="token")
        if chunk.choices and chunk.choices[0].finish_reason == "stop":
            yield ServerSentEvent(raw_data="[DONE]", event="done")
            return


@app.post("/chat/stream", response_class=EventSourceResponse)
async def chat_stream(req: ChatRequest, request: Request):
    return EventSourceResponse(
        tokens(req, request),
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no",
        },
    )

Two small details carry the weight here.

  • request.is_disconnected() ends the loop early if the user closes the tab, so you stop spending tokens on an audience that left.
  • chunk.choices[0].delta.content is where each token arrives. When finish_reason hits "stop", you send a [DONE] marker and close.

Start it and test with curl:

uvicorn main:app --host 0.0.0.0 --port 8000

curl -N -X POST http://localhost:8000/chat/stream \
  -H "Content-Type: application/json"   -d '{"message":"Explain SSE in two sentences."}'

The -N flag stops curl from buffering, so words appear in the terminal as they arrive one by one.

Client: render tokens with fetch

You might expect EventSource here. It has two limits that make it wrong for a chat endpoint: it only sends GET and it cannot set custom headers. For a POST call that also carries auth, fetch against a ReadableStream is the production-safe path. You parse the SSE lines yourself.

const controller = new AbortController();

const res = await fetch("/chat/stream", {
  method: "POST",
  signal: controller.signal,
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ message: input, history: history }),
});

const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
let assistantText = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  const lines = buffer.split("\n");
  buffer = lines.pop();
  for (const line of lines) {
    if (!line.startsWith("data:")) continue;
    const payload = line.slice(5).trim();
    if (payload === "[DONE]") break;
    assistantText += JSON.parse(payload).delta;
    render(assistantText);
  }
}

// from a stop button handler: controller.abort();

The loop reads the body in chunks, splits on newlines, and keeps the last partial line in buffer until more data arrives. Because you hold the ReadableStream, you get cancellation for free: stop the generation mid-stream by calling controller.abort().

When to pick SSE over WebSockets

SSE runs over plain HTTP, moves data one way (server to client), and reconnects automatically if the connection drops. WebSockets are bidirectional. Choose them only when the client must push mid-stream, like a chat where the user wants to interrupt and steer the model. If all you need is "return the tokens, nothing else," SSE is less machinery.

One honest caveat, straight from the MDN docs: over HTTP/1.1 a browser allows only about six open EventSource connections per origin across all tabs. HTTP/2 negotiates a far higher limit. The fetch-based reader above sidesteps that cap anyway, because it is not an EventSource object at all.

Production notes

  • Reverse proxies buffer by default. Behind Nginx, Cloudflare, or an ALB, intermediate proxies may hold the whole body before forwarding. The X-Accel-Buffering: no header in the code tells them to pass chunks through immediately. Without it, streaming silently turns back into a long wait.
  • Watch proxy timeouts. Several proxies default to about 60s of silence. If your model can pause between tokens, send a small keepalive comment periodically or raise the proxy read timeout.
  • Keep the headers Cache-Control: no-cache and Connection: keep-alive so nothing caches a stream that should never be cached.
  • Track usage if you bill tokens. Add stream_options={"include_usage": true} to the create call; the token totals come back in the final chunk.

Go further

Once tokens stream, layer these on: a typed tool_call and tool_result event so multi-step agents render progressively instead of freezing during a function call, a client-side abort button so users can stop runaway generations, and a time-to-first-token counter so you can prove the change actually helped.

Referensi

Need Help Implementing This?

I help teams design and build scalable cloud infrastructure, DevOps pipelines, and production-grade systems.

Book a Free Consultation