Run Your Own LLM API with vLLM: Server, Streaming, Tool Calls, and Real Metrics

A single 8B model fits on one 24GB GPU. Ten concurrent users do not fit on a naive Python wrapper around model.generate(), because that wrapper runs one request per forward pass and leaves most of the GPU idle while it waits for the last token.

vLLM replaces that loop with a scheduler. New requests join the running batch between decode steps, the KV cache is paged instead of preallocated per request, and the HTTP layer implements the OpenAI API. Client code that talks to a hosted model keeps working after you change base_url.

What follows is the setup I would run on a fresh box: the install, the launch flags that actually change behavior, a streaming client, tool calling, and the metrics that tell you whether the box is keeping up.

Prerequisites

• Linux host with an NVIDIA GPU. The example targets a 24GB card (RTX 4090, L4, A10G). Check with nvidia-smi.
• A driver new enough for CUDA 12. The wheel ships its own CUDA runtime, so you do not need the full toolkit.
• Python 3.11 to 3.14. PyPI metadata for vLLM 0.32.0 lists >=3.11,<3.15. Pick 3.12 unless you have a reason.
• uv, plus around 25GB of disk for weights and the compile cache.
• No Hugging Face token for Qwen/Qwen3-8B. Gated checkpoints such as Llama need HF_TOKEN exported.

Step 1: install vLLM

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv ~/vllm --python 3.12
source ~/vllm/bin/activate
uv pip install vllm
vllm --version

Version 0.32.0 was the current release when this was written (18 October 2026). Do not install into the system interpreter. vLLM pins specific torch and CUDA wheels and will fight with whatever else lives there.

Optional, and worth it:

vllm download-kernels

That pulls precompiled kernels so the first startup spends less time compiling.

Step 2: start the server

export VLLM_API_KEY="$(openssl rand -hex 24)"

vllm serve Qwen/Qwen3-8B \
  --served-model-name local-8b \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --api-key "$VLLM_API_KEY" \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

What each flag does:

--served-model-name sets the alias that shows up in /v1/models. Without it, every client has to pass the full Hugging Face repo path and breaks the day you swap checkpoints.

--max-model-len 8192 caps context at 8K tokens. Qwen3-8B supports 40,960, but vLLM sizes the KV cache pool from this number. A larger window means fewer concurrent requests on the same card. 8K is a sane starting point for chat and RAG answers.

--gpu-memory-utilization 0.90 is the fraction of VRAM vLLM may claim for weights, activations, CUDA graphs, and KV cache. Lower it if the card is shared with something else.

--api-key only authenticates requests under /v1, /v2, and /inference. Other routes on the same port stay open, including /invocations, which exposes the same inference capability. Bind to 127.0.0.1 or put a reverse proxy in front instead of exposing port 8000 to the internet.

--enable-auto-tool-choice and --tool-call-parser hermes let the model emit function calls. Qwen chat templates use the Hermes tool format, and this is the same pair vLLM uses in its own Qwen3 example. If your clients never set strict on their tools, add --tool-strict-level parameter to enforce every argument schema at the server.

Two lines in the startup log decide your capacity. Read them before tuning anything:

GPU KV cache size: N tokens
Maximum concurrency for 8,192 tokens per request: Mx

If M is below the concurrency you expect, no amount of client retry logic will fix it. You need a smaller --max-model-len, a quantized checkpoint, or another GPU.

Step 3: verify and stream a completion

curl -s http://127.0.0.1:8000/v1/models \
  -H "Authorization: Bearer $VLLM_API_KEY" | jq '.data[].id'

import os
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key=os.environ["VLLM_API_KEY"],
)

stream = client.chat.completions.create(
    model="local-8b",
    messages=[
        {"role": "system", "content": "Answer in two short paragraphs."},
        {"role": "user", "content": "Explain continuous batching."},
    ],
    temperature=0.3,
    max_tokens=400,
    stream=True,
)

for chunk in stream:
    text = chunk.choices[0].delta.content
    if text:
        print(text, end="", flush=True)

pip install openai first. Any OpenAI-compatible client works here, including the JavaScript SDK, LangChain, and the Anthropic-style clients that vLLM also exposes. Streaming matters for perceived latency: time to first token drops to the prefill cost plus one decode step.

Step 4: wire tool calling

tools = [{
    "type": "function",
    "function": {
        "name": "get_order_status",
        "description": "Look up an order by its ID.",
        "strict": True,
        "parameters": {
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
            "additionalProperties": False,
        },
    },
}]

messages = [{"role": "user", "content": "Where is order A-1042?"}]
resp = client.chat.completions.create(
    model="local-8b", messages=messages, tools=tools, tool_choice="auto"
)
msg = resp.choices[0].message

if msg.tool_calls:
    call = msg.tool_calls[0]
    messages.append(msg)
    messages.append({
        "role": "tool",
        "tool_call_id": call.id,
        "content": '{"status": "in transit", "eta": "2 days"}',
    })
    final = client.chat.completions.create(
        model="local-8b", messages=messages, tools=tools
    )
    print(final.choices[0].message.content)

strict: True constrains the argument schema during decoding, not just the call envelope. It only works cleanly if the schema follows the OpenAI strict style: additionalProperties: false on every object, every field listed in required, and optional fields written as ["string", "null"]. A tool without strict keeps unconstrained arguments, even when another tool in the same request opts in.

Most agent frameworks never set the field, which is why the server has --tool-strict-level. function constrains the envelope and the function name for every request; parameter also pins every argument schema, as if each tool were strict. Under auto, the default for most parsers, grammar only kicks in once at least one tool opts in.

Keep your own validation anyway. A grammar can guarantee valid JSON and still allow a valid value with the wrong business meaning, such as {"order_id": "1042"} when your system expects A-1042. That check belongs in the handler.

Step 5: make prefix caching pay off

Automatic prefix caching is on by default in vLLM V1. Disable it with --no-enable-prefix-caching if you want it off. One behavior worth knowing: a request that asks for prompt logprobs ignores the cache and recomputes the whole prompt to produce them, so keep that option out of hot paths.

The cache hashes prompt tokens block by block and only stores full blocks. A shared prefix has to match exactly, block for block, or the reuse stops at the first mismatch. Two habits follow from that:

1. Put everything static at the front: system prompt, tool schemas, retrieved document. Put the variable part, such as the user question or a timestamp, at the end.
2. Do not interpolate anything into the top of the system prompt. A single "Current time: 14:02:11" in the first block changes its hash, and every request becomes a cache miss.

On a multi-tenant box, pass cache_salt in the request body. The salt is mixed into the hash of the first block, so one tenant cannot reuse another tenant's cached tokens. That closes a timing side channel where latency reveals whether someone else already asked a similar question.

Step 6: read the metrics before you guess

curl -s http://127.0.0.1:8000/metrics | grep -E 'prefix_cache|num_requests|kv_cache_usage'

What to look at:

vllm:prefix_cache_hits against vllm:prefix_cache_queries gives the hit rate in tokens. On a RAG workload with a long shared system prompt, a rate under 0.2 means your prompt layout is defeating the cache.

vllm:num_requests_waiting climbing while vllm:num_requests_running stays flat means admission is limited by KV cache, not by compute. More compute will not help.

vllm:kv_cache_usage_perc near 1.0 together with a rising vllm:num_preemptions means requests are being evicted mid-generation and restarted later. That shows up as ugly tail latency.

vllm:time_to_first_token_seconds is the histogram users feel. A p99 of 8 seconds on a chat product is a product problem, not a metrics problem.

For Prometheus, the counter names usually carry the _total suffix. Confirm with the grep above before you paste a query, since the exported name differs between builds:

sum(rate(vllm:prefix_cache_hits_total[5m]))
/
sum(rate(vllm:prefix_cache_queries_total[5m]))

Load test it with the bundled benchmark, not with curl in a loop:

vllm bench serve \
  --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --model local-8b \
  --dataset-name random \
  --random-input-len 2000 \
  --random-output-len 200 \
  --num-prompts 200 \
  --max-concurrency 16 \
  --request-rate 8 \
  --header "Authorization=Bearer $VLLM_API_KEY" \
  --save-result

Run it once at --max-concurrency 1 for a baseline, then at the concurrency you expect in production. The report prints TTFT, time per output token, and inter-token latency at p99. If throughput barely moves between the two runs, batching is not helping and the KV pool is the bottleneck.

Step 7: do the KV cache arithmetic

Qwen3-8B has 36 layers, 8 KV heads, and head_dim 128. In bf16, each token costs:

2 (K and V) x 36 layers x 8 heads x 128 head_dim x 2 bytes = 147,456 bytes

That is 144 KiB per token. An 8K sequence needs 1.125 GiB of KV cache. A 32K sequence needs 4.5 GiB. Weights take about 16GB, so on a 24GB card at 0.90 utilization there is roughly 4 to 5GB left for KV after activations and CUDA graphs. Call it three or four full 8K sequences in flight.

Treat that as an estimate, because activation memory and graph capture vary with batch size and model. The startup log reports the real number.

Step 8: run it as a service

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env VLLM_API_KEY="$VLLM_API_KEY" \
  -p 127.0.0.1:8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-8B \
  --served-model-name local-8b \
  --max-model-len 8192 \
  --max-num-queued-reqs 64

The bind mount keeps the weights across restarts. --ipc=host is required once you add --tensor-parallel-size above 1. --max-num-queued-reqs 64 caps the in-flight queue: past that, the server returns HTTP 503 so your load balancer or client can retry on another instance instead of letting latency grow without bound.

Health check: curl -f http://127.0.0.1:8000/health. Restart policy on the container, or a systemd unit with Restart=always, and a startup probe with a generous initialDelaySeconds. Loading 16GB of weights and capturing CUDA graphs takes minutes, not seconds.

When to use vLLM vs alternatives

Ollama is the right call for a laptop, a demo, or a single developer iterating on prompts. It manages model downloads, runs GGUF quantizations, and needs one command. It can serve parallel requests, but it is tuned for one user at a time, not a queue of them.

llama.cpp's server is the one to pick when there is no NVIDIA GPU: CPU inference, Apple Silicon, small VRAM. Slower per token on a big model, but it runs where vLLM cannot.

vLLM earns its complexity when several users hit the same model, when you need an OpenAI-compatible endpoint for existing code, and when prefix caching or KV cache metrics change your bill. It also gives you tensor parallelism for models that do not fit on one card.

A hosted API still wins on total cost at low volume. One 24GB GPU rented by the month costs more than a few million tokens from a provider. The break-even point depends on your token mix, so measure before you migrate anything.

Troubleshooting

ValueError: The model's max seq len is larger than the maximum number of tokens that can be stored in KV cache means the configured window does not fit the available cache. Lower --max-model-len or free VRAM.

OOM while loading weights means the checkpoint does not fit. Check nvidia-smi for a leftover process, then move to an FP8 or AWQ checkpoint rather than shrinking the context.

Warmup failures right after vllm serve usually come from a model whose architecture needs a newer vLLM than the installed one. Read the traceback: it names the model class.

503 responses with a queue cap set are the feature working, not a bug. Scale out or raise the cap if you would rather queue than shed load.

Conclusion

The three commands worth memorizing are vllm serve with a context limit, curl /metrics, and vllm bench serve. Everything else is tuning around those.

Next steps that pay off: put the server behind Caddy or nginx with real auth, add a Prometheus scrape and an alert on num_preemptions, then measure prefix cache hit rate on your actual prompts. If the hit rate is low, restructure prompts before buying a bigger GPU. It is usually cheaper.

References

• vLLM OpenAI-compatible server: https://docs.vllm.ai/en/latest/serving/online_serving/openai_compatible_server/
• vLLM production metrics: https://docs.vllm.ai/en/latest/usage/metrics/
• vLLM tool calling: https://docs.vllm.ai/en/latest/features/tool_calling/
• vLLM parallelism and scaling, which documents the KV cache log lines: https://docs.vllm.ai/en/latest/serving/parallelism_scaling/