vLLM tutorial: serve your first model with Docker

What you will build

Your application sends a chat request to a local HTTP server. vLLM loads the model, prepares its inputs, schedules generation on the GPU, and returns a response in an OpenAI-compatible format. The model supplies the learned parameters; vLLM supplies the serving machinery. A successful request checks that the whole path works.

We use Qwen/Qwen3-0.6B, a small, publicly accessible text model, and vLLM v0.30.0. Both the container digest and the model revision are pinned below. The small model keeps this first exercise manageable; it is not a recommendation about the quality your application requires.

Start with a compatible Linux GPU host

Use a Linux machine with Docker Engine, Bash, Python 3, curl, and the NVIDIA Container Toolkit configured for Docker. You need permission to run Docker and network access to Docker Hub and Hugging Face. Run every shell block in the same terminal so the environment variables remain available.

The pinned image uses CUDA 13.0. Use an NVIDIA R580 or newer driver and a supported GPU; vLLM documents compute capability 7.5 or higher for this release. Check the GPU requirements and NVIDIA driver compatibility table before renting a machine. This CUDA container is not a Mac GPU setup.

For a comfortable first attempt, use a GPU with at least 8 GiB of VRAM available and ample disk space for the container and download cache. That is a practical starting budget, not a tested minimum. Weight size alone understates runtime memory: activations, cached attention state, and execution buffers also need space. Avoid sharing the GPU with another inference job during this exercise.

nvidia-smi
docker version
python3 --version
df -h

Pin the artifacts and start the server

The tag identifies the release; the digest selects the exact published container manifest. The model and tokenizer share the same Hugging Face commit. The dedicated Docker volume keeps downloaded files across container restarts, without mounting your normal Hugging Face credentials.

VLLM_IMAGE='vllm/vllm-openai:v0.30.0@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90'
MODEL_REVISION='c1899de289a04d12100db370d81485cdf75e47ca'
export VLLM_API_KEY="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"

docker pull "$VLLM_IMAGE"
docker run --rm --runtime nvidia --gpus device=0 \
  --entrypoint nvidia-smi "$VLLM_IMAGE"
docker volume create inference-tutorial-hf

The GPU check should display device information from inside the container. If it fails, fix the driver or container runtime before loading a model. We select GPU 0 explicitly; change that selection if your machine assigns another GPU to this exercise.

docker run -d --name inference-tutorial \
  --runtime nvidia --gpus device=0 \
  --shm-size 2g \
  -p 127.0.0.1:8000:8000 \
  -v inference-tutorial-hf:/root/.cache/huggingface \
  --env VLLM_API_KEY \
  "$VLLM_IMAGE" \
  --model Qwen/Qwen3-0.6B \
  --revision "$MODEL_REVISION" \
  --tokenizer-revision "$MODEL_REVISION" \
  --served-model-name tutorial-model \
  --host 0.0.0.0 --port 8000 \
  --dtype float16 \
  --max-model-len 4096 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.80 \
  --generation-config vllm

docker logs -f inference-tutorial

Wait for the server to finish loading and report that it is accepting requests. The first run includes downloading weights and preparing execution, so startup is separate from request latency. Press Ctrl+C to stop following the logs; the detached container keeps running.

The server listens on all interfaces inside its container, while Docker publishes its port only on the host loopback address. Keep that mapping for this exercise. The generated key is passed through vLLM’s documented VLLM_API_KEY environment variable. For an externally reachable service, put authentication, TLS, access controls, and request limits at a gateway.

This follows the official Docker deployment pattern, using a bounded shared-memory allocation and a local port binding. No Hugging Face token or remote model code execution is required for this public model.

Understand the limits before changing them

These options are defined in the versioned engine arguments. A 4,096-token context includes chat formatting tokens: a 3,900-token formatted prompt cannot request 256 additional tokens. Increasing context length or concurrency changes memory pressure, so change one setting at a time and retain the startup log.

Check readiness and send a chat request

curl --fail --silent --show-error \
  http://127.0.0.1:8000/health

curl --fail --silent --show-error \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  http://127.0.0.1:8000/v1/models

A healthy response should have HTTP status 200; the health handler returns an empty body. The model list should contain tutorial-model, which is the API alias configured above. A connection failure means the server is not ready or the port mapping is wrong. An authorization failure usually means you changed terminals or regenerated the key after starting the container.

The following client uses only Python’s standard library. Qwen3 supports a thinking mode; this request disables it through chat_template_kwargs, following the model card and vLLM’s chat API documentation. The request includes explicit sampling settings and an output cap.

python3 - <<'PY'
import json
import os
import time
import urllib.error
import urllib.request

payload = {
    "model": "tutorial-model",
    "messages": [{
        "role": "user",
        "content": "Explain a KV cache in three short sentences."
    }],
    "max_tokens": 256,
    "temperature": 0.7,
    "top_p": 0.8,
    "top_k": 20,
    "min_p": 0,
    "seed": 42,
    "stream": False,
    "chat_template_kwargs": {"enable_thinking": False}
}
request = urllib.request.Request(
    "http://127.0.0.1:8000/v1/chat/completions",
    data=json.dumps(payload).encode("utf-8"),
    headers={
        "Content-Type": "application/json",
        "Authorization": "Bearer " + os.environ["VLLM_API_KEY"]
    },
    method="POST"
)
start = time.perf_counter()
try:
    with urllib.request.urlopen(request, timeout=120) as response:
        result = json.load(response)
except urllib.error.HTTPError as exc:
    raise SystemExit(f"HTTP {exc.code}: {exc.read().decode('utf-8')}")
except (urllib.error.URLError, TimeoutError) as exc:
    raise SystemExit(f"Request failed: {exc}")
elapsed = time.perf_counter() - start
choice = result["choices"][0]
print(choice["message"]["content"])
print("Finish reason:", choice["finish_reason"])
print("Token usage:", result.get("usage"))
print(f"End-to-end seconds: {elapsed:.3f}")
PY

Inspect the answer, the token counts, and finish_reason. A value of length means generation hit a limit; it does not prove the answer finished. Exact wording can differ across executions and hardware even with a seed. Review the small model’s explanation for correctness rather than treating a fluent answer as a successful quality test.

This timer covers the HTTP request through receipt of the full response, including any queueing. It does not measure time to first token or isolate GPU decoding. One request is a functional smoke test, not a throughput benchmark. For performance work, save a representative prompt set, warm the server, vary concurrency, and report response lengths, failures, and latency distributions alongside hardware and versions.

Troubleshoot the first failing stage

• Container cannot see the GPU: repeat the container nvidia-smi check. Inspect the NVIDIA runtime configuration and host driver before changing model settings.
• Image or architecture error: run uname -m and inspect the image manifest. The pinned manifest includes AMD64 and ARM64 variants; the host still needs a supported NVIDIA GPU and driver. CPU emulation does not supply CUDA hardware.
• Out of memory: inspect docker logs inference-tutorial and nvidia-smi. Stop competing GPU jobs, reduce --max-num-seqs to 1, or lower the context limit. Increasing the memory fraction only helps when that memory is actually free.
• Chat-template error: check that the model and tokenizer use the same pinned revision. This Qwen checkpoint includes a template; replacing it with an arbitrary base model may require a different request format.
• Download or startup failure: inspect the first error in the logs, available disk space, and access to Hugging Face. Repeatedly restarting a large download without finding the cause makes diagnosis harder.

If memory fails during CUDA graph preparation, --enforce-eager is another diagnostic option. It changes execution and may affect speed, so record it rather than silently comparing that run with a default run. See vLLM’s memory conservation guide. To change startup flags, save the logs, remove the existing container, and rerun the launch command.

Save the setup and release the GPU

Keep the launch command, image digest, model revision, GPU name, driver version, and representative requests with your results. Save logs before removing the container. Avoid saving a full container inspection as a shareable artifact: its environment contains your API key.

docker logs inference-tutorial > inference-tutorial.log 2>&1
docker image inspect "$VLLM_IMAGE" --format '{{json .RepoDigests}}'
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv

docker stop inference-tutorial
docker rm inference-tutorial
unset VLLM_API_KEY

The dedicated model-cache volume remains for the next run. If you no longer need this tutorial’s downloads, remove that volume with docker volume rm inference-tutorial-hf. Stopping the container releases its GPU resources; it does not stop billing for a rented GPU machine. Shut down the rented instance separately when your work is complete.