vLLM Free code examples in Python and JavaScript

Linux and an NVIDIA GPU is the supported path. --torch-backend=auto picks the CUDA build that matches the driver, which is the step that otherwise goes wrong.

uv pip install vllm --torch-backend=auto

# A served deployment should require a key. The server reads it from
# the environment and clients send it as a normal bearer token.
export VLLM_API_KEY="a-long-random-string"

vllm serve Qwen/Qwen2.5-1.5B-Instruct --api-key "$VLLM_API_KEY"

An OpenAI-compatible server, so the client is the OpenAI client. The model name is the full Hugging Face repo id — not a short alias.

import os
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key=os.environ["VLLM_API_KEY"],
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",     # the full repo id
    messages=[{"role": "user", "content": "Explain a database index."}],
)

print(response.choices[0].message.content)

THESE THREE ARE THE DEPLOYMENT. max-num-seqs is how many sequences may be in flight at once — the batching width. gpu-memory-utilization is how much of the card vLLM may claim for the KV cache, and a bigger cache is what allows a wider batch.

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --api-key "$VLLM_API_KEY" \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

# max-num-seqs        concurrent sequences. Higher = more throughput
#                     and higher per-request latency.
# gpu-memory-utilization  fraction of VRAM for weights + KV cache.
#                     0.90 is aggressive; leave headroom if anything
#                     else shares the card.
# max-model-len       context per sequence. THIS IS THE ONE PEOPLE
#                     OVERSPEND ON: the KV cache is proportional to
#                     it, so asking for 128k when you use 4k cuts
#                     the batch width by a factor of thirty.

Run this against one GPU. Sequential requests measure latency; concurrent ones measure throughput, and the gap between them IS continuous batching. If total time barely moves as concurrency rises, the GPU was idle before.

import asyncio
import time

from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="http://localhost:8000/v1",
    api_key=os.environ["VLLM_API_KEY"],
)

async def one(prompt: str) -> int:
    response = await client.chat.completions.create(
        model="Qwen/Qwen2.5-1.5B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=128,
    )
    return response.usage.completion_tokens

async def measure(concurrency: int) -> None:
    started = time.perf_counter()
    counts = await asyncio.gather(
        *(one(f"Explain concept number {i}.") for i in range(concurrency)))
    elapsed = time.perf_counter() - started

    print(f"{concurrency:>3} concurrent  "
          f"{sum(counts) / elapsed:7.0f} tokens/sec  {elapsed:5.1f}s")

for concurrency in (1, 8, 32, 128):
    asyncio.run(measure(concurrency))

The metrics endpoint is Prometheus format and carries the numbers that matter: how many sequences are running, how many are queued, and how full the KV cache is.

curl -s http://localhost:8000/metrics | grep -E 'num_requests|gpu_cache_usage'

# vllm:num_requests_running      in flight right now
# vllm:num_requests_waiting      QUEUED. Persistently above zero means
#                                max-num-seqs or the KV cache is the
#                                bottleneck, not the GPU's compute.
# vllm:gpu_cache_usage_perc      KV cache fullness. Pinned near 1.0
#                                with requests waiting means
#                                max-model-len is too generous.