Fundamentals of Efficient LLM Inference with vLLM

Suppose a class of 30 pupils sends questions about the same passage to a reading assistant. The first pupil asks for a one-sentence answer. The second asks for an explanation with examples. All 30 requests use the same model, and every prompt begins with the same passage. What should the server compute once? What must it keep for each pupil? When the first answer finishes, who gets to use the GPU next?

Those questions explain much of how vLLM works. We’ll follow two pupils through the server, then run the experiment with Qwen, compress the model and measure the difference.

Identify what lives on the GPU

A model server has three kinds of information to manage during inference, the process of using a trained model to produce a prediction.

A token is a piece of text the model represents with an integer ID. The tokenizer converts the passage and question into IDs. The model converts those IDs into vectors and transforms them through successive layers. The GPU performs the calculations. vLLM decides which requests to run together, assigns space for their caches and returns the generated text to the application.

Picture one shared instruction manual and a notebook for each pupil’s question. The weights are the manual. The cache is the notebook. A new pupil can use the manual immediately, but their question still needs working space.

Follow the first question through the model

The application constructs a prompt from the passage followed by the question, “How do bees help a flowering plant produce seeds?” Suppose the formatted prompt contains exactly 1,000 tokens.

During prefill, the GPU processes those 1,000 known tokens. Within an attention layer, the model uses learned weights to produce a query, a key and a value for each position. A query is used to score relevant keys. The scores determine how much each corresponding value contributes. For the question about seeds, this gives the model a way to use information from the part of the passage about pollen transfer.

The model saves the keys and values at each attention layer. In a causal language model, later words cannot change what an earlier position could see when it was processed. The server can therefore use those saved values again. At the end of prefill, the last position supplies scores for the possible next tokens. The server selects the first answer token from those scores.

During decode, the selected token becomes the input for the next step. Its query consults the cached context, its own key and value are added, and the model supplies scores for the following token. The table shows why users experience an initial wait followed by a stream of text.

The old keys and values save the GPU from rebuilding the passage’s representations at every step. The GPU still has to read the relevant cached values. With full attention, a longer passage therefore makes each subsequent decode step more expensive. How KV caching works

Work out what will fit

Start with the weights. A model with 70 billion parameters stored at 16 bits per parameter needs approximately 140 GB for its raw weights. Eight-bit storage brings the theoretical figure to 70 GB. Four-bit storage brings it to 35 GB. These are decimal gigabytes. A real checkpoint also contains metadata, quantization scales and sometimes layers stored at higher precision.

Now estimate the notebook. For a conventional decoder with full attention and uniform attention layers, the approximate KV storage for one token across the whole model is given by the following expression.

2 × number of layers × KV heads per layer × head dimension × bytes per element

The first factor of two counts the key and the value. As a worked example, take 32 layers, eight KV heads, a head dimension of 128 and two bytes per element. That gives 128 KiB per token. A request retaining 4,096 tokens needs 512 MiB. Twenty independent requests of that length need about 10 GiB of cache before block overhead. These illustrative dimensions are separate from the small Qwen model used below.

There are two practical consequences. First, buying enough memory to load the weights can still leave too little room to serve the class. Second, halving the bytes used by the weights leaves more room for notebooks, while the bytes required by each notebook depend on the cache format and architecture.

Capacity also differs from bandwidth, the rate at which data can move. Weights and caches generally reside in GPU device memory. The GPU brings portions closer to the arithmetic units, using smaller, faster on-chip storage. During prefill, many known prompt positions can be processed together. During ordinary decode, each request contributes one new token at a time. At small batch sizes, reading the weights and cached state can take longer than performing the arithmetic. Batching lets the GPU use a layer’s weights for several requests in the same step. vLLM performance tuning

See what changes when the second pupil arrives

The second pupil sends the same passage followed by “What might happen if fewer bees visited these flowers?” Three parts of vLLM now work together.

PagedAttention determines where the notebook pages live. Suppose the first request’s first three logical cache blocks occupy physical blocks 7, 21 and 4 in GPU memory. A block table records their order. The attention operation uses that table to find the correct keys and values. When the sequence grows, vLLM can assign another available block. This avoids reserving a single continuous region large enough for every request’s maximum possible length. A partially filled last block can still contain unused slots. PagedAttention paper

Prefix caching determines which pages already contain the required work. The pupils’ prompts start with identical instructions and passage tokens. If matching cache blocks are still available, the second request can reuse them and compute the remaining suffix, including its own question. Each pupil still gets a separately generated answer. Putting a different pupil name or timestamp before the passage changes the token prefix at that point and can prevent reuse of the passage that follows. This makes prompt ordering a serving decision as well as a writing decision. Automatic prefix caching

Continuous batching determines who gets the next GPU step. While both pupils are generating, the scheduler can batch their next tokens together. When the first pupil’s one-sentence answer finishes, the request releases its active references to its cache blocks. The scheduler can admit a waiting pupil when the token and memory budgets allow, even while the second pupil’s longer answer continues. Eligible prefix blocks can remain available for reuse until reclaimed. A long incoming passage can also be processed in smaller chunks so existing answers can continue streaming between chunks. Inside the vLLM inference engine

Prepare a Qwen baseline and an FP8 checkpoint

For the hands-on example, use a Linux machine with Python 3.11, a compatible CUDA driver and an NVIDIA GPU that supports the FP8 inference path, such as an H100. The model and evaluation datasets require an initial download. Run the commands from one project directory. The compression, serving and load-testing environments have separate dependencies.

mkdir -p vllm-tutorial/models vllm-tutorial/results
cd vllm-tutorial

python3.11 -m venv .compress
.compress/bin/python -m pip install llmcompressor huggingface_hub
.compress/bin/hf download Qwen/Qwen2.5-1.5B-Instruct --local-dir models/qwen-original

Save the following as compress_qwen.py. The recipe quantizes the targeted linear layers’ weights using FP8 and configures dynamic activation quantization. At inference time, activation scales are determined from the current input. The recipe excludes the output projection, lm_head. This particular recipe can run without a calibration dataset. LLM Compressor FP8 example

from pathlib import Path

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from transformers import AutoModelForCausalLM, AutoTokenizer

original = Path("models/qwen-original")
compressed = Path("models/qwen-fp8")

tokenizer = AutoTokenizer.from_pretrained(original)
model = AutoModelForCausalLM.from_pretrained(original, dtype="auto")
recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head"],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained(compressed)
tokenizer.save_pretrained(compressed)

for directory in (original, compressed):
    size = sum(path.stat().st_size for path in directory.glob("*.safetensors"))
    print(f"{directory.name}: {size / 2**30:.2f} GiB of checkpoint tensors")

.compress/bin/python compress_qwen.py

Compare the two printed sizes. This measures checkpoint tensors on disk. On the running server, GPU memory also covers the KV pool, temporary buffers and runtime overhead. A smaller checkpoint can therefore give you a larger cache pool even when a GPU monitor reports similar total memory use.

Serve the model and observe a cache hit

Install vLLM and the evaluation harness in a fresh environment. Start with the original model so you can measure a baseline. The inference server stays in the foreground in terminal A. vLLM installation and quickstart

python3.11 -m venv .serve
.serve/bin/python -m pip install vllm 'lm_eval[vllm]'

.serve/bin/vllm serve ./models/qwen-original \
  --served-model-name Qwen/Qwen2.5-1.5B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 4096 --enable-prefix-caching

In terminal B, open the same project directory. Check that the model is available, then save the next Python block as ask_classroom.py.

curl -fsS http://127.0.0.1:8000/v1/models

import json
from urllib.request import Request, urlopen

base_url = "http://127.0.0.1:8000"
passage = (
    "A bee visits a flower to collect nectar. Pollen from the anthers sticks "
    "to its body. At another flower of the same species, some pollen can reach "
    "the stigma. This transfer is called pollination. After fertilisation, "
    "seeds can develop. The number of visits and the amount of compatible "
    "pollen transferred can affect how many seeds the plant produces."
)
questions = [
    "How can a bee help a plant produce seeds?",
    "What might happen if fewer bees visited these flowers?",
]

def show_cache_counters():
    with urlopen(f"{base_url}/metrics", timeout=10) as response:
        lines = response.read().decode().splitlines()
    for line in lines:
        if line.startswith(("vllm:prefix_cache_hits_total",
                            "vllm:prefix_cache_queries_total")):
            print(line)

show_cache_counters()
for question in questions:
    prompt = f"Use this passage to answer in one sentence.\n\n{passage}\n\n{question}"
    payload = {
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
        "max_tokens": 64,
    }
    request = Request(
        f"{base_url}/v1/chat/completions",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urlopen(request, timeout=120) as response:
        result = json.load(response)
    print(question, result["choices"][0]["message"]["content"], sep="\n")
    show_cache_counters()

python3 ask_classroom.py

On a freshly started, otherwise idle server, watch the hit counter before the first question and after the second. The second question has a different answer but shares enough initial tokens for full cache blocks to match on common block sizes. The counter measures reuse across the server, so inspect its change during this experiment. If your release exposes different metric names, search the raw endpoint with curl -fsS http://127.0.0.1:8000/metrics | grep prefix_cache. Metrics may update periodically. If the counter has yet to move, fetch it again after a reporting interval. A short prompt can produce a cache hit with little visible latency difference. A longer shared document makes the saved prefill work easier to measure. vLLM metrics

Benchmark both deployments under the same load

A single successful answer establishes that the API works. To learn whether a deployment can serve a class, send overlapping requests. In terminal B, install GuideLLM. These commands use version 0.5.3 and its benchmark CLI. GuideLLM 0.5.3

python3.11 -m venv .bench
.bench/bin/python -m pip install 'guidellm==0.5.3'

Define this Bash function in terminal B. It runs a fixed arrival rate for one minute and saves each run in its own directory.

bench_cli="$PWD/.bench/bin/guidellm"
run_load_test() {
  local variant="$1"
  local rate="$2"
  local run_dir="results/${variant}-${rate}rps"
  mkdir -p "$run_dir"
  (
    cd "$run_dir" || exit 1
    "$bench_cli" benchmark \
      --target http://127.0.0.1:8000 \
      --profile constant --rate "$rate" --max-seconds 60 \
      --data "prompt_tokens=256,output_tokens=128"
  )
}

run_load_test original 1
run_load_test original 4

Stop the baseline server with Ctrl+C in terminal A. Run the same serve command again, changing only ./models/qwen-original to ./models/qwen-fp8. Keeping the same served model name lets the same client and tokenizer settings work for both. Once the endpoint is ready, repeat the classroom requests as the same warm-up and rerun the two load levels in terminal B.

python3 ask_classroom.py
run_load_test fp8 1
run_load_test fp8 4

Read the metrics together. If moving from one to four requests per second increases total output while p95 TTFT rises sharply and ITL stays steady, inspect the waiting queue and prefill times first. If a longer shared passage improves TTFT on repeat questions while ITL stays similar, the pattern is consistent with prefix reuse. Record GPU type, package versions, checkpoint revision, prompt and output lengths, and warm-up procedure alongside the measurements. Repeat runs before treating small timing differences as meaningful.

Check what compression did to model quality

Stop the FP8 server before running the next commands. The evaluation harness loads its own model on the GPU. Run both checkpoints on the same examples. WikiText reports, among other measures, word perplexity. A lower value means the model assigned higher likelihood to the held-out text under that scoring procedure. HellaSwag checks whether the model ranks the expected continuation above alternatives.

for variant in original fp8
do
  .serve/bin/lm-eval run \
    --model vllm \
    --model_args "pretrained=./models/qwen-${variant},max_model_len=4096" \
    --tasks wikitext,hellaswag \
    --batch_size auto --limit 100 \
    --output_path "results/quality-${variant}"
done

The 100-example limit makes this an initial comparison. Inspect disagreements and repeat on a larger set for any deployment decision. For the classroom assistant, also score answers to real passage questions. For example, give credit for explaining pollen transfer between flowers and check whether the answer incorrectly equates pollination with fertilisation. A general benchmark can miss that distinction even when pupils depend on it. Evaluation harness · WikiText task definition

For the final comparison, choose an acceptable p95 wait and a minimum passage-question score. At each tested load, count the successful requests that meet the latency target. Divide the serving cost for the measured interval by that count, comparing only deployments that meet the quality threshold. If both models fit on the same rented GPU and complete the same number of acceptable requests, a smaller checkpoint alone has not reduced the bill. The saving appears when the compact model can serve additional acceptable work on that GPU or fit the workload onto a cheaper one.

This tutorial builds on the topics in the DeepLearning.AI and Red Hat short course taught by Cedric Clyburn. The classroom prompts, worked memory example and comparison procedure are provided here for readers to run themselves. Fast and Efficient LLM Inference with vLLM