Continuous batching explained: how LLM servers scale

Why LLM servers batch requests

During decode, a model produces one token per request per step, and each step reads every weight from GPU memory. Serving a single request leaves most of the GPU’s arithmetic idle. Running many requests in the same step reuses each weight read for all of them, so total tokens per second rise sharply while each step takes only a little longer. LLM inference explained covers the decode loop in more detail.

The hard part is that requests arrive at different times and produce very different numbers of tokens. One user asks for a one-word label, another for a long essay. How the server groups this changing mix decides both throughput and how long each user waits.

The problem with static batching

Static batching collects a fixed group of requests, runs them together until all are finished, then starts the next group. It is simple, but it wastes capacity in two ways:

• A request that finishes early leaves its slot empty until the longest request in its batch is done.
• New requests wait for the whole batch to finish before they can start, even when slots are free.

Because output lengths vary widely in real traffic, a batch is often held open for one long answer while most of its slots sit idle.

What is continuous batching?

Continuous batching, also called iteration-level scheduling or in-flight batching, makes a scheduling decision at every model step instead of once per batch. When a request finishes, it leaves the batch immediately and a waiting request takes its place on the next step. The batch is always refilling, so GPU slots stay busy and new requests start sooner.

The idea was introduced by the Orca serving system. Orca: A Distributed Serving System for Transformer-Based Generative Models describes iteration-level scheduling and selective batching. Today vLLM, SGLang, and other major inference engines use it by default.

Simulate static and continuous batching

This small simulation gives eight requests to a server with four slots. Each request needs a different number of decode steps. It counts steps rather than seconds and ignores prefill, so it isolates the scheduling effect.

# Output lengths (in decode steps) for eight requests that arrive together.
lengths = [12, 3, 5, 12, 2, 4, 3, 7]
slots = 4


def static_batching(lengths, slots):
    """Each batch runs until its longest request finishes."""
    step, finish = 0, []
    for start in range(0, len(lengths), slots):
        batch = lengths[start:start + slots]
        finish += [step + length for length in batch]
        step += max(batch)
    return step, finish


def continuous_batching(lengths, slots):
    """A waiting request takes a slot as soon as one frees up."""
    waiting = list(enumerate(lengths))
    running, finish, step = {}, [0] * len(lengths), 0
    while waiting or running:
        while waiting and len(running) < slots:
            index, length = waiting.pop(0)
            running[index] = length
        step += 1
        for index in list(running):
            running[index] -= 1
            if running[index] == 0:
                finish[index] = step
                del running[index]
    return step, finish


for name, schedule in (("static", static_batching), ("continuous", continuous_batching)):
    total, finish = schedule(lengths, slots)
    mean = sum(finish) / len(finish)
    print(f"{name:<11} last finish: step {total:>2}   mean finish: step {mean:.1f}")

static      last finish: step 19   mean finish: step 12.0
continuous  last finish: step 15   mean finish: step 8.6

With static batching, the second group cannot start until step 12, because two requests in the first group need twelve steps. With continuous batching, the short requests hand their slots to waiting ones as soon as they finish. The same work completes four steps sooner and the average request finishes about 28 percent earlier. These are simulated step counts, not measurements; the gap in real systems depends on how much output lengths vary.

Why continuous batching needs careful memory management

Every running request holds a KV cache that grows by one token per step. Since the server does not know in advance how long each answer will be, reserving the maximum length for every request wastes memory and limits the batch size. Early systems lost a large share of cache memory this way.

PagedAttention, the design behind vLLM, stores the cache in fixed-size blocks allocated on demand, much like pages in an operating system. Requests take memory only as they grow, so more of them fit in the batch. When memory still runs out, the scheduler preempts a request: it pauses it, frees its blocks, and later resumes it by recomputing or restoring its cache. Frequent preemption is a sign that the server admits more requests than its memory can hold.

Chunked prefill: mixing new prompts with ongoing answers

A new request must first run prefill over its whole prompt. A long prompt can take far longer than a decode step, and if the server runs it on its own, every user who is mid-answer sees a pause in their stream. Chunked prefill splits long prompts into pieces and runs each piece in the same step as the decode tokens of other requests, so streams keep moving. SARATHI and Sarathi-Serve describe the technique.

The chunk size creates a tradeoff. The vLLM tuning guide explains that a smaller token budget per step gives better inter-token latency, because fewer prefill tokens slow down decodes, while a larger budget gives a better time to first token. It also notes that chunked prefill is enabled by default where possible. vLLM: optimization and tuning.

Settings that control batching in vLLM and SGLang

Raising the running request limit increases throughput until memory or compute runs out, after which queueing and preemption make latency worse. Lowering the token budget smooths streaming for users already receiving answers but makes new requests wait longer for their first token. SGLang: server arguments lists its scheduler options, including scheduling policies.

Continuous batching FAQ

Is continuous batching the same as dynamic batching? Not quite. Dynamic batching usually means grouping requests that arrive within a short time window, then running the group to completion. Continuous batching reschedules at every step, so requests join and leave individually.

Does continuous batching make a single request faster? No. A lone request runs at the same speed. The benefit appears under load, where requests start sooner and the GPU does more useful work per step.

Do I need to turn it on? Usually not. Modern engines such as vLLM and SGLang schedule this way by default. What you tune are the limits in the table above.

How do I know if batching is working well? Measure latency and throughput together at realistic load. LLM inference metrics explains TTFT, TPOT, and goodput. To compare engines, see vLLM vs SGLang.