Continuous Batching: Complete Technical Guide

Overview

Continuous Batching (also called Dynamic Batching) is a scheduling technique for LLM inference that dynamically adds and removes requests from the active batch as tokens are generated, maximizing GPU utilization and throughput.

• Introduced: Popularized by vLLM (2023)
• Key Innovation: Remove completed requests immediately, add new ones
• Impact: 2-4x throughput improvement over static batching
• Adopted by: vLLM, HuggingFace TGI, SGLang, and modern inference engines
• Critical for: Production LLM serving at scale

---

The Problem: Static (Static Window) Batching

Traditional Batching Approach

Batch Processing (Static):

Setup: Decide batch size = 4 requests
  - Request 1: Expected length 10 tokens
  - Request 2: Expected length 5 tokens
  - Request 3: Expected length 8 tokens
  - Request 4: Expected length 12 tokens

Execution Timeline:
- ┌────────────────────────────────────────┐
    - Iteration 1 (token 1)                  │
  - ┬──────────┬──────────┬───────┤
          - Req 1 ✓  │ Req 2 ✓  │ Req 3 ✓  │ Req 4 ✓ │
          - token 1  │ token 1  │ token 1  │ token 1 │
  - ┴──────────┴──────────┴───────┘

Iteration 2 (token 2)
  - ┬──────────┬──────────┬───────┤
          - Req 1 ✓  │ Req 2 ✓  │ Req 3 ✓  │ Req 4 ✓ │
          - token 2  │ token 2  │ token 2  │ token 2 │
  - ┴──────────┴──────────┴───────┘

Iteration 3 (token 3)
  - ┬──────────┬──────────┬───────┤
          - Req 1 ✓  │ Req 2 ✓  │ Req 3 ✓  │ Req 4 ✓ │
          - token 3  │ token 3  │ token 3  │ token 3 │
  - ┴──────────┴──────────┴───────┘

Iteration 4 (token 4)
  - ┬──────────┬──────────┬───────┤
          - Req 1 ✓  │ Req 2 ✗  │ Req 3 ✓  │ Req 4 ✓ │
          - token 4  │ DONE!    │ token 4  │ token 4 │
  - ┴──────────┴──────────┴───────┘

Iteration 5 (token 5)
  - ┬──────────┬──────────┬───────┤
          - Req 1 ✓  │ WAITING  │ Req 3 ✓  │ Req 4 ✓ │
          - token 5  │ (waste)  │ token 5  │ token 5 │
  - ┴──────────┴──────────┴───────┘

Iteration 8 (token 8)
  - ┬──────────┬──────────┬───────┤
          - Req 1 ✓  │ WAITING  │ Req 3 ✗  │ Req 4 ✓ │
          - token 8  │ (waste)  │ DONE!    │ token 8 │
  - ┴──────────┴──────────┴───────┘

Iteration 10 (token 10)
  - ┬──────────┬──────────┬───────┤
          - Req 1 ✗  │ WAITING  │ WAITING  │ Req 4 ✓ │
          - DONE!    │ (waste)  │ (waste)  │ token 10│
  - ┴──────────┴──────────┴───────┘

Iteration 12 (token 12)
  - ┬──────────┬──────────┬───────┤
          - WAITING  │ WAITING  │ WAITING  │ Req 4 ✗ │
          - (waste)  │ (waste)  │ (waste)  │ DONE!   │
  - ┴──────────┴──────────┴───────┘

Total iterations: 12
Active requests: 4, 4, 4, 4, 3, 3, 3, 2, 2, 1, 1, 1
Average batch utilization: (4+4+4+4+3+3+3+2+2+1+1+1)/12 = 32/12 = 2.67 (67% efficiency)

WASTE: Slots are occupied by completed requests!

Why This Is Inefficient

Problems with static batching:
1. Batch size locked in at start
2. Completed requests hold batch slots
3. New requests must wait for entire batch to finish
4. Uneven request lengths waste GPU time
5. GPU sits idle while waiting for slowest request

Real-World Impact: Throughput Loss

Scenario: 100 requests arrive in system
Average request length: 100 tokens
Batch size: 4 requests

Static Batching Timeline:
  - Batch 1 (Requests 1-4): Takes 200 iterations (max length = 200)
  - Batch 2 (Requests 5-8): Takes 150 iterations
  - Batch 3 (Requests 9-12): Takes 180 iterations
  - ...
  - Batch 25 (Requests 97-100): Takes 120 iterations
  - Total iterations: ~4500

If each iteration = 1ms: 4.5 seconds total
Throughput: 100 requests / 4.5s = 22 requests/second ❌

(Note: This is very slow!)

---

The Solution: Continuous Batching

Core Concept

Instead of a fixed batch for entire generation, dynamically manage the batch:

Continuous Batching:

Time 0: [Request 1, Request 2, Request 3, Request 4]
        Generate token 1 for all

Time 1: [Request 1, Request 2, Request 3, Request 4]
        Generate token 2 for all

Time 2: Request 2 completes (5 tokens)
        ADD Request 5 (new arrival)
        [Request 1, Request 3, Request 4, Request 5]
        Generate token 3 for all

Time 3: [Request 1, Request 3, Request 4, Request 5]
        Generate token 4 for all

Time 4: Request 2 is done (free slot)
        Request 3 completes (8 tokens)
        ADD Request 6 (new arrival)
        [Request 1, Request 4, Request 5, Request 6]
        Generate token 5 for all

Time 5: Request 4 completes (12 tokens)
        ADD Request 7, Request 8 (new arrivals)
        [Request 1, Request 5, Request 6, Request 7, Request 8]

...continue until all requests done...

KEY: New requests added as soon as slots free!
NO WASTED SLOTS!

Visual Comparison

STATIC BATCHING (Fixed slots per batch):
- ┌─────┬─────┬─────┬─────┐
          - R1  │ R2  │ R3  │ R4  │  Batch 1
          - ✓✓✓ │ ✓✓✓ │ ✓✓✓ │ ✓✓✓ │  (4 slots)
          - ✓✓✓ │ ✓✓✓ │ ✓✓✓ │ ✓✓✓ │
          - ✓✓✓ │ ✗   │ ✓✓✓ │ ✓✓✓ │  (3 slots, 1 wasted)
          - ✓✓  │ ✗   │ ✓✓✓ │ ✓✓✓ │  (3 slots, 1 wasted)
          - ✗   │ ✗   │ ✗   │ ✓✓✓ │  (1 slot, 3 wasted)
  - ┴─────┴─────┴─────┘
Total efficiency: 40/50 = 80%

CONTINUOUS BATCHING (Dynamic slots):
- ┌─────┬─────┬─────┬─────┐
          - R1  │ R2  │ R3  │ R4  │  Token 1
  - ┼─────┼─────┼─────┤
          - R1  │ R2  │ R3  │ R4  │  Token 2
  - ┼─────┼─────┼─────┤
          - R1  │ R5  │ R3  │ R4  │  Token 3 (R2 removed, R5 added)
  - ┼─────┼─────┼─────┤
          - R1  │ R5  │ R3  │ R4  │  Token 4
  - ┼─────┼─────┼─────┤
          - R1  │ R5  │ R6  │ R4  │  Token 5 (R3 removed, R6 added)
  - ┼─────┼─────┼─────┤
          - R1  │ R5  │ R6  │ R7  │  Token 6 (R4 removed, R7 added)
  - ┴─────┴─────┴─────┘
Total efficiency: 28/28 = 100%

---

How Continuous Batching Works

The Scheduler: Core Component

class ContinuousBatchScheduler:
    """Manages dynamic batch scheduling"""

    def __init__(self, max_batch_size: int = 32):
        self.max_batch_size = max_batch_size
        self.running_requests = []  # Active requests
        self.waiting_queue = []      # Requests waiting to start

    def schedule(self):
        """
        Main scheduling loop - runs once per token generation
        """
        # Step 1: Remove completed requests
        self.running_requests = [
            req for req in self.running_requests 
            if not req.is_completed()
        ]

        # Step 2: Add new requests if space available
        while (len(self.running_requests) < self.max_batch_size and 
               self.waiting_queue):
            new_req = self.waiting_queue.pop(0)
            self.running_requests.append(new_req)
            print(f"Added {new_req.id} to batch (batch size: {len(self.running_requests)})")

        # Step 3: Generate one token for all running requests
        batch_ids = [req.id for req in self.running_requests]
        outputs = self.model.forward(self.running_requests)

        # Step 4: Update request states
        for req, output in zip(self.running_requests, outputs):
            req.tokens.append(output)

        return len(self.running_requests)  # Current batch size

    def add_request(self, request):
        """New request arrives"""
        self.waiting_queue.append(request)
        print(f"Queued {request.id}")

Timeline: Step-by-Step Execution

Setup:
Max batch size: 4
Requests arrive over time:
- t=0: Req1, Req2, Req3, Req4 (all at start)
- t=0.1s: Req5 arrives
- t=0.2s: Req6 arrives
- Req1 length: 10 tokens
- Req2 length: 5 tokens
- Req3 length: 8 tokens
- Req4 length: 12 tokens
- Req5 length: 6 tokens
- Req6 length: 7 tokens

Execution:

Token 1 (t=0 + inference_time):
  - Running: [Req1, Req2, Req3, Req4]
  - Batch size: 4/4
  - Waiting: []
  - Action: Forward pass for all 4

Token 2 (t + inference_time):
  - Running: [Req1, Req2, Req3, Req4]
  - Batch size: 4/4
  - Waiting: []
  - Action: Forward pass for all 4

Token 3 (t + inference_time):
  - Running: [Req1, Req2, Req3, Req4]
  - Batch size: 4/4
  - Waiting: [Req5]
  - Action: Forward pass for all 4
  - Note: Req5 arrived but batch full

Token 4 (t + inference_time):
  - Running: [Req1, Req2, Req3, Req4]
  - Batch size: 4/4
  - Waiting: [Req5, Req6]
  - Action: Forward pass for all 4

Token 5 (t + inference_time):
  - Remove: Req2 completed (5 tokens done)
  - Running: [Req1, Req3, Req4] → Add Req5
  - Running: [Req1, Req3, Req4, Req5]
  - Batch size: 4/4
  - Waiting: [Req6]
  - Action: Forward pass for all 4 (new composition!)

Token 6 (t + inference_time):
  - Running: [Req1, Req3, Req4, Req5]
  - Batch size: 4/4
  - Waiting: [Req6]
  - Action: Forward pass for all 4

Token 7 (t + inference_time):
  - Remove: Req3 completed (8 tokens done)
  - Running: [Req1, Req4, Req5] → Add Req6
  - Running: [Req1, Req4, Req5, Req6]
  - Batch size: 4/4
  - Waiting: []
  - Action: Forward pass for all 4

Token 8 (t + inference_time):
  - Running: [Req1, Req4, Req5, Req6]
  - Batch size: 4/4
  - Waiting: []
  - Action: Forward pass for all 4

Token 10 (t + inference_time):
  - Remove: Req4 completed (12 tokens done)
  - Running: [Req1, Req5, Req6]
  - Batch size: 3/4
  - Waiting: []
  - Action: Forward pass for all 3

Token 11 (t + inference_time):
  - Remove: Req5 completed (6 tokens done)
  - Running: [Req1, Req6]
  - Batch size: 2/4
  - Waiting: []
  - Action: Forward pass for all 2

Token 12 (t + inference_time):
  - Remove: Req6 completed (7 tokens done)
  - Running: [Req1]
  - Batch size: 1/4
  - Waiting: []
  - Action: Forward pass for 1

Token 13 (t + inference_time):
  - Remove: Req1 completed (10 tokens done)
  - Running: []
  - Batch size: 0/4
  - Waiting: []
  - Status: ALL DONE!

Key observations:
- Batch size varies: 4, 4, 4, 4, 4, 4, 4, 4, 3, 2, 1
- New requests added immediately when slot frees
- No wasted iterations with empty slots
- Total tokens processed: 10+5+8+12+6+7 = 48
- With 11 iterations: 48 total generations
- Average batch size: 48/11 ≈ 4.36
  (Better than static: max would be constrained)

---

Performance Impact Analysis

Throughput Improvement

Scenario: 100 requests
Average request length: 100 tokens
Batch size: 32 requests

STATIC BATCHING:
  - Process in batches of 32
  - Batch 1 (Req 1-32): Max length = ~200 tokens (assume varied)
  - Batch 2 (Req 33-64): ~200 tokens
  - Batch 3 (Req 65-96): ~200 tokens
  - Batch 4 (Req 97-100): Only 4 requests, 200 iterations (waste!)
  - Plus idle time during slow requests
  - Total: ~250-300 iterations
  - Throughput: 100 requests / 300 iterations ≈ 0.33 req/iter

CONTINUOUS BATCHING:
  - Maintain 32-request batch throughout
  - New requests added as slots free
  - Batch always full (except at end)
  - Total tokens: 100 × 100 = 10,000
  - Iterations: 10,000 / 32 ≈ 312 iterations
  - But: More efficient iteration (batch always full)
  - Throughput: 100 requests / 312 iterations ≈ 0.32 req/iter
  - BUT in wall-clock time:
  - - Each iteration is faster (no idle)
  - - Overall 2-4x wall-clock speedup!

Why? Better GPU utilization:
- GPU stays at full capacity
- No wasted compute slots
- Memory-bound operations are more efficient

Real Benchmarks (from vLLM paper)

Model: Llama 2 7B
Hardware: Single A100 GPU
Request length distribution: Exponential (10-2048 tokens)
Batch size: 32 requests

Metric                    Static      Continuous
─────────────────────────────────────────────────
Throughput (req/sec)      6.7         22.4
Tokens/sec                800         3200
Batch utilization         25%         85%
GPU utilization           40%         92%
Latency (p50)             1.2s        0.4s
Latency (p99)             8.5s        2.1s

Improvement:             3.3x throughput, 4x latency reduction

Why the Improvement?

GPU Compute has two phases:
1. Forward pass (compute tokens)
2. KV cache access (read/write memory)

Static batching:
  - Token generation: GPU busy
  - Waiting for slow requests: GPU idle (20-30% utilization)
  - KV cache: Fragmented (variable sizes)

Continuous batching:
  - Token generation: GPU busy (always)
  - New request added immediately: No idle time
  - KV cache: More predictable pattern
  - Result: 80-90% GPU utilization consistently

---

Implementation Details

Request State Management

class Request:
    def __init__(self, request_id: str, prompt: str, max_tokens: int):
        self.id = request_id
        self.prompt = prompt
        self.max_tokens = max_tokens

        # State tracking
        self.tokens_generated = 0
        self.status = "waiting"  # waiting, running, completed
        self.start_time = None
        self.end_time = None

        # Output
        self.output_tokens = []
        self.attention_mask = None
        self.kv_cache = None

    def is_completed(self) -> bool:
        """Check if request is done"""
        return self.tokens_generated >= self.max_tokens

    def step(self, output_token: int, kv_cache):
        """One token generation step"""
        self.tokens_generated += 1
        self.output_tokens.append(output_token)
        self.kv_cache = kv_cache

        if self.is_completed():
            self.status = "completed"
            self.end_time = time.time()

Batch Manager Implementation

class BatchManager:
    """Manages batch of requests during generation"""

    def __init__(self, requests: List[Request]):
        self.requests = requests
        self.current_batch_idx = 0

    def prepare_batch_inputs(self):
        """Prepare model inputs for current batch"""
        batch_ids = []
        batch_attention_mask = []
        batch_kv_cache = []

        for req in self.requests:
            if not req.is_completed():
                batch_ids.append(req.output_tokens[-1])  # Last generated token
                # Attention mask based on KV cache size
                seq_len = len(req.output_tokens)
                batch_attention_mask.append(
                    torch.ones(seq_len, dtype=torch.long)
                )
                batch_kv_cache.append(req.kv_cache)

        # Stack into batch tensors
        return {
            "input_ids": torch.stack(batch_ids),
            "attention_mask": torch.stack(batch_attention_mask),
            "kv_cache": batch_kv_cache
        }

    def update_from_outputs(self, outputs):
        """Update requests with model outputs"""
        output_idx = 0
        for req in self.requests:
            if not req.is_completed():
                output_token = outputs[output_idx]
                kv_cache = outputs['kv_cache'][output_idx]
                req.step(output_token, kv_cache)
                output_idx += 1

    def get_completed_requests(self):
        """Return and remove completed requests"""
        completed = [req for req in self.requests if req.is_completed()]
        self.requests = [req for req in self.requests if not req.is_completed()]
        return completed

Main Inference Loop

def inference_loop_continuous_batching(
    model,
    incoming_requests_queue,
    max_batch_size: int = 32,
    max_iterations: int = 100000
):
    """
    Main inference loop with continuous batching
    """
    running_requests = []
    completed_requests = []

    for iteration in range(max_iterations):
        # Step 1: Remove completed requests from batch
        newly_completed = [req for req in running_requests if req.is_completed()]
        running_requests = [req for req in running_requests if not req.is_completed()]
        completed_requests.extend(newly_completed)

        # Step 2: Add new requests from queue
        while (len(running_requests) < max_batch_size and 
               not incoming_requests_queue.empty()):
            new_req = incoming_requests_queue.get_nowait()
            running_requests.append(new_req)
            new_req.status = "running"
            new_req.start_time = time.time()
            print(f"[Iter {iteration}] Started {new_req.id}, "
                  f"batch_size={len(running_requests)}")

        # Step 3: If no requests, wait for new arrivals
        if not running_requests:
            print("Batch empty, waiting for requests...")
            time.sleep(0.1)
            continue

        # Step 4: Prepare batch inputs
        batch_mgr = BatchManager(running_requests)
        batch_inputs = batch_mgr.prepare_batch_inputs()

        # Step 5: Forward pass
        with torch.no_grad():
            outputs = model(**batch_inputs)

        # Step 6: Update request states
        batch_mgr.update_from_outputs(outputs)

        # Step 7: Log status
        if iteration % 100 == 0:
            print(f"[Iter {iteration}] "
                  f"Running: {len(running_requests)}, "
                  f"Completed: {len(completed_requests)}, "
                  f"Queued: {incoming_requests_queue.qsize()}")

        # Step 8: Check termination
        if (not running_requests and 
            incoming_requests_queue.empty() and 
            len(completed_requests) > 0):
            print("All requests completed!")
            break

    return completed_requests

---

Scheduling Strategies

1. FIFO (First-In-First-Out)

class FIFOScheduler:
    """Process requests in order of arrival"""

    def schedule(self):
        # Remove completed
        self.running = [r for r in self.running if not r.is_completed()]

        # Add waiting (in order)
        while len(self.running) < self.max_batch_size and self.waiting:
            self.running.append(self.waiting.pop(0))

        # Process batch
        return self.forward_pass(self.running)

# Pros: Fair, simple, predictable
# Cons: Doesn't optimize for latency

2. SJF (Shortest Job First)

class SJFScheduler:
    """Prioritize short requests for lower latency"""

    def schedule(self):
        # Remove completed
        self.running = [r for r in self.running if not r.is_completed()]

        # Sort waiting by expected length (shortest first)
        self.waiting.sort(key=lambda r: r.max_tokens)

        # Add shortest requests first
        while len(self.running) < self.max_batch_size and self.waiting:
            self.running.append(self.waiting.pop(0))

        # Process batch
        return self.forward_pass(self.running)

# Pros: Better latency for short requests
# Cons: May starve long requests

3. Preemptive SJF

class PreemptiveSJFScheduler:
    """Allow preemption of long requests by short ones"""

    def schedule(self):
        # Check if new short request arrived
        if self.waiting and self.waiting[0].max_tokens < 20:
            # Preempt longest running request
            longest_req = max(self.running, key=lambda r: r.max_tokens)
            self.waiting.insert(0, longest_req)
            self.running.remove(longest_req)

            # Add short request
            self.running.append(self.waiting.pop(0))
        else:
            # Normal FIFO
            self.running = [r for r in self.running if not r.is_completed()]
            while len(self.running) < self.max_batch_size and self.waiting:
                self.running.append(self.waiting.pop(0))

        return self.forward_pass(self.running)

# Pros: Better p99 latency
# Cons: Overhead from preemption

4. Length-Aware Scheduling

class LengthAwareScheduler:
    """Balance batch diversity to minimize tail latency"""

    def schedule(self):
        # Remove completed
        self.running = [r for r in self.running if not r.is_completed()]

        # Aim for mix of short and long requests
        avg_running_length = np.mean([r.max_tokens for r in self.running])

        while len(self.running) < self.max_batch_size and self.waiting:
            # If batch has long requests, prefer short
            if avg_running_length > 100:
                next_req = min(self.waiting, key=lambda r: r.max_tokens)
            # If batch has short requests, prefer long
            else:
                next_req = max(self.waiting, key=lambda r: r.max_tokens)

            self.running.append(next_req)
            self.waiting.remove(next_req)

        return self.forward_pass(self.running)

# Pros: Balanced latency across request types
# Cons: More complex logic

---

Challenges and Solutions

Challenge 1: Variable KV Cache Sizes

Problem:
Each request has different KV cache size:
- Request 1: 100-token KV cache
- Request 2: 2000-token KV cache
- Request 3: 500-token KV cache
- Total: 2600 token * model_dim memory (fragmented!)

Solution: PagedAttention
- Divide KV cache into fixed pages (512 tokens)
- Allocate pages on-demand
- Free pages immediately when request completes
- Result: Non-contiguous allocation is fine!

Challenge 2: Attention Computation Inefficiency

Problem:
Attention matrix size varies per request:
- Request 1: 100 × 100 = 10K attention scores
- Request 2: 2000 × 2000 = 4M attention scores
- Request 3: 500 × 500 = 250K attention scores

Can't batch efficiently (different sizes!)

Solution 1: Pad to max length (waste)
Solution 2: Process separately (serial)
Solution 3: Use flash attention (kernel level optimization)

Challenge 3: Request Arrival Unpredictability

Problem:
Can't predict when requests will arrive:
- Batch might be underutilized
- New requests might arrive in bursts
- Need to balance latency vs throughput

Solution: Adaptive batching
- Prefill phase: Process prompt with multiple requests
- Decode phase: Keep batch size optimal
- Dynamic timeout: Wait briefly for new arrivals

Challenge 4: Memory Overhead

Problem:
More batching = more memory for KV cache:
- Single request (2048 tokens): 1GB KV cache
- 32 requests × varied lengths: 24GB KV cache
- Hits memory limits quickly

Solution: KV cache quantization + paging
- Quantize to int8: 50% memory reduction
- PagedAttention: 50% memory reduction
- Combined: 75% memory reduction

---

Continuous Batching in vLLM

How vLLM Implements It

# vLLM's approach (simplified)

class LLMEngine:
    def generate(self, requests):
        """Main generation loop"""
        while self.has_requests():
            # Step 1: Schedule (continuous batching logic)
            scheduled_requests = self.scheduler.schedule(
                running_requests=self.running,
                waiting_requests=self.waiting
            )

            # Step 2: Execute (process scheduled requests)
            outputs = self.execute_step(scheduled_requests)

            # Step 3: Finalize (update states)
            for req, output in zip(scheduled_requests, outputs):
                req.add_token(output)

                # Automatically remove when complete
                if req.is_finished():
                    self.running.remove(req)
                    self.finished.add(req)

            # Step 4: Add waiting requests (if space)
            self.add_new_requests()

class Scheduler:
    def schedule(self, running, waiting):
        """
        vLLM's continuous batching scheduler
        """
        # Phase 1: Prefill (process prompts, cache KV)
        prefill_reqs = self.get_prefill_requests(running, waiting)

        # Phase 2: Decode (generate tokens, use cache)
        decode_reqs = self.get_decode_requests(running)

        # Phase 3: Schedule (decide what to execute)
        if prefill_reqs and can_fit(prefill_reqs):
            return prefill_reqs
        else:
            # Add as many decode requests as possible
            return decode_reqs[:self.max_batch_size]

Key Features

vLLM's continuous batching:
✓ Seamless addition/removal of requests
✓ Per-request scheduling (not per-token)
✓ Adaptive batch sizing
✓ Integration with PagedAttention for efficient memory
✓ Support for priority/weighted scheduling
✓ Streaming outputs

---

Comparison: Scheduling Approaches

Approach            Throughput  Latency (p50)  Latency (p99)  Complexity
─────────────────────────────────────────────────────────────────────────
Static (Batch=32)   1x          1x             1x             Low
Static (Batch=64)   1.2x        1.1x           1.2x           Low
Continuous FIFO     2.5x        0.6x           0.8x           Medium
Continuous SJF      2.5x        0.4x           0.5x           High
Continuous Adaptive 2.8x        0.5x           0.4x           Very High

---

Real-World Performance Examples

Example 1: Chat Application

Setup:
- Model: Llama 2 7B
- Hardware: 1x A100 GPU
- Request pattern: Chat (exponential length distribution)
- Max batch size: 32

Static Batching Results:
  - Throughput: 8 requests/sec
  - Avg latency: 2.1 seconds
  - P99 latency: 12.5 seconds
  - GPU util: 45%

Continuous Batching Results:
  - Throughput: 25 requests/sec (3.1x improvement!)
  - Avg latency: 0.8 seconds (2.6x faster!)
  - P99 latency: 3.2 seconds (3.9x faster!)
  - GPU util: 88%

Operational Impact:
  - Cost per 1M requests: $8.00 → $2.60 (67% savings!)
  - Can now serve 3x more users on same hardware

Example 2: Batch Processing (Document Analysis)

Setup:
- Model: Llama 2 13B
- Hardware: 8x A100 GPUs
- Task: Analyze 10,000 documents
- Request pattern: Uniform length (all ~500 tokens)

Static Batching:
  - Batch size: 64
  - Processing time: ~2 hours
  - GPU utilization: 70%

Continuous Batching:
  - Effective batch size: 96 (higher fill)
  - Processing time: ~1.4 hours (30% faster)
  - GPU utilization: 92%

Why less improvement here?
- Uniform request length → less fragmentation
- Static batching works well for uniform requests
- Continuous batching shines with variable lengths!

---

Best Practices

✅ Do's

1. Use continuous batching for any production serving
2. Pair with PagedAttention for memory efficiency
3. Implement adaptive batch sizing based on request arrival rate
4. Monitor GPU utilization (should be 80%+ with continuous batching)
5. Use KV cache quantization to maximize batch size
6. Implement request prioritization for SLAs
7. Profile different schedulers for your workload
8. Track metrics: throughput, latency p50/p99, GPU util

❌ Don'ts

1. ❌ Use static batching in production (unless uniform workload)
2. ❌ Ignore variable request lengths
3. ❌ Set batch size too large (memory OOM)
4. ❌ Set batch size too small (GPU underutilized)
5. ❌ Forget to implement request timeout/deadline
6. ❌ Mix requests of very different priority without scheduling
7. ❌ Ignore KV cache fragmentation
8. ❌ Assume single-GPU behavior scales to multi-GPU

---

Monitoring and Metrics

Key Metrics to Track

class ContinuousBatchingMetrics:
    def __init__(self):
        self.throughput = []  # requests/sec
        self.latency_p50 = []
        self.latency_p99 = []
        self.batch_size = []
        self.gpu_utilization = []
        self.cache_efficiency = []

    def log_iteration(self, iteration_time, batch_size, requests_completed):
        self.throughput.append(requests_completed / iteration_time)
        self.batch_size.append(batch_size)

    def report(self):
        return {
            "avg_throughput": np.mean(self.throughput),
            "avg_batch_size": np.mean(self.batch_size),
            "gpu_util": self.gpu_utilization[-1],
            "p99_latency": np.percentile(self.latency_p99, 99),
        }

---

Future Improvements

Active Research

1. Speculative Decoding: Prefetch likely tokens
2. Adaptive Batch Sizing: Dynamically adjust max_batch_size
3. Request Reordering: Reorder mid-generation for efficiency
4. GPU-CPU Offloading: Overflow KV cache to CPU
5. Compression: Dynamically compress KV cache

Emerging Techniques

• Hierarchical Batching: Multi-level batch management
• Predictive Scheduling: ML-based request scheduling
• Adaptive Precision: Vary precision based on token importance

---

Key Takeaways

🔑 Continuous batching maintains full batch size throughout generation
📊 2-4x throughput improvement over static batching
⚡ Better GPU utilization (80-90% vs 40-50%)
💾 Requires PagedAttention + KV cache quantization
📈 P99 latency: 4x reduction (critical for user experience)
🎯 Essential for production LLM serving

---

Further Reading

• vLLM Paper: "Efficient Memory Management for Large Language Model Serving"
• HF TGI: Text Generation Inference (similar approach)
• Orca/SGLang: Similar continuous batching implementations
• Adaptive Batching Research: Recent papers on dynamic scheduling

---

Related Notes

• Pagedattention
• Kv Cache
• 01 Vllm
• Llm Inference Optimization
• Throughput Vs Latency Tradeoffs