Inference Latency vs Throughput Interview Preparation Guide

Introduction

In the landscape of 2026 AI engineering, balancing inference latency and throughput is the defining challenge of model serving. Latency refers to the time taken to process a single request, while throughput measures the number of requests processed per unit of time. As LLMs and multimodal models become central to production infrastructure, engineers must navigate the inherent tension between these metrics. High throughput often requires batching, which can increase individual request latency. Conversely, optimizing for ultra-low latency often leads to underutilization of expensive GPU resources. This topic is essential for ML Engineers, AI Infrastructure Engineers, and Backend Architects. Interviewers ask about this to test your understanding of hardware constraints, queueing theory, and the trade-offs required to meet strict Service Level Agreements (SLAs). Junior candidates are expected to understand the basic definitions and the impact of batching; senior candidates must demonstrate proficiency in designing systems that optimize for both, utilizing advanced techniques like continuous batching, speculative decoding, and model parallelism to maximize hardware efficiency without violating latency budgets.

Why It Matters

The business value of optimizing inference latency vs throughput is measured directly in infrastructure cost and user experience. For a real-time chatbot, a latency increase of 100ms can lead to a 5% drop in user engagement. Conversely, for a batch processing pipeline, throughput is the primary driver of cost-efficiency. In 2026, the shift toward agentic workflows and multi-agent systems has made this trade-off more complex; agents often require multiple sequential calls, making latency cumulative. A strong interview answer demonstrates that you view these metrics not as isolated numbers, but as variables in a cost-performance optimization problem. You must be able to explain how to tune batch sizes to saturate GPU compute units (SMs) while keeping the queueing delay within the limits defined by your SLA. Weak answers treat latency and throughput as independent variables or suggest 'just adding more GPUs' without acknowledging the cost and memory bandwidth bottlenecks. A high-signal candidate will discuss the specific impact of memory-bound vs compute-bound operations on the latency-throughput curve, showing they understand the underlying hardware architecture of modern AI accelerators.

Core Concepts

Architecture Overview

The inference serving pipeline involves a request arriving at a load balancer, being routed to a model instance, and processed through a scheduler that manages memory and compute resources. The scheduler must balance incoming requests to maximize throughput while minimizing the latency of individual requests.

Data Flow

Incoming requests are queued, then admitted into the scheduler, which allocates KV cache blocks. The compute engine performs forward passes, and the results are streamed back to the user.

[User Request]
       ↓
[Load Balancer]
       ↓
[Request Scheduler]
    ↙        ↘
[KV Cache]  [GPU Compute]
    ↘        ↙
[Response Stream]
       ↓
[User Output]

Key Components

Tools & Frameworks

Design Patterns

Continuous Batching Pattern Execution Pattern

Implementing a scheduler that injects new requests into the batch as soon as previous requests finish, rather than waiting for the entire batch to complete.

Trade-offs: Increases throughput significantly but requires complex memory management for the KV cache.

Speculative Drafting Pattern Latency Optimization

Using a small model to generate a sequence of tokens and a large model to verify them in a single batch pass.

Trade-offs: Reduces latency for the large model but increases total compute cycles required.

Dynamic Batching Pattern Throughput Optimization

Configuring the server to wait for a short window (e.g., 5ms) to collect multiple requests before executing a forward pass.

Trade-offs: Improves throughput but introduces a fixed latency floor for every request.

Common Mistakes

Production Considerations

Reliability	Use circuit breakers to prevent cascading failures when latency spikes; implement health checks that monitor GPU health, not just process status.
Scalability	Horizontal scaling of model replicas; use a distributed scheduler to manage requests across a cluster of GPUs.
Performance	Target P99 latency for TTFT; use profiling tools like Nsight Systems to identify kernel-level bottlenecks.
Cost	Optimize for throughput to reduce cost-per-request; use spot instances for non-real-time batch inference.
Security	Rate limit requests to prevent DoS attacks that target expensive GPU compute; sanitize inputs to prevent prompt injection.
Monitoring	Track TTFT, throughput (tokens/sec), GPU utilization, and KV cache memory pressure.

Key Trade-offs

•Latency vs Throughput

•Memory Usage vs Compute Speed

•Model Size vs Inference Speed

Scaling Strategies

•Horizontal Pod Autoscaling

•Model Parallelism

•Request Queueing

Optimisation Tips

•Use PagedAttention to reduce memory fragmentation.

•Enable FP8 quantization to increase throughput.

•Tune batch size based on real-world request distribution.

FAQ

How do latency and throughput differ in inference?

Latency is the time taken to process a single request (usually measured in milliseconds), while throughput is the number of requests or tokens processed per second. They are often inversely related: increasing throughput via batching typically increases individual request latency.

Why is batching used in LLM inference?

Batching allows the GPU to process multiple requests simultaneously, amortizing the cost of loading model weights and maximizing compute utilization. This significantly increases total throughput, though it can increase the latency for individual requests if the batch size is too large.

What is Time to First Token (TTFT)?

TTFT is the latency between sending a request and receiving the first generated token. It is a critical metric for user-perceived performance in interactive applications, as it dictates how quickly the user sees the system start responding.

What is the difference between compute-bound and memory-bound inference?

Compute-bound inference is limited by the GPU's floating-point performance (TFLOPS), while memory-bound inference is limited by the speed at which data can be moved from GPU memory (HBM) to the compute cores. Most LLM inference is memory-bound.

How does PagedAttention help with throughput?

PagedAttention manages the KV cache in non-contiguous blocks, similar to virtual memory in operating systems. This reduces memory fragmentation, allowing for larger batch sizes and more concurrent requests, which directly increases throughput.

Can I improve both latency and throughput simultaneously?

Yes, through architectural optimizations like speculative decoding, kernel fusion, and efficient memory management. These techniques reduce the compute or memory work per token, allowing the system to handle more requests (throughput) faster (latency).

What is the role of the scheduler in an inference server?

The scheduler manages the queue of incoming requests, decides which requests to batch together, and handles the allocation of GPU resources and KV cache memory. It is the core component for balancing latency and throughput.

Why does increasing batch size cause OOM errors?

Each request in a batch requires its own KV cache memory. If the batch size is too large, the total memory required for all KV caches plus the model weights exceeds the GPU's available HBM, leading to an Out-of-Memory (OOM) error.

What is speculative decoding?

Speculative decoding uses a small, fast model to draft a sequence of tokens, which are then verified by a larger, slower model in a single forward pass. This reduces the number of times the large model needs to be run, significantly lowering latency.

How do I choose the right batch size?

The optimal batch size is found through load testing. You increase the batch size until the latency for your P99 target is reached or until GPU memory utilization becomes the limiting factor. It depends on the model, hardware, and traffic pattern.