marp: true theme: default paginate: true
Workload-Aware Selection of CUDA Graph Modes
Literature review consultation
Core question: Under different workloads, which vLLM execution mode offers a useful latency and GPU-memory balance?
This is a proposed study. No experiment has been completed.
Why I chose this topic
- While preparing for graduate job applications, I studied vLLM and SGLang.
- vLLM offers several CUDA Graph execution modes.
- I wanted to know when each mode is worth using in a changing LLM-serving workload.
My paper explains why that choice needs evidence.
What is vLLM?
vLLM is a system for serving a large language model to many requests.
- Prefill: process an incoming prompt.
- Decode: generate new tokens step by step.
- KV cache: retain information needed for later tokens.
Requests arrive and finish at different times, so the batch being processed changes. [3]
Kernel, launch, and CUDA Graph
| Term | Plain meaning |
|---|---|
| GPU kernel | One piece of work executed on the GPU. |
| Kernel launch | The CPU submits that piece of work to the GPU. |
| CUDA Graph | A recorded sequence of GPU operations that can be replayed through one graph launch. |
A CUDA Graph can still contain many kernels. Its main benefit here is reducing repeated CPU submission overhead. [1]
Why is mode selection difficult?
Round 1: A and B are decoding
Round 2: C enters for prefill; A and B keep decoding
Round 3: A finishes; B and C keep decoding
- The batch size and mix of prefill/decode work change.
- A captured graph must remain compatible with the work replayed.
- Capture time, GPU memory use, and fallback can also matter. [1], [2], [5]
Question: When do launch-time savings justify those costs?
What are the vLLM modes? [5]
| Mode | Basic idea |
|---|---|
NONE |
Run without CUDA Graph replay. |
PIECEWISE |
Graph-compatible parts use graphs; other parts run eagerly. |
FULL |
Use a full graph when the batch and backend support it. |
FULL_DECODE_ONLY |
Use a full graph for uniform decode; other batches run without a graph. |
FULL_AND_PIECEWISE |
Use full graphs for uniform decode and piecewise graphs for other supported batches. |
The vLLM documentation describes runtime dispatch and possible downgrade or fallback. FULL_AND_PIECEWISE is the documented default for the relevant v1 setup, with higher capture-time and memory costs. [5]
Six sources, six roles
Five research papers and one vLLM technical document.
| Source | One-sentence contribution to my review |
|---|---|
| GraCE [1] | Short-kernel launches can be costly; graph deployment should consider costs and benefits. |
| FlashInfer [2] | Attention scheduling can adapt to changing requests while retaining graph compatibility. |
| PagedAttention [3] | Request lengths and KV-cache allocations change during LLM serving. |
| PyTorch 2 [4] | Full or partial computation graphs can balance graph optimization with Python flexibility. |
| vLLM documentation [5] | Describes the current modes and how vLLM selects among them. |
| MPK [6] | A persistent mega-kernel offers a different way to reduce launch overhead. |
Important distinction: PyTorch's captured computation graph in [4] is not itself a CUDA Graph.
What is MPK? [6]
MPK = Mirage Persistent Kernel.
- Its compiler and runtime organize much of model inference into one persistent mega-kernel.
- CUDA Graph replay can launch a recorded sequence containing multiple kernels.
- MPK is a separate technical route, not one of vLLM's CUDA Graph modes.
Why it appears in my review: It shows that launch overhead can be addressed at more than one execution layer.
My draft, paragraph by paragraph
| Paragraph | One-sentence role |
|---|---|
| 1 | Introduce launch overhead, CUDA Graphs, and dynamic LLM workloads. |
| 2 | Discuss eager execution, partial computation graphs, and selective graph use. |
| 3 | Explain changing requests and the graph modes available in vLLM. |
| 4 | Introduce MPK and show that the studies evaluate different problems and settings. |
| 5 | State the missing controlled comparison. |
| 6 | State the proposed research aim. |
Revision focus: Make paragraphs 4–6 form one clear chain.
How the research gap follows
Launch overhead can matter [1]
↓
Graph replay helps but has constraints and costs [1], [6]
↓
LLM-serving batches change [2], [3]
↓
vLLM offers several execution modes [5]
↓
The reviewed studies test different systems and conditions [1]–[4], [6]
↓
They do not directly establish workload-to-mode boundaries for vLLM
Gap: The reviewed sources lack a controlled comparison of these vLLM modes as batch stability and request load vary. This is a conclusion about the sources reviewed, not a universal claim about all prior research.
Proposed aim and contribution
Hold constant: vLLM version, model, GPU, attention backend.
Vary: batch stability, request load, and execution mode.
Measure: latency, GPU memory use, capture time, and relevant fallback behavior.
Aim: Identify workload conditions under which each mode offers a favorable latency–memory trade-off.
Expected contribution: A practical mode-selection guide. Results are not yet known.
Teacher's feedback and my revision plan
The teacher said the ideas are properly referenced and relevant literature is discussed. The main request is to summarize the gap before presenting the research question.
| Feedback | Planned revision |
|---|---|
| MPK needs a clearer transition. | Explain why this alternative matters to the comparison, or shorten it. |
| Gap and aim feel separate. | Combine the final two paragraphs into gap → consequence → question/aim. |
implementation of? above the title. |
Ask whether the title should specify vLLM. |
| Abbreviation, author names, verbs, References formatting. | Define LLM once; correct grammar; use surnames in prose and hanging indentation in References. |
The teacher's example about “two gaps” is worth clarifying; my current argument has one main gap and a consequence that follows from it.
Questions for discussion
- Does the review clearly lead from the paper comparisons to my research gap?
- Is one main gap with a clear consequence easier to follow than two separate gaps?
- Did “implementation of?” mean that I should add “in vLLM” to the title?
- Is MPK relevant enough after I explain the transition?
Peer check: After reading my conclusion, can you say what evidence is missing and what I propose to investigate?
Sources
[1] GraCE, OSDI 2026
[2] FlashInfer, MLSys 2025
[3] PagedAttention, SOSP 2023
[4] PyTorch 2, ASPLOS 2024
[5] vLLM CUDA Graphs design documentation, v0.21.0
[6] MPK, OSDI 2026