Literature Review Consultation Speech
Topic: Workload-Aware Selection of CUDA Graph Modes for Low-Latency LLM Serving
Use: A complete spoken explanation for a teacher-and-peer consultation. At a measured pace, allow about seven to nine minutes. If speaking time is short, use the two-minute version at the end. The Chinese notes below each section explain the meaning; speak the English paragraphs in your own voice.
Status: This is a proposed comparison. No experiment or performance result is being claimed.
1. Why I chose the topic
Hello everyone. While preparing for graduate job applications, I have been studying large language model inference and serving frameworks, especially vLLM and SGLang. I noticed that vLLM offers several ways to use CUDA Graphs. That made me ask a practical question: if the requests arriving at a model server keep changing, when is each execution mode a good choice? My literature review develops that question and explains why a controlled comparison would be useful.
中文意思： 我在准备校招时学习推理框架，看到vLLM有不同的CUDA Graph配置，因此想研究不同负载下该怎么选。文章的任务是提出有依据的研究问题。
2. The system and the technical problem
First, vLLM is a system for running a large language model and serving many users' requests. It processes the prompt in a prefill stage and then generates new tokens in a decode stage. It also manages the KV cache, which stores information needed for later tokens. As requests arrive, finish, or move between stages, the group of requests processed together changes.
A GPU kernel is one piece of GPU work. The CPU must launch, or submit, that work to the GPU. One inference iteration may involve many short kernels. When the CPU repeatedly launches them one by one, the accumulated launch time can matter. A CUDA Graph records a compatible sequence of GPU operations and lets the CPU replay that sequence with one graph launch. The graph still contains multiple kernels. It reduces repeated submission work; it does not automatically merge all the kernels into one.
中文意思： vLLM同时处理用户请求；prefill处理提示词，decode生成新词元。请求进出使batch变化。kernel是GPU执行的一项计算，launch是CPU提交这项计算。CUDA Graph把多项操作记录起来反复启动，图内部仍有多个kernel。
3. Why choosing a mode is difficult and what MPK means
This creates a trade-off. Replaying a graph can save CPU launch time, but the recorded execution must be compatible with the current batch. Capturing graphs can also take time and use extra GPU memory. vLLM therefore offers modes such as no CUDA Graph, piecewise graphs, full graphs, and combined modes that select a path for different batch types. The question is not simply whether graphs are fast. It is which mode gives a useful latency and memory balance under a particular workload.
One of my sources discusses MPK, or Mirage Persistent Kernel. MPK is a compiler and runtime system that puts much of the inference work into one persistent mega-kernel. Unlike a CUDA Graph, which can replay many separate kernels together, MPK changes how the GPU work is organized. It is an alternative approach to launch overhead, not a CUDA Graph mode in vLLM.
中文意思： graph可能节省CPU启动时间，却有capture时间、显存和batch兼容性成本。MPK是一条不同技术路线：尽量把工作放进一个持续运行的巨大kernel；它不在vLLM的graph模式列表中。
4. What my six sources contribute
I use six sources, although one is a technical document rather than a research paper. GraCE [1] explains why repeated launches of short kernels can be costly and why CUDA Graphs should be deployed selectively. FlashInfer [2] shows how attention scheduling can adapt to changing requests while remaining compatible with CUDA Graphs. PagedAttention [3] explains why request lengths and KV-cache allocations change during LLM serving. PyTorch 2 [4] shows how a compiler can capture full or partial computation graphs while keeping Python flexible; its computation graphs should not be confused with CUDA Graphs. The vLLM documentation [5] describes the execution modes and their runtime selection in the version I reviewed. Finally, MPK [6] offers the persistent mega-kernel approach as another way to reduce launch overhead.
中文意思： [1]解释开销和选择性使用；[2]解释动态attention调度；[3]解释请求与KV cache为何动态；[4]解释编译图和partial graph；[5]说明vLLM实际有哪些模式；[6]提供另一条mega-kernel路线。[5]是技术文档，不是论文。
5. How my draft is organized
My first paragraph introduces launch overhead, CUDA Graphs, and the difficulty of changing workloads. The second discusses eager execution, partial computation graphs, and selective CUDA Graph use. The third explains dynamic LLM requests and the modes available in vLLM. The fourth introduces MPK and compares what the studies actually evaluate. The final two paragraphs state the missing evidence and my proposed research aim. In revision, I want the connection between those last parts to be more direct.
中文意思： 按原稿顺序复述六段：背景、两种折中、LLM动态性与vLLM模式、MPK与研究差异、gap、aim。你会修改最后两段的连接。
6. The gap, research aim, and possible contribution
The gap follows from a comparison of these sources. They study different systems and use different models, hardware, software, workloads, and measures. Their findings help explain the problem, but they do not directly establish when a particular vLLM CUDA Graph mode is preferable under the same experimental conditions. This is a gap in the comparison evidence provided by the sources I reviewed; I am not claiming that nobody has studied CUDA Graphs in LLM serving.
My proposed study would keep the vLLM version, model, GPU, and attention backend fixed. I would vary batch stability and request load, compare the available execution modes, and measure latency, graph capture costs, and GPU memory use. The intended outcome is a practical guide for matching workload conditions to modes. I have not run this experiment yet, so I am not claiming that any mode is always best.
中文意思： 不同论文研究的问题和实验条件不一样，无法直接比较出当前vLLM的适用边界。拟议研究固定系统条件，只改变batch稳定性、请求负载和mode，比较延迟、capture和显存，形成选择依据。还没有实验结果。
7. What I learned from the teacher's feedback
My teacher's comments say that my ideas are appropriately referenced and that I discuss relevant literature. The main revision is in the conclusion: I should summarize the gap clearly before presenting my research question. The comments also suggest a better transition into the MPK discussion and possibly combining the final two paragraphs. I will define “large language model” at first mention, correct the author names and verb forms in the body, and format the References list with proper indentation. I would like to ask whether my title should specify vLLM, and whether the link from the literature critique to my proposed comparison is now clear enough.
中文意思： 老师肯定了引用与选文献，主要要求是结尾先归纳gap再提出研究问题；还要改MPK过渡、缩写、正文作者写法、动词和参考文献缩进。标题的批注有歧义，现场确认。
If I am asked a follow-up question
What is the main gap?
The reviewed sources do not give a controlled comparison that maps changing vLLM workloads to the most suitable execution mode.
Have you run the experiments?
No. My literature review motivates a proposed study; it does not report new benchmark results.
Why include MPK?
It addresses launch overhead through a different execution design. I need to make that connection clearer in my draft.
What does batch stability mean?
It means how much the number and type of requests in a batch change between inference iterations.
Two-minute version if speaking time is short
While preparing for graduate job applications, I have been learning vLLM and LLM inference. vLLM serves many model requests, and those requests change as users arrive, finish, or move from prompt processing to token generation. That raised my question: when should vLLM use each CUDA Graph mode?
A GPU kernel is one piece of GPU work, and the CPU launches it. CUDA Graphs can replay a recorded sequence of these operations with less repeated CPU launch overhead. However, graph capture takes time and memory, and changing batches can affect which graph can be used.
My sources discuss different parts of this problem: GraCE addresses launch costs; FlashInfer and PagedAttention examine dynamic serving; PyTorch 2 addresses graph capture; vLLM documentation describes the modes; and MPK presents a different mega-kernel approach. Together, they explain the problem, but they do not directly compare vLLM modes under the same workload conditions.
I propose comparing these modes while varying batch stability and request load, with the system and hardware held fixed. My teacher's main advice is to summarize this gap clearly before stating the research question. I would like feedback on whether that connection now makes sense.
Sources used in the draft
[1] Ghosh et al., GraCE, OSDI 2026
[2] Ye et al., FlashInfer, MLSys 2025
[3] Kwon et al., PagedAttention, SOSP 2023
[4] Ansel et al., PyTorch 2, ASPLOS 2024
[5] vLLM CUDA Graphs design documentation, v0.21.0
[6] Cheng et al., MPK, OSDI 2026