A study path through all 60 lectures of the GPU MODE series, grouped into eight tracks. Slide PDFs are in Books/GPU-MODE-Lectures/ and the videos are on YouTube. The companion textbook is Programming Massively Parallel Processors (PMPP).
How to study a lecture: read the track note → watch the video with the PDF open → profile one kernel yourself.
Tracks
| # | Track | Lectures | Focus |
|---|---|---|---|
| 1 | CUDA Foundations | 1–5, 8, 16, 41 | Execution model, memory hierarchy, profiling |
| 2 | Parallel Algorithms & Fused Kernels | 9, 12, 13, 18–21, 24 | Reduce, scan, Flash/Ring Attention, fusion |
| 3 | Triton, Compilers & Kernel DSLs | 10, 14, 28, 29, 42, 79, 104, 106, Bonus | Writing kernels at a higher level |
| 4 | CUTLASS, CuTe, Tensor Cores | 15, 23, 36, 37, 43, 57, 75, 86, 103, 114 | Peak GEMM/attention on modern NVIDIA hardware |
| 5 | Quantization, Sparsity & Numerics | 7, 11, 30, 33, 34, 38, 69, 84 | Low precision and skipping work |
| 6 | Multi-GPU & Distributed | 6, 17, 39, 67, 70, 78 | NCCL/NVSHMEM collectives, parallelism strategies |
| 7 | LLM Systems & Inference | 22, 32, 35, 40, 71, 72, 74 | Serving engines, long context, attention variants |
| 8 | Beyond NVIDIA | 25, 26, 27, 31 | AMD, Intel, Apple, WebGPU |
1 Foundations ──► 2 Algorithms ──┬──► 3 Triton & DSLs ──┐
│ ├──► 4 CUTLASS / CuTe ─┴──► 5 Quantization ──┐
│ └──► 6 Multi-GPU ────────────────────────────┴──► 7 LLM Systems
└──► 8 Beyond NVIDIALecture Index
| Lecture | Title | Track | Slides |
|---|---|---|---|
| 1 | Profiling and Integrating CUDA Kernels in PyTorch | 1 | |
| 2 | Recap Ch. 1–3 from the PMPP Book | 1 | |
| 3 | Getting Started With CUDA | 1 | — |
| 4 | Intro to Compute and Memory Architecture | 1 | |
| 5 | Going Further with CUDA for Python Programmers | 1 | — |
| 6 | Optimizing PyTorch Optimizers | 6 | — |
| 7 | Advanced Quantization | 5 | — |
| 8 | CUDA Performance Checklist | 1 | — |
| 9 | Reductions | 2 | — |
| 10 | Build a Prod Ready CUDA Library | 3 | — |
| 11 | Sparsity | 5 | |
| 12 | Flash Attention | 2 | — |
| 13 | Ring Attention | 2 | |
| 14 | Practitioner's Guide to Triton | 3 | — |
| 15 | CUTLASS | 4 | — |
| 16 | Hands-on Profiling | 1 | — |
| Bonus | CUDA C++ llm.cpp | 3 | — |
| 17 | GPU Collective Communication (NCCL) | 6 | |
| 18 | Fused Kernels | 2 | — |
| 19 | Data Processing on GPUs | 2 | — |
| 20 | Scan Algorithm | 2 | — |
| 21 | Scan Algorithm Part 2 | 2 | — |
| 22 | Speculative Decoding in vLLM | 7 | — |
| 23 | Tensor Cores | 4 | — |
| 24 | Scan at the Speed of Light | 2 | — |
| 25 | Speaking Composable Kernel (AMD) | 8 | |
| 26 | SYCL Mode (Intel GPU) | 8 | — |
| 27 | gpu.cpp | 8 | — |
| 28 | Liger Kernel | 3 | — |
| 29 | Triton Internals | 3 | |
| 30 | Quantized Training | 5 | |
| 31 | Beginner's Guide to Metal Kernels | 8 | |
| 32 | Unsloth — LLM Systems Engineering | 7 | — |
| 33 | BitBLAS | 5 | |
| 34 | Low Bit Triton Kernels | 5 | — |
| 35 | SGLang Performance Optimization | 7 | |
| 36 | CUTLASS and Flash Attention 3 | 4 | |
| 37 | SASS & GPU Microarchitecture | 4 | |
| 38 | Lowbit Kernels for ARM CPU | 5 | |
| 39 | TorchTitan | 6 | — |
| 40 | FlashInfer | 7 | — |
| 41 | CUDA Docs for Humans | 1 | — |
| 42 | Mosaic GPU | 3 | — |
| 43 | int8 Matmul on Turing | 4 | |
| 57 | CuTe | 4 | |
| 67 | NCCL & NVSHMEM | 6 | — |
| 69 | Quartet: 4-bit Training | 5 | — |
| 70 | Fault-Tolerant Communication Collectives | 6 | — |
| 71 | FlexOlmo | 7 | |
| 72 | Efficient Long-Context Modeling | 7 | |
| 74 | Positional Encodings and PaTH Attention | 7 | |
| 75 | GPU Programming Fundamentals + ThunderKittens | 4 | PDF · PDF |
| 78 | Iris: Multi-GPU Programming in Triton | 6 | |
| 79 | Mirage (MPK): Mega Kernels | 3 | |
| 84 | Numerics and AI | 5 | |
| 86 | Introduction to CuTeDSL | 4 | |
| 103 | CuTe Layout Algebra | 4 | |
| 104 | Gluon: Tile-Based GPU Programming | 3 | |
| 106 | HF Kernels | 3 | — |
| 114 | PyCuTe | 4 |
Lectures without a PDF are video only or have slides hosted externally. The track notes link to those slides.