Not confident in your math background? This page helps you assess your readiness and find the best free resources to fill any gaps.
The math required for GPU programming is more accessible than it looks. If you can follow recipes and understand that 3 rows of 4 items equals 12 items, you already have the intuition. These resources will help you formalize that intuition into the precise language GPUs speak.
Take this 5-question quiz to assess your current readiness. Be honest with yourself—the goal is to identify what you need to learn, not to pass a test.
Understanding vectors and matrices is fundamental. You don't need proof-heavy linear algebra—focus on building visual and computational intuition.
The gold standard for visual intuition. Covers vectors, linear transformations, matrix multiplication, and eigenvalues with stunning animations. Watch chapters 1-7 for GPU programming prerequisites.
Structured course with practice problems. Great for reinforcing concepts after watching 3Blue1Brown. Focus on "Vectors and spaces" and "Matrix transformations" units.
Full university course. More rigorous and complete coverage. Watch lectures 1-10 for core concepts. Professor Strang is legendary for making linear algebra accessible.
Matrix multiplication is the core operation in GPU computing. You need to understand both the mechanics (how to compute it) and the geometry (what it means).
Explains why matrix multiplication works the way it does. Understanding this transforms matmul from "mysterious formula" to "obvious composition of transformations."
Clear explanation of matrix multiplication with step-by-step examples. Shows how each output element is computed from rows and columns. Essential for understanding tiled matrix multiplication in CUDA.
Step-by-step walkthrough with practice exercises. Covers dimension rules, the mechanics of computation, and common properties. Complete the exercises until matmul feels automatic.
GPUs use various floating point formats (FP32, FP16, FP8, FP4). Understanding the bit layout helps you reason about precision, overflow, and quantization.
Toggle individual bits and see how the value changes. Best way to build intuition for sign, exponent, and mantissa. Try entering values like 0.1 to see why it can't be exactly represented.
Clear explanation of IEEE 754 format with hand-drawn diagrams. Covers why floats exist, how they work, and common pitfalls like precision loss.
The canonical reference by David Goldberg. Dense but comprehensive. Read sections 1-2 for fundamentals. Return to later sections when you encounter specific precision issues.
Real-world examples of floating point disasters: the Patriot missile bug, Ariane 5 explosion. Understanding these failures helps you write robust numerical code.
Softmax uses exp(), log-sum-exp tricks prevent overflow, and understanding growth rates helps you reason about numerical stability.
Beautiful explanation of why e appears everywhere. Understanding e^x's unique property (its derivative equals itself) explains why it's used in softmax and probability.
Text-based explanation with interactive examples. Great complement to the 3Blue1Brown video. Focuses on building intuition rather than formal proofs.
Complete coverage of logarithms with practice problems. Focus on "Introduction to logarithms" and "Properties of logarithms." The log-sum-exp trick uses these properties directly.
GPU kernels constantly convert between multi-dimensional indices and linear memory addresses. This is mechanical but crucial to get right.
Official NumPy documentation explaining strides and memory layout. Understanding this directly translates to CUDA memory access patterns.
Clear explanation with diagrams comparing row-major (C) vs column-major (Fortran) ordering. Essential for understanding why memory access patterns matter for performance.
Quick reference with formulas for index conversion. Bookmark this for when you're writing CUDA kernels and need to convert between 2D and linear indices.
For a matrix with dimensions [rows][cols] in row-major order:
linear_index = row * cols + col
For element [2][3] in a 5x6 matrix: 2 * 6 + 3 = 15
Choose the path that matches your current level. Time estimates assume focused study.
New to the math or haven't used it in years. Start here if the diagnostic quiz felt unfamiliar.
You've learned this before but it's rusty. Scored 2-3 on the diagnostic.
You scored 4-5 but missed specific questions. Focus only on what you need.
How to get the most out of these resources.
Pause videos frequently. Before watching how a matrix multiplication is computed, try to do it yourself. Predict what will happen next.
Practice problems cement understanding. If a resource has exercises, do them. You should be able to compute a 2x3 times 3x2 matrix multiplication by hand without looking anything up.
float.exposed and matrix visualizers let you experiment. "What happens if I flip this bit?" builds deeper understanding than reading about it.
20 hours over 2 weeks sticks better than 20 hours in 2 days. Review previous material briefly before starting new topics.
The goal isn't memorization—it's internalization. When someone says "3x4 times 4x2" and you instantly know the result is 3x2, you're ready.
Once you can score 5/5 on the diagnostic quiz, you have the math foundation needed for GPU kernel development.
Start: Attention Math Foundations Back to Course Index