Practice Notebooks

Verify GPU setup and dependencies

Establish CPU performance benchmarks

Instant GPU speedup with drop-in NumPy replacement

Understand SMs, warps, and the execution model

Write your first GPU kernel with Triton

Profile and understand memory access patterns

Implement tiled memory access for better cache utilization

Achieve 500+ GFLOPS with optimized matmul

Learn to use Nsight Compute for kernel analysis

Optimize global memory access patterns

Eliminate shared memory bottlenecks

Overlap compute and memory operations

Hardware-accelerated async data movement

Use MMA operations for matrix math

Put it all together for peak performance

Implement basic Q K^T computation

Why naive softmax fails at scale

Numerical stability with max subtraction

Complete attention implementation

Single-pass softmax algorithm

Block-wise computation for memory efficiency

Production-grade fused attention kernel

Convert between floating point formats

Symmetric and asymmetric quantization

Integer quantization for inference

NVIDIA's 4-bit floating point format

Efficient key-value cache management

Combine quantization with FlashAttention

Deploy optimized kernels in serving systems