The Parallel Mindset

Why Parallelism is Hard

You've written sequential code for years. Your intuition says: "make the thing faster." But parallel programming requires a different question: "how do I divide this work?"

Sequential thinking leads to wrong parallel code. Consider summing an array:

# Sequential (correct, simple)
total = 0
for x in array:
    total += x  # Each step depends on the previous

# Naive parallel (WRONG - race condition!)
total = 0
parallel_for x in array:
    total += x  # Multiple threads read/write total

The naive parallel version has a race condition: multiple threads read total, add their value, and write back—overwriting each other's work.

Common parallel hazards include:

• Race conditions: Multiple threads access shared data, at least one writes
• Deadlocks: Threads wait for each other in a cycle, forever
• Data dependencies: Step N needs the result of step N-1

The good news: GPU programming has patterns that avoid these hazards. The bad news: you need to learn to recognize which patterns apply to your problem.

The Speedup Limit

Amdahl's Law tells us the maximum speedup from parallelization. If part of your program is inherently sequential, it limits how fast the whole program can go—no matter how many processors you throw at it.

The devastating insight: as N → ∞, speedup approaches 1/S. If 10% of your code is sequential, maximum speedup is 10x. Period.

Categories of Parallelism

Not all problems parallelize equally. Recognizing which category your problem falls into determines your implementation strategy.

From CPU Vectors to GPU Warps

If you've used NumPy, you already understand parallel thinking at a basic level. a + b on arrays doesn't loop—it operates on all elements simultaneously. This is SIMD (Single Instruction, Multiple Data).

Modern CPUs have SIMD units like AVX-512 that process 8-16 elements at once. GPUs take this further with SIMT (Single Instruction, Multiple Threads): 32 threads execute in lockstep.

The key difference: a GPU doesn't have 32 lanes—it has thousands of warps, each with 32 threads. A B200 can have ~200,000+ threads in flight.

# NumPy: implicit parallelism (CPU SIMD under the hood)
result = np.exp(x) + np.log(y)  # Operates on all elements

# GPU thinking: explicit parallelism
# "What does ONE thread do to ONE element?"
# Then run that 1 million times in parallel

Thinking in Parallel

The hardest part of GPU programming isn't the syntax—it's the mental model. You must stop thinking about the computation and start thinking about how to divide the computation.

This mental shift takes practice. The good news: once it clicks, you'll see parallelization opportunities everywhere—even in code you thought was inherently sequential.

Hands-On Labs

Further Reading