GPU & CUDA quiz

Question 1

What is the CUDA thread hierarchy and how does it map to GPU hardware?

Accepted Answer

Threads → Blocks → Grid, with 32 threads per hardware warp scheduled on Streaming Multiprocessors

Answer

Threads → Warps → Blocks → Grid, all independently scheduled

Answer

Blocks → Threads → Grid, with blocks running on separate GPUs

Answer

Grid → SM → Warp → Thread, fully managed by cuDNN

Question 2

What is warp divergence and how does it hurt GPU performance?

Accepted Answer

Threads in the same warp taking different branches must execute serially with inactive threads masked, reducing throughput

Answer

Divergence causes warps to split across multiple SMs

Answer

Warp divergence is automatically resolved by the CUDA compiler

Answer

Divergence only occurs with more than 64 threads per block

Question 3

What is GPU occupancy and how do you maximize it?

Accepted Answer

Ratio of active warps to max warps per SM; limited by registers, shared memory, and threads per SM

Answer

The percentage of GPU cores actively computing vs. idle

Answer

Number of CUDA blocks executing concurrently across all GPUs

Answer

Total memory bandwidth utilization across all SMs

Question 4

What is pinned (page-locked) memory and why does it accelerate host-device transfers?

Accepted Answer

Page-locked host memory that allows DMA direct transfers at full PCIe bandwidth without intermediate staging copies

Answer

Memory allocated on the GPU's L2 cache permanently

Answer

Unified memory that is pinned to a specific NUMA node

Answer

GPU memory with ECC protection enabled

Question 5

What are tensor cores and which operations benefit most from them?

Accepted Answer

Matrix multiply-accumulate hardware accelerating GEMM operations in FP16/BF16/TF32 for 2-8x throughput

Answer

Cache units that store frequently accessed tensors near the CUDA cores

Answer

Dedicated hardware for PyTorch autograd computations

Answer

Vector units for element-wise activation function computation

Question 6

What is gradient checkpointing and what trade-off does it make?

Accepted Answer

Trades 33% more compute for drastically reduced activation memory by recomputing activations during backprop

Answer

Saves GPU memory by offloading optimizer states to CPU

Answer

Checkpoints model weights to disk every N steps

Answer

An alternative optimizer that uses less memory than Adam

Question 7

What is the difference between DataParallel and DistributedDataParallel in PyTorch?

Accepted Answer

DDP uses separate processes and NCCL AllReduce for efficient gradient sync; DP uses threads with GPU-0 bottleneck

Answer

DP supports multi-node; DDP is single-node only

Answer

They perform identically; DDP is deprecated

Answer

DP uses NVLink; DDP uses PCIe for communication

Question 8

How does Flash Attention improve attention computation efficiency?

Accepted Answer

Uses tiling to compute attention in shared memory without materializing the O(N²) matrix, reducing global memory IO

Answer

A faster attention kernel using INT8 quantization for the K and V matrices

Answer

A sparse attention pattern that skips 50% of attention weights

Answer

Flash Attention replaces softmax with a linear approximation

Question 9

What is the Roofline model and how do you use it to classify kernel performance?

Accepted Answer

Classifies kernels as compute-bound (right of ridge) or memory-bound (left of ridge) based on arithmetic intensity vs bandwidth ceiling

Answer

A model for predicting GPU training time from dataset size

Answer

A graph showing GPU utilization over training epochs

Answer

A hardware specification chart for different NVIDIA GPU generations

Question 10

What is cuDNN and what types of operations does it optimize?

Accepted Answer

GPU-accelerated DNN primitives (conv, pooling, norm, attention) with Tensor Core support and auto-algorithm selection

Answer

A CUDA compiler extension for deep learning code

Answer

A distributed training framework built on top of NCCL

Answer

A model serialization format for CUDA-compatible hardware

Question 11

How do you implement custom CUDA kernels in Python using CuPy or PyTorch?

Accepted Answer

CuPy RawKernel, PyTorch cpp_extension, or Triton @jit decorator for Python-native GPU kernel authoring

Answer

Only possible via ctypes bindings to precompiled .so files

Answer

Custom kernels require writing PTX assembly directly

Answer

Python cannot call custom CUDA kernels; must use C++

Question 12

What is paged attention and how does it solve LLM inference memory challenges?

Accepted Answer

Manages KV cache in fixed-size pages (OS virtual memory analogy) to eliminate fragmentation and increase LLM inference throughput

Answer

A sparse attention pattern that pages through attention heads sequentially

Answer

A disk-based offloading technique for KV cache in ultra-long contexts

Answer

Flash Attention's memory allocation strategy for the Q/K/V projection matrices

Question 13

What is torch.compile() and how does it accelerate PyTorch model training?

Accepted Answer

Compiles model via TorchDynamo + TorchInductor to fuse kernels and reduce memory allocations for 30-100% speedup

Answer

Converts PyTorch models to ONNX format for TensorRT deployment

Answer

Compiles Python code to C++ for faster data loading

Answer

Enables ahead-of-time compilation of CUDA kernels to PTX

Question 14

What is tensor parallelism and how does it differ from pipeline parallelism for LLM training?

Accepted Answer

TP splits weight matrices within a layer across GPUs; PP assigns different layers to different GPUs

Answer

TP distributes batches; PP distributes layers in the same way

Answer

Both are identical; the names reflect different frameworks

Answer

TP requires InfiniBand; PP uses NVLink only

Question 15

What is CUDA graph and how does it reduce kernel launch overhead?

Accepted Answer

Records GPU operations and replays them in one API call, eliminating per-kernel CPU launch overhead of 3-5 μs

Answer

A visual debugger for CUDA kernel execution flow

Answer

A static analysis tool for detecting CUDA memory errors

Answer

A profiling trace format for Nsight Systems

Question 16

What is the difference between compute-bound and memory-bandwidth-bound GPU kernels?

Accepted Answer

Compute-bound: arithmetic units saturated; memory-bound: data fetch rate limits throughput

Answer

Compute-bound kernels are always faster than memory-bound kernels

Answer

Memory-bound kernels use more CUDA cores than compute-bound kernels

Answer

The distinction only applies to CPU; GPU kernels are always compute-bound

Question 17

What is CUDA constant memory and when should you use it?

Accepted Answer

64 KB read-only cache with broadcast for uniform warp access; ideal for constants/lookup tables accessed by all threads identically

Answer

A persistent memory region that survives across kernel launches like a global variable

Answer

Cache for thread-local variables that exceed register capacity

Answer

An alternative to shared memory with higher bandwidth but less capacity

Question 18

How does CUDA kernel fusion reduce memory bandwidth consumption?

Accepted Answer

Combines multiple kernels into one, eliminating intermediate global memory reads/writes for 2-5x bandwidth reduction

Answer

Fuses GPU cores together for higher compute throughput

Answer

Combines forward and backward passes into a single kernel

Answer

A compiler optimization that merges identical kernels running in parallel streams

Test yourself