Distributed Training quiz

Question 1

What is the difference between data parallelism and model parallelism?

Accepted Answer

Data parallelism replicates the model and splits data; model parallelism splits the model across devices

Answer

Data parallelism splits the model; model parallelism splits the data

Answer

They are identical when using gradient checkpointing

Answer

Model parallelism is only useful for RNN architectures

Question 2

What is the ZeRO optimizer and what are its three stages?

Accepted Answer

Stage 1 shards optimizer states; Stage 2 adds gradient sharding; Stage 3 adds parameter sharding

Answer

Stage 1 shards parameters; Stage 2 shards gradients; Stage 3 shards optimizer states

Answer

All three stages shard the same data with different compression ratios

Answer

ZeRO only works with the SGD optimizer

Question 3

What is NCCL and why is it preferred over Gloo for GPU training?

Accepted Answer

NCCL is preferred for GPUs as it uses NVLink/RDMA for high-bandwidth GPU-to-GPU communication

Answer

Gloo is preferred for GPU training due to better fault tolerance

Answer

NCCL and Gloo produce identical throughput on modern InfiniBand networks

Answer

NCCL only supports all-reduce; Gloo supports all collective operations

Question 4

What is pipeline parallelism and how does the 1F1B schedule reduce the pipeline bubble?

Accepted Answer

1F1B interleaves forward and backward micro-batches across stages to reduce idle bubble time

Answer

1F1B means one GPU does forward; another does backward, with no overlap

Answer

1F1B requires twice as much memory as GPipe but is slower

Answer

1F1B is only used in tensor parallelism, not pipeline parallelism

Question 5

How does DistributedSampler ensure non-overlapping data across GPUs?

Accepted Answer

DistributedSampler assigns non-overlapping index strides per rank, requiring set_epoch for re-shuffling

Answer

DistributedSampler randomly assigns overlapping subsets to each rank

Answer

DistributedSampler is only needed when num_workers > 0

Answer

DistributedSampler automatically calls set_epoch each epoch internally

Question 6

What is the difference between synchronous and asynchronous distributed SGD?

Accepted Answer

Sync SGD has all workers all-reduce before updating; async SGD allows stale gradient updates

Answer

Async SGD converges faster and is used in all modern LLM training

Answer

Sync SGD is only possible with a parameter server architecture

Answer

Async SGD and sync SGD produce identical convergence given enough epochs

Question 7

What is the critical difference between nn.DataParallel and DistributedDataParallel?

Accepted Answer

DDP uses separate processes per GPU avoiding the GIL; DataParallel uses threads and has GPU 0 bottleneck

Answer

DataParallel is faster for multi-GPU training due to shared memory

Answer

DataParallel supports multi-node training; DDP does not

Answer

They are equivalent in performance for up to 4 GPUs

Question 8

How does gradient checkpointing interact with pipeline parallelism?

Accepted Answer

Checkpointing complements pipeline parallelism by reducing per-stage activation memory, enabling more micro-batches

Answer

Gradient checkpointing is incompatible with pipeline parallelism

Answer

Pipeline parallelism already eliminates the need for gradient checkpointing

Answer

Gradient checkpointing doubles pipeline bubble fraction

Question 9

What is reduce-scatter and how is it used in FSDP?

Accepted Answer

reduce-scatter reduces and distributes disjoint gradient chunks to each rank; used by FSDP for ZeRO-3

Answer

reduce-scatter broadcasts the full gradient to all ranks

Answer

reduce-scatter is only available in the Gloo backend

Answer

FSDP uses all-reduce instead of reduce-scatter for better compatibility

Question 10

What is the all-gather collective operation and where is it used in distributed inference?

Accepted Answer

all-gather collects chunks from each rank and concatenates them so all ranks hold the full tensor

Answer

all-gather reduces tensor values and sends the sum to all ranks

Answer

all-gather only works for 1D tensors along dim=0

Answer

all-gather is equivalent to broadcast from rank 0

Question 11

What is expert parallelism in the context of Mixture of Experts (MoE) models?

Accepted Answer

Expert parallelism assigns experts to different GPUs with all-to-all routing of tokens

Answer

Expert parallelism replicates all experts on every GPU for redundancy

Answer

Expert parallelism is identical to tensor parallelism applied to MoE layers

Answer

Expert parallelism only works with a fixed number of experts equal to world_size

Question 12

What is the FP8 training format and which hardware supports it?

Accepted Answer

FP8 uses E4M3 or E5M2 formats and is natively accelerated on NVIDIA Hopper H100 GPUs

Answer

FP8 training uses 8-bit fixed-point without any exponent bits

Answer

FP8 training is supported on all NVIDIA GPUs from Volta onwards

Answer

FP8 training always produces higher loss than BF16 training

Question 13

How does bfloat16 differ from float16 for distributed training stability?

Accepted Answer

bfloat16 has wider dynamic range (8 exponent bits) matching float32, avoiding gradient overflow without loss scaling

Answer

bfloat16 has more mantissa bits than float16 and is more numerically precise

Answer

float16 and bfloat16 are identical except for hardware acceleration differences

Answer

bfloat16 requires GradScaler just like float16

Question 14

What is the collective operation barrier() and why must it be used carefully?

Accepted Answer

barrier() forces all ranks to wait until every rank reaches it; overuse causes deadlocks and slowdowns

Answer

barrier() should be called after every optimizer.step() for safety

Answer

barrier() is the fastest way to synchronize gradients across GPUs

Answer

barrier() is deprecated in PyTorch 2.x; use dist.all_reduce instead

Question 15

What is the Chinchilla scaling law's implication for distributed training resource allocation?

Accepted Answer

Chinchilla showed optimal tokens per parameter is ~20; models should train longer rather than grow larger

Answer

Chinchilla showed larger models always outperform smaller models given equal compute

Answer

Chinchilla scaling law only applies to image models, not language models

Answer

Chinchilla recommends 100 tokens per parameter for optimal training

Question 16

How does FSDP handle model saving and loading (state_dict)?

Accepted Answer

FSDP offers FULL, SHARDED, and LOCAL state_dict types; SHARDED is preferred for large models

Answer

FSDP always saves a full model checkpoint from rank 0 automatically

Answer

FSDP cannot save checkpoints during training; only after training ends

Answer

FSDP state_dict is identical to DDP state_dict with no special handling needed

Question 17

What is MFU (Model FLOP Utilization) and how is it measured?

Accepted Answer

MFU = observed_throughput × model_FLOPs / GPU_peak_TFLOPS; measures actual vs theoretical compute use

Answer

MFU measures GPU memory usage as a fraction of total available VRAM

Answer

MFU is only applicable to convolutional networks, not transformers

Answer

MFU above 80% indicates the training run has a bug causing excessive computation

Question 18

What is ring all-reduce topology and why is it bandwidth-optimal?

Accepted Answer

Ring all-reduce is bandwidth-optimal: each GPU sends 2(N-1)/N × tensor_size regardless of N

Answer

Ring all-reduce is bandwidth-suboptimal because each GPU sends data N times

Answer

Ring all-reduce requires a physical ring network topology on the hardware

Answer

Ring all-reduce only works for gradient tensors, not activation tensors

Question 19

What is the purpose of the find_unused_parameters argument in DDP?

Accepted Answer

find_unused_parameters scans backward outputs to exclude unused params from sync; needed for dynamic graphs

Answer

find_unused_parameters speeds up DDP by skipping gradient communication for all parameters

Answer

find_unused_parameters should always be True for maximum safety

Answer

find_unused_parameters is required for gradient accumulation with DDP

Question 20

What is the broadcast collective and what is a common use case in training?

Accepted Answer

broadcast copies a tensor from a source rank to all other ranks; used for model initialization

Answer

broadcast sends a tensor from all ranks to rank 0 for aggregation

Answer

broadcast is equivalent to all-reduce with ReduceOp.SUM

Answer

broadcast only works within a single node, not across nodes

Question 21

What is the advantage of using activation offloading to CPU in FSDP?

Accepted Answer

Activation offloading moves activations to CPU to free GPU VRAM at the cost of PCIe transfer latency

Answer

Activation offloading eliminates the need for gradient checkpointing entirely

Answer

Activation offloading is faster than keeping activations on GPU for all models

Answer

FSDP activation offloading requires a separate CPU memory allocator

Question 22

What is the purpose of static graph optimization in DDP (static_graph=True)?

Accepted Answer

static_graph=True disables per-iteration graph traversal overhead, enabling gradient bucket pre-building

Answer

static_graph=True enables dynamic graph updates across iterations

Answer

static_graph=True is required for mixed-precision training with DDP

Answer

static_graph=True increases memory usage for better throughput

Question 23

What does torch.distributed.all_to_all do and in which parallelism strategy is it used?

Accepted Answer

all_to_all performs a personalized exchange where each rank sends different chunks to every other rank

Answer

all_to_all is equivalent to broadcast from each rank simultaneously

Answer

all_to_all is used exclusively in data parallelism gradient synchronization

Answer

all_to_all requires all tensors to have the same dtype as the communication backend

Question 24

What is gradient staleness in asynchronous training and how does bounded staleness mitigate it?

Accepted Answer

Bounded staleness (SSP) limits how far fast workers advance ahead of slow workers, bounding staleness to τ steps

Answer

Bounded staleness eliminates all gradient staleness in asynchronous training

Answer

Gradient staleness only affects the first epoch of training

Answer

Bounded staleness is only used in synchronous DDP training

Question 25

What is checkpointing frequency and why is it critical for large-scale distributed training?

Accepted Answer

Checkpoint frequency balances lost work on failure vs I/O overhead; async checkpointing reduces pause

Answer

Frequent checkpointing has no downsides and should be done after every step

Answer

Checkpointing is only needed for runs longer than 7 days

Answer

Distributed checkpoint always requires saving from rank 0 only

Test yourself