Distributed training and scaling quiz

Question 1

In synchronous data-parallel training, what is the primary role of the all-reduce collective operation?

Accepted Answer

Aggregates gradients across workers and returns the same averaged result to all of them

Answer

Splits the model parameters across workers to save memory

Answer

Shuffles the training data between workers each epoch

Answer

Broadcasts the optimizer state only from rank 0 to all ranks

Question 2

What memory components do the three stages of ZeRO (Zero Redundancy Optimizer) progressively shard across data-parallel ranks?

Accepted Answer

Stage1=optimizer states, Stage2=+gradients, Stage3=+parameters

Answer

Stage1=parameters, Stage2=+activations, Stage3=+gradients

Answer

Stage1=gradients, Stage2=+activations, Stage3=+optimizer states

Answer

Stage1=activations, Stage2=+parameters, Stage3=+optimizer states

Question 3

How does tensor (intra-layer) parallelism differ from pipeline (inter-layer) parallelism?

Accepted Answer

Tensor parallelism splits within a layer; pipeline parallelism splits across layers

Answer

Tensor parallelism splits across layers; pipeline parallelism splits within a layer

Answer

Both split the data batch but use different optimizers

Answer

Tensor parallelism replicates layers; pipeline parallelism replicates the batch

Question 4

In pipeline parallelism, what is the 'bubble' and how does increasing the number of micro-batches affect it?

Accepted Answer

Idle stage time during fill/drain; more micro-batches shrink it as (P-1)/(M+P-1)

Answer

Memory fragmentation; more micro-batches increase it

Answer

Gradient staleness; micro-batches have no effect on it

Answer

Network packet loss; more micro-batches always worsen it

Question 5

What is gradient accumulation and why is it used in distributed training?

Accepted Answer

Summing gradients over micro-batches before stepping to emulate a larger batch

Answer

Compressing gradients before all-reduce to save bandwidth

Answer

Caching gradients on disk for later replay

Answer

Averaging gradients across epochs to smooth training

Question 6

How does PyTorch FSDP (Fully Sharded Data Parallel) reduce memory compared to standard DDP?

Accepted Answer

It shards params/grads/optimizer states and gathers them on demand instead of replicating

Answer

It uses fp8 weights while DDP uses fp32

Answer

It offloads all activations to CPU automatically

Answer

It removes the need for any cross-GPU communication

Question 7

What problem does activation (gradient) checkpointing solve, and what is its cost?

Accepted Answer

Saves memory by recomputing activations in backward instead of storing them

Answer

Saves compute by caching gradients across steps

Answer

Reduces network traffic by checkpointing to a remote server

Answer

Improves accuracy by averaging activations over micro-batches

Question 8

When scaling the global batch size by a factor k, what does the linear scaling rule prescribe for the learning rate, and what caveat applies?

Accepted Answer

Scale LR linearly by k, with warmup; it breaks down at very large batches

Answer

Scale LR by sqrt(k); no warmup needed at any scale

Answer

Keep LR fixed; only the batch size matters

Answer

Divide LR by k to compensate for larger batches

Question 9

Why is ring all-reduce considered bandwidth-optimal and how much data does each worker send?

Accepted Answer

Per-worker traffic is ~2x the tensor size, roughly independent of N

Answer

Per-worker traffic grows linearly with N

Answer

Per-worker traffic grows as N^2

Answer

All traffic flows through a single coordinator node

Question 10

What is the difference between all-gather and reduce-scatter collective operations?

Accepted Answer

All-gather collects all shards on every rank; reduce-scatter reduces then splits results across ranks

Answer

All-gather reduces values; reduce-scatter only copies values

Answer

They are identical operations with different names

Answer

Reduce-scatter gathers all data on rank 0 only

Question 11

In Mixture-of-Experts (MoE) training, what communication pattern dominates and why?

Accepted Answer

All-to-all for token dispatch and combine

Answer

All-reduce of full dense gradients each step

Answer

Broadcast of expert weights from rank 0

Answer

Point-to-point pipeline sends only

Question 12

What is 3D parallelism in large-scale LLM training?

Accepted Answer

Combining data, tensor, and pipeline parallelism together

Answer

Training in 3 spatial dimensions of a tensor

Answer

Using 3 separate optimizers per model

Answer

Replicating the model exactly 3 times

Question 13

Why must BatchNorm be handled specially in data-parallel training, and what is the common solution?

Accepted Answer

Local BN stats are computed per-GPU; SyncBatchNorm all-reduces stats across GPUs

Answer

BN is unaffected by data parallelism and needs no change

Answer

BN must be disabled entirely in distributed training

Answer

SyncBatchNorm broadcasts stats only from rank 0

Question 14

In mixed-precision training, what is the purpose of dynamic loss scaling?

Accepted Answer

Prevents fp16 gradient underflow by scaling the loss up then unscaling gradients

Answer

Speeds up matrix multiplies by reducing precision

Answer

Compresses gradients for faster all-reduce

Answer

Balances load across GPUs

Question 15

What advantage does bf16 offer over fp16 for large-model training?

Accepted Answer

bf16 has fp32's exponent range, so it usually avoids loss scaling

Answer

bf16 has more mantissa bits than fp16

Answer

bf16 uses half the memory of fp16

Answer

bf16 is an integer format, fp16 is floating point

Question 16

What is the key difference between synchronous and asynchronous SGD in distributed training?

Accepted Answer

Sync waits for all workers (stragglers); async uses stale gradients without waiting

Answer

Sync uses stale gradients; async is always exact

Answer

They differ only in learning rate

Answer

Async guarantees identical replicas, sync does not

Question 17

In a parameter-server architecture, what are the roles of servers and workers, and what is a key drawback?

Accepted Answer

Servers hold/update params, workers compute gradients; servers become a bandwidth bottleneck

Answer

Workers hold params, servers compute gradients; no bottleneck exists

Answer

Both compute identical gradients redundantly

Answer

Servers only store data, workers store parameters

Question 18

What is sequence parallelism and how does it complement tensor parallelism in transformers?

Accepted Answer

Splits activations along the sequence dim to reduce TP's replicated activation memory

Answer

Splits the batch dimension like data parallelism

Answer

Runs different sequences on different epochs

Answer

Replicates the full sequence on every device

Question 19

Why does the 1F1B (one-forward-one-backward) pipeline schedule reduce peak memory versus GPipe's all-forward-then-all-backward?

Accepted Answer

1F1B interleaves backward early, bounding stored activations by pipeline depth not micro-batch count

Answer

1F1B uses fewer micro-batches so it is faster

Answer

1F1B eliminates the pipeline bubble entirely

Answer

GPipe uses more compute per micro-batch

Question 20

What does NCCL provide and why is its topology awareness important?

Accepted Answer

GPU collective primitives that exploit interconnect topology for bandwidth

Answer

A data loader that shards datasets across nodes

Answer

A model-parallel partitioner for transformers

Answer

A checkpointing format for optimizer state

Question 21

What is gradient bucketing in DDP and how does it improve performance?

Accepted Answer

Groups gradients into buckets and overlaps their all-reduce with backward compute

Answer

Compresses gradients into quantized buckets

Answer

Stores gradients in CPU memory buckets

Answer

Sorts gradients by magnitude before updating

Question 22

How does the LAMB optimizer enable very large batch training where plain SGD/Adam fail?

Accepted Answer

It applies a per-layer trust ratio (weight-norm/update-norm) for stable large-batch steps

Answer

It removes momentum to reduce memory

Answer

It uses fp8 gradients for speed

Answer

It synchronizes gradients less often than Adam

Question 23

What is the difference between weak scaling and strong scaling efficiency in distributed training?

Accepted Answer

Strong scaling fixes total work and adds GPUs; weak scaling grows work with GPUs

Answer

Weak scaling fixes total work; strong scaling grows work with GPUs

Answer

Both keep per-GPU batch constant

Answer

They refer to fp16 vs fp32 precision

Question 24

Why can naive checkpoint saving in large distributed jobs become a bottleneck, and what mitigates it?

Accepted Answer

Centralized full-state saves stall I/O; sharded parallel/async checkpointing fixes it

Answer

Checkpointing requires recomputation like activation checkpointing

Answer

Saving requires an all-to-all of all parameters

Answer

There is no way to reduce checkpoint cost

Question 25

What is the communication volume per training step for tensor parallelism of a transformer's MLP+attention block, in terms of all-reduce count?

Accepted Answer

About 4 all-reduces per layer (2 forward + 2 backward)

Answer

Exactly 1 all-reduce per entire model per step

Answer

No all-reduce; only point-to-point sends

Answer

1 all-gather per parameter

Question 26

How does gradient compression (e.g., quantization or top-k sparsification) help scaling, and what is its main risk?

Accepted Answer

Cuts communicated bits/elements but risks convergence harm without error feedback

Answer

Increases gradient precision to speed convergence

Answer

Eliminates the need for all-reduce entirely

Answer

Always improves accuracy with no downside

Question 27

In FSDP/ZeRO-3, why is prefetching of the next layer's parameters important?

Accepted Answer

It overlaps the next layer's parameter all-gather with current compute to avoid stalls

Answer

It compresses parameters before gathering

Answer

It avoids needing any all-gather

Answer

It loads data batches earlier

Question 28

What is CPU/NVMe offloading in frameworks like DeepSpeed ZeRO-Offload, and what is the trade-off?

Accepted Answer

Moving states to CPU/NVMe to fit larger models, at the cost of throughput

Answer

Moving compute to the CPU to improve speed

Answer

Caching activations on GPU to save memory

Answer

Compressing weights into NVMe with no speed cost

Question 29

When combining tensor, pipeline, and data parallelism, what is the standard rule for placing each across the hardware hierarchy?

Accepted Answer

Tensor within node, pipeline across nodes, data parallel outermost

Answer

Data parallel within node, tensor across nodes

Answer

Pipeline within node, tensor across all nodes randomly

Answer

All three must use identical group sizes on one GPU

Question 30

What is the role of a distributed sampler (e.g., DistributedSampler) in data-parallel training?

Accepted Answer

Gives each rank a disjoint shard of data per epoch via set_epoch reseeding

Answer

Replicates the full dataset to every rank identically

Answer

Compresses the dataset for faster loading

Answer

Synchronizes gradients across ranks

Test yourself