PyTorch quiz

Question 1

What is the difference between torch.Tensor and torch.tensor (lowercase)?

Accepted Answer

torch.tensor is a factory that infers dtype; torch.Tensor is the class constructor

Answer

torch.Tensor infers dtype from data; torch.tensor uses float32 always

Answer

They are identical aliases for the same operation

Answer

torch.Tensor copies data; torch.tensor shares memory

Question 2

What does torch.no_grad() do and how does it differ from model.eval()?

Accepted Answer

torch.no_grad() disables gradients; model.eval() changes layer behavior like Dropout/BN

Answer

They are identical and interchangeable

Answer

model.eval() disables gradients; torch.no_grad() changes Dropout behavior

Answer

torch.no_grad() only affects BatchNorm; model.eval() disables autograd

Question 3

How does autocast work in PyTorch's Automatic Mixed Precision (AMP)?

Accepted Answer

autocast selectively casts ops to lower precision where safe, keeping sensitive ops in float32

Answer

autocast converts all ops to float16 unconditionally

Answer

autocast only affects backward pass gradient computation

Answer

autocast is only for CPU and has no GPU effect

Question 4

What is gradient clipping and why is it used in training?

Accepted Answer

It prevents exploding gradients by capping gradient magnitude before the optimizer step

Answer

It adds noise to gradients to improve generalization

Answer

It skips gradient updates when loss is too large

Answer

It normalizes the loss before backward pass

Question 5

What is TorchScript and what are its two main modes?

Accepted Answer

torch.jit.trace records ops on example inputs; torch.jit.script compiles Python with type inference

Answer

torch.jit.trace handles all control flow; torch.jit.script only handles linear graphs

Answer

Both modes produce identical IR regardless of model structure

Answer

TorchScript only works on CPU deployments

Question 6

What is the difference between .detach() and .data in PyTorch tensors?

Accepted Answer

.detach() is safe and detaches from graph; .data bypasses autograd checks and is risky

Answer

.detach() copies data; .data shares memory with the original

Answer

.data is the recommended modern API; .detach() is deprecated

Answer

Both are identical in PyTorch 2.x

Question 7

What is gradient accumulation and when is it used?

Accepted Answer

It accumulates gradients over multiple micro-batches to simulate a larger batch size

Answer

It averages gradients across GPUs in distributed training

Answer

It clips gradients to prevent overflow

Answer

It reduces the learning rate proportionally to batch size

Question 8

What is the difference between nn.Sequential and nn.ModuleList?

Accepted Answer

nn.Sequential has automatic sequential forward; nn.ModuleList requires manual forward

Answer

nn.Sequential requires manual forward; nn.ModuleList has automatic sequential forward

Answer

Both require manual forward implementations

Answer

nn.ModuleList is deprecated in PyTorch 2.x

Question 9

What is torch.vmap and how does it enable batched computation?

Accepted Answer

vmap vectorizes a function over batch dimensions, enabling per-sample computations without loops

Answer

vmap is a distributed training utility for multi-GPU batching

Answer

vmap is equivalent to DataParallel for model parallelism

Answer

vmap only works with RNN architectures

Question 10

What is the difference between torch.save/torch.load and state_dict for model persistence?

Accepted Answer

state_dict is recommended — it saves parameters without binding to class definitions

Answer

Saving the whole model is recommended for robustness

Answer

Both approaches produce identical files

Answer

state_dict cannot be loaded if model architecture changes even slightly

Question 11

Explain the concept of dynamic computation graphs in PyTorch vs static graphs in TensorFlow 1.x.

Accepted Answer

PyTorch builds graphs dynamically per forward pass; TF 1.x used pre-compiled static graphs

Answer

PyTorch uses static graphs; TensorFlow 1.x uses dynamic graphs

Answer

Both frameworks use identical graph construction approaches

Answer

Dynamic graphs prevent any form of JIT optimization in PyTorch

Question 12

What is the purpose of weight_decay in Adam optimizer and how does it differ from L2 regularization?

Accepted Answer

AdamW decouples weight decay from adaptive gradient scaling; standard Adam mixes them

Answer

weight_decay in Adam and AdamW are mathematically identical

Answer

L2 regularization is stronger than weight decay at all learning rates

Answer

weight_decay only affects the bias terms in linear layers

Question 13

What is einsum notation and what are its advantages for tensor operations?

Accepted Answer

einsum expresses tensor contractions concisely using index notation and dispatches to optimized routines

Answer

einsum is slower than explicit matrix operations due to parsing overhead

Answer

einsum only supports 2D matrix operations

Answer

einsum is a deprecated API replaced by torch.tensordot

Question 14

How does PyTorch handle integer vs floating-point tensor operations in autograd?

Accepted Answer

Autograd only tracks floating-point tensors; integer tensors are not differentiable

Answer

Autograd tracks both integer and float tensors equally

Answer

Only float64 tensors are tracked by autograd

Answer

Integer tensors are automatically cast to float by autograd

Question 15

What is the difference between torch.nn.functional and torch.nn modules?

Accepted Answer

nn.functional provides stateless functions; nn modules store parameters and state

Answer

nn.functional modules store parameters; nn modules are stateless

Answer

They produce different numerical results for the same operation

Answer

nn.functional is deprecated and will be removed in PyTorch 3.0

Question 16

What is torch.nn.SyncBatchNorm and when is it used?

Accepted Answer

SyncBatchNorm synchronizes batch statistics across all distributed processes

Answer

SyncBatchNorm speeds up training by skipping normalization on small batches

Answer

SyncBatchNorm is identical to LayerNorm for transformer models

Answer

SyncBatchNorm only works with Gloo communication backend

Question 17

What is Tensor Parallelism at the PyTorch level (torch.distributed.tensor)?

Accepted Answer

DTensor provides a sharded tensor abstraction enabling tensor parallelism across GPUs

Answer

DTensor replaces DDP for all distributed training scenarios

Answer

DTensor only supports row-wise sharding of linear layers

Answer

DTensor requires a separate MPI installation and is not native to PyTorch

Question 18

What is the difference between torch.stack and torch.cat?

Accepted Answer

torch.cat concatenates along existing dim; torch.stack creates a new dimension

Answer

torch.stack concatenates along existing dim; torch.cat creates a new dim

Answer

Both operations produce identical results for 1D tensors

Answer

torch.stack only works on 3D or higher tensors

Question 19

What is a named tensor and how does it help avoid shape bugs?

Accepted Answer

Named tensors attach dimension names to help catch shape errors but are experimental

Answer

Named tensors are required for all tensor operations in PyTorch 2.x

Answer

Named tensors automatically reshape tensors to a canonical format

Answer

Named tensors are only supported for 1D and 2D tensors

Question 20

What is the role of torch.distributed.init_process_group in multi-GPU training?

Accepted Answer

It initializes the communication backend and process group required for all distributed ops

Answer

It is only needed for CPU-based distributed training

Answer

It automatically sets the learning rate based on world_size

Answer

It replaces DataLoader in distributed settings

Question 21

What is Channels Last memory format and when is it beneficial?

Accepted Answer

Channels Last (NHWC) stores channel values contiguously per spatial location, improving cache use

Answer

Channels Last is the default memory format in PyTorch

Answer

Channels Last is only supported for 1D convolutions

Answer

Channels Last format is deprecated in favor of Channels First in PyTorch 2.x

Question 22

What is the purpose of torch.utils.data.Sampler and give two concrete examples.

Accepted Answer

Sampler defines index selection strategy; examples include DistributedSampler and WeightedRandomSampler

Answer

Sampler controls the loss function during DataLoader iteration

Answer

Sampler only supports sequential or random access patterns

Answer

Sampler is deprecated; use batch_size parameter in DataLoader instead

Question 23

What does zero_grad(set_to_none=True) do and why is it faster?

Accepted Answer

set_to_none=True sets grad to None instead of zeros, avoiding a memset and improving speed

Answer

set_to_none=True fills gradients with zeros, which is faster than None assignment

Answer

set_to_none=True is only used in distributed training contexts

Answer

set_to_none=True was deprecated in PyTorch 2.0 for safety reasons

Question 24

How do you perform a numerically stable log-sum-exp in PyTorch?

Accepted Answer

torch.logsumexp subtracts the max before exponentiation for numerical stability

Answer

torch.logsumexp is numerically unstable and should be avoided

Answer

torch.logsumexp only works on 1D tensors

Answer

torch.logsumexp is equivalent to torch.sum(x.exp()).log() with no stability fix

Question 25

What are torch.nn.ParameterList and torch.nn.ParameterDict used for?

Accepted Answer

ParameterList/Dict register Parameter collections so they appear in parameters() and state_dict()

Answer

ParameterList is identical to a Python list for PyTorch purposes

Answer

They are deprecated in favor of ModuleList in PyTorch 2.x

Answer

They only work with float64 precision parameters

Test yourself