Knowledge distillation quiz

Question 1

In Hinton et al. (2015), what is the primary role of the temperature parameter T in knowledge distillation?

Accepted Answer

It softens the teacher's output distribution to expose inter-class similarities (dark knowledge)

Answer

It regularizes the student by adding Gaussian noise to its logits

Answer

It scales the learning rate during the distillation phase

Answer

It controls the ratio of CE loss to KL loss in the combined objective

Question 2

Write the standard Hinton KD training loss combining hard-label cross-entropy and soft-target KL divergence.

Accepted Answer

L = (1-α)·CE(hard) + α·T²·KL(soft_teacher ‖ soft_student)

Answer

L = α·CE(hard) + (1-α)·MSE(student_logits, teacher_logits)

Answer

L = (1-α)·CE(hard) + α·KL(soft_teacher ‖ soft_student) without T² scaling

Answer

L = CE(hard) + α·BCE(soft_teacher, soft_student)

Question 3

What is 'dark knowledge' in the context of knowledge distillation?

Accepted Answer

Relative probabilities assigned to non-target classes by the teacher, revealing inter-class similarity

Answer

The compressed weight matrices in the distilled student model

Answer

Negative log-likelihoods computed on unlabeled auxiliary data

Answer

Knowledge stored in the teacher's batch-normalization statistics

Question 4

What distinguishes response-based, feature-based, and relation-based knowledge distillation?

Accepted Answer

Response-based uses logits; feature-based matches intermediate activations; relation-based transfers structural relationships between samples/features

Answer

They differ only in the optimizer used: Adam, SGD, and RMSprop respectively

Answer

Response-based uses MSE; feature-based uses KL; relation-based uses cross-entropy

Answer

They refer to distillation at inference time, train time, and post-training respectively

Question 5

FitNets (Romero et al., 2015) introduced hint-based training. What problem does it solve and how?

Accepted Answer

It solves the gradient vanishing problem in deep thin students by using intermediate hint layers for staged pre-training

Answer

It solves overfitting by freezing the teacher's weights during student training

Answer

It solves data imbalance by reweighting samples using teacher confidence scores

Answer

It solves capacity mismatch by expanding the student width to match the teacher

Question 6

What is the 'capacity gap' problem in knowledge distillation, and what strategies mitigate it?

Accepted Answer

When a very small student cannot fit complex teacher soft targets, degrading below direct training; mitigated by intermediate teaching assistants or curriculum distillation

Answer

When the teacher's vocabulary size exceeds the student's embedding dimension

Answer

When the student overfits the teacher's soft targets due to too large a learning rate

Answer

When GPU memory prevents loading both teacher and student simultaneously

Question 7

Describe the Attention Transfer (AT) method by Zagoruyko & Komodakis (2017).

Accepted Answer

It transfers normalized channel-aggregated spatial activation maps from teacher to student via L2 loss

Answer

It fine-tunes the student's self-attention heads using teacher cross-attention outputs

Answer

It replaces the student's attention layers with frozen copies of the teacher's

Answer

It uses the teacher's gradient saliency maps as pseudo-labels for the student

Question 8

How does Relational Knowledge Distillation (RKD, Park et al. 2019) differ from instance-level feature matching?

Accepted Answer

RKD transfers pairwise/triplet relational structures between samples rather than individual embedding vectors, preserving global embedding geometry

Answer

RKD matches individual sample embeddings while instance-level matching uses triplet loss

Answer

RKD uses teacher logits while instance-level distillation uses intermediate activations

Answer

RKD is only applicable to contrastive self-supervised pretraining setups

Question 9

What is Born-Again Networks (BAN), and why can a student trained on its own teacher outperform the teacher?

Accepted Answer

Soft labels regularize better than hard labels, and ensembling multiple generations amplifies this; the student can exceed the teacher without additional capacity

Answer

The student trains on more data than the teacher by using data augmentation

Answer

The student uses a larger architecture while inheriting teacher weights as initialization

Answer

The student benefits from gradient checkpointing not available to the original teacher

Question 10

Explain Deep Mutual Learning (DML, Zhang et al. 2018) and how it differs from traditional KD.

Accepted Answer

DML trains a cohort of peer networks that teach each other online with no fixed pretrained teacher

Answer

DML uses a pretrained large teacher to supervise multiple student networks in parallel

Answer

DML distills a single teacher into multiple task-specific students via multi-task learning

Answer

DML applies distillation only during the final fine-tuning phase using exponential moving average teachers

Question 11

How does DistilBERT achieve its compression compared to BERT-base, and what distillation signals does it use?

Accepted Answer

DistilBERT uses 6 layers initialized from BERT's layers and trains with soft MLM logits, cosine hidden-state similarity, and hard MLM loss combined

Answer

DistilBERT uses 8 layers, weight sharing, and only hard MLM labels for compression

Answer

DistilBERT applies post-training quantization to BERT-base without any knowledge transfer

Answer

DistilBERT removes attention heads and uses only the teacher's embedding layer via L2 regression

Question 12

What is data-free knowledge distillation (DFKD) and what generator-based approach is commonly used?

Accepted Answer

DFKD trains a generator to produce synthetic pseudo-data that activates the teacher, then trains the student on those samples with teacher soft labels

Answer

DFKD uses the student's own training data augmented with Mixup to substitute for original data

Answer

DFKD distills only the teacher's embedding layer weights without any forward passes

Answer

DFKD applies pruning masks derived from teacher weight magnitudes to the student

Question 13

What is Noisy Student Training and how does it relate to knowledge distillation?

Accepted Answer

Noisy Student is an iterative self-training KD where a noised student trains on teacher pseudo-labels on unlabeled data and becomes the next teacher

Answer

Noisy Student distills only the teacher's last layer using MSE on logits with input Gaussian noise

Answer

Noisy Student uses a smaller student than the teacher and knowledge transfer via attention maps

Answer

Noisy Student applies label smoothing to hard teacher labels to simulate soft targets

Question 14

How does knowledge distillation relate to label smoothing mathematically?

Accepted Answer

Label smoothing is a degenerate KD where the implicit 'teacher' is a uniform distribution, providing entropy but no inter-class structure

Answer

Label smoothing is exactly equivalent to KD with a teacher trained on the same dataset

Answer

Label smoothing is only applicable to multi-label classification, while KD applies to single-label tasks

Answer

Label smoothing replaces the KL divergence in KD with cross-entropy, making them mathematically identical

Question 15

In knowledge distillation for object detection, why is it harder than for classification, and how does Mimicking (Li et al. 2017) address it?

Accepted Answer

Detection requires matching spatially structured feature maps and must handle foreground/background imbalance; Mimicking uses feature-map L2 loss focused on foreground regions

Answer

Detection distillation is easier because bounding boxes serve as additional supervision signals

Answer

Detection distillation only transfers the classification head of the teacher, ignoring regression heads

Answer

Mimicking trains the student from scratch using teacher NMS outputs as hard pseudo-labels

Question 16

What is the Progressive Knowledge Distillation strategy, and when is it preferred over single-step distillation?

Accepted Answer

Progressive KD chains intermediate-capacity teachers, reducing the capacity gap at each step, preferred when direct student is too small for single-step distillation

Answer

Progressive KD trains the student on progressively harder augmentations from the teacher

Answer

Progressive KD freezes earlier student layers first and unfreezes them progressively during training

Answer

Progressive KD applies KD only during the last few epochs of standard training

Question 17

What is 'online distillation' versus 'offline distillation', and give an example of each?

Accepted Answer

Offline distillation uses a pre-trained fixed teacher; online distillation trains teacher and student simultaneously

Answer

Offline distillation uses unlabeled data; online distillation uses labeled data

Answer

Offline distillation matches logits; online distillation matches intermediate features

Answer

Offline distillation is task-specific; online distillation is task-agnostic

Question 18

How does Contrastive Representation Distillation (CRD, Tian et al. 2020) transfer knowledge?

Accepted Answer

CRD formulates distillation as contrastive learning using InfoNCE to maximize mutual information between teacher and student representations

Answer

CRD uses a triplet loss between teacher logits, student logits, and a negative teacher logit

Answer

CRD applies contrastive loss only to the final classification layer outputs of teacher and student

Answer

CRD uses teacher-generated adversarial examples as contrastive negatives for the student

Question 19

Why is KL(teacher || student) used in KD rather than KL(student || teacher), and what are the practical implications?

Accepted Answer

KL(teacher || student) is used because it is mean-seeking, forcing the student to cover all teacher probability modes including dark knowledge

Answer

KL(student || teacher) is used because it is computationally cheaper to compute

Answer

Both directions are equivalent under the softmax temperature parameterization used in KD

Answer

KL(teacher || student) is preferred because it is upper-bounded by the cross-entropy loss

Question 20

What is self-distillation within a single network, and how does Be Your Own Teacher (BYOT) implement it?

Accepted Answer

Self-distillation uses the network's own deep classifier outputs as soft teachers for intermediate auxiliary classifiers within the same model

Answer

Self-distillation reuses the same network weights for both teacher and student by using dropout masks

Answer

Self-distillation trains a smaller copy of the model on the original model's gradients

Answer

Self-distillation applies EMA of model weights as a teacher following the BYOL framework

Question 21

How is knowledge distillation applied in Neural Architecture Search (NAS) to reduce search cost?

Accepted Answer

In OFA-style NAS, a supernet is trained once and sub-networks are distilled from it via soft labels, decoupling training cost from deployment architecture variety

Answer

NAS uses KD to transfer architecture search policies between GPU types

Answer

KD in NAS transfers hyperparameter sensitivity from larger models to architecture controllers

Answer

NAS uses KD to share gradient information between candidate architecture cells during search

Question 22

Explain how knowledge distillation is used to train task-specific smaller models from large language models (LLMs).

Accepted Answer

A small student LM trains on soft token-level distributions or reasoning traces from a large teacher LLM, retaining most task accuracy at lower inference cost

Answer

LLM distillation only transfers the final linear head weights from the large model to the student

Answer

LLM distillation requires identical tokenizers and cannot transfer across tokenizer vocabularies

Answer

LLM distillation is impossible without access to the teacher model's weight gradients

Question 23

What is 'sequence-level knowledge distillation' (Kim & Rush 2016) and how does it differ from word-level distillation?

Accepted Answer

Sequence-level KD distills full generated sequences from the teacher as hard targets, while word-level KD matches per-token probability distributions with teacher-forcing

Answer

Sequence-level KD uses BLEU score as the distillation objective instead of cross-entropy

Answer

Sequence-level KD is only applicable to autoregressive models with greedy decoding

Answer

Word-level KD generates sequences via beam search; sequence-level KD uses step-by-step teacher distributions

Question 24

How is knowledge distillation applied to Graph Neural Networks (GNNs), and what unique challenge arises?

Accepted Answer

GNN distillation faces irregular structure (no fixed spatial order), variable graph sizes, and differing receptive fields; methods transfer pairwise node similarities or local topology

Answer

GNN distillation is straightforward since node features have fixed dimension across teacher and student

Answer

GNN distillation uses attention rollout to align teacher and student message-passing layers

Answer

GNN distillation requires the teacher and student to share identical adjacency matrices during training

Question 25

What is Probabilistic Knowledge Transfer (PKT) and how does it extend feature-level distillation?

Accepted Answer

PKT models the teacher feature space as a kernel similarity distribution and trains the student to match it via KL divergence, preserving relational structure across dimensional differences

Answer

PKT transfers teacher probability estimates from a Bayesian teacher to a deterministic student

Answer

PKT applies probabilistic dropout to teacher activations before transferring to the student

Answer

PKT is a variant of variational autoencoders used to compress teacher weight distributions

Question 26

In federated learning, how does knowledge distillation enable model aggregation without sharing raw model weights?

Accepted Answer

Federated KD (FedDF) aggregates client soft predictions on public data to distill a global model, sharing no raw weights and supporting heterogeneous architectures

Answer

Federated KD shares client model weights only after encrypting them with homomorphic encryption

Answer

Federated KD requires all clients to use identical model architectures for prediction aggregation

Answer

Federated KD averages client model weights then fine-tunes the averaged model using hard labels

Question 27

What is 'activation boundary distillation' (Heo et al. 2019) and what knowledge does it target?

Accepted Answer

Activation boundary distillation matches which ReLU neurons fire (binary activation signs) between teacher and student, capturing feature-space partition structure

Answer

Activation boundary distillation matches the gradients of teacher activations with respect to inputs

Answer

Activation boundary distillation uses the teacher's decision boundary as a hard class separator during student training

Answer

Activation boundary distillation freezes teacher neurons that fire on more than 50% of training samples

Question 28

How does knowledge distillation interact with model quantization, and what is quantization-aware distillation?

Accepted Answer

QAD trains a quantized student to match a full-precision teacher during QAT, using soft targets to recover accuracy lost from quantization noise

Answer

QAD applies post-training quantization to the teacher model before transferring its weights to the student

Answer

QAD uses the teacher to select which weights to quantize based on gradient sensitivity

Answer

QAD is incompatible with INT8 quantization and only applies to mixed-precision FP16 training

Question 29

What is 'dataset distillation' (Wang et al. 2018), and how does it differ from model knowledge distillation?

Accepted Answer

Dataset distillation meta-learns synthetic images via bi-level optimization so models trained only on them approximate full-dataset performance; it compresses data, not models

Answer

Dataset distillation applies knowledge distillation loss to compress dataset labels into soft pseudo-labels

Answer

Dataset distillation is another name for coreset selection where representative real samples are chosen

Answer

Dataset distillation uses a pretrained teacher to generate augmented samples for student training

Question 30

What is task-agnostic versus task-specific knowledge distillation in the context of pre-trained language models?

Accepted Answer

Task-agnostic KD distills the general pre-trained model for any downstream use; task-specific KD distills after fine-tuning for a single task, achieving higher per-task accuracy

Answer

Task-agnostic KD trains the student on task data without soft labels; task-specific KD uses soft labels from the fine-tuned teacher

Answer

Task-agnostic KD is only applicable to generative models; task-specific KD applies to discriminative models

Answer

Task-specific KD requires larger student models than task-agnostic KD to maintain accuracy

Test yourself