Loss functions explained quiz

Question 1

What is a loss function in machine learning, and what role does it play during training?

Accepted Answer

A function that measures prediction error and is minimized during training

Answer

A function that randomly initializes the model weights

Answer

A function that counts the number of layers in a network

Answer

A function that loads the dataset into memory

Question 2

What is the difference between a loss function and a cost function as the terms are commonly used?

Accepted Answer

Loss is for one example; cost is the average loss over the dataset

Answer

Loss is for the dataset; cost is for one example

Answer

They always refer to completely unrelated quantities

Answer

Cost is only used in classification, loss only in regression

Question 3

Write the formula for Mean Squared Error (MSE) and explain what each part represents.

Accepted Answer

(1/n) Σ (yᵢ − ŷᵢ)²

Answer

(1/n) Σ |yᵢ − ŷᵢ|

Answer

(1/n) Σ (yᵢ − ŷᵢ)

Answer

Σ yᵢ · ŷᵢ

Question 4

Why does MSE penalize large errors more heavily than small ones?

Accepted Answer

The squaring makes the penalty grow quadratically with error size

Answer

It multiplies large errors by a fixed constant of 10

Answer

It ignores small errors entirely

Answer

It uses absolute values which amplify large errors

Question 5

What is Mean Absolute Error (MAE), and how does its sensitivity to outliers compare to MSE?

Accepted Answer

Average of absolute errors; more robust to outliers than MSE

Answer

Average of squared errors; less robust than MSE

Answer

Sum of errors; equally sensitive as MSE

Answer

Product of errors; ignores outliers completely

Question 6

What is the Huber loss and why is it considered a compromise between MSE and MAE?

Accepted Answer

Quadratic for small errors, linear for large errors, controlled by δ

Answer

Always linear regardless of error size

Answer

Always quadratic regardless of error size

Answer

Random switching between MSE and MAE

Question 7

Why is Mean Squared Error generally a poor choice as the loss function for a classification problem with class labels?

Accepted Answer

It gives weak gradients with sigmoid and mishandles categorical labels

Answer

It is too computationally expensive for classification

Answer

It cannot be computed when there are more than two classes

Answer

It always outputs negative values for classification

Question 8

Write the binary cross-entropy (log loss) formula for a single example and describe its behavior.

Accepted Answer

−(y·log(p) + (1−y)·log(1−p))

Answer

(y − p)²

Answer

|y − p|

Answer

y·p + (1−y)·(1−p)

Question 9

In binary cross-entropy, what happens to the loss when the model predicts a probability very close to 0 for an example whose true label is 1?

Accepted Answer

The loss tends toward infinity (very large)

Answer

The loss becomes exactly zero

Answer

The loss becomes negative

Answer

The loss stays constant at one

Question 10

What is categorical cross-entropy, and how does it relate to a softmax output layer?

Accepted Answer

−Σ yᵢ·log(pᵢ), used with softmax over the classes

Answer

Σ (yᵢ − pᵢ)², used with a linear output

Answer

Average of absolute errors over classes

Answer

The maximum logit value across classes

Question 11

What does the softmax function do, and why is it commonly combined with cross-entropy loss?

Accepted Answer

Turns logits into a probability distribution summing to 1

Answer

Rounds logits to the nearest integer

Answer

Selects only the largest logit and discards the rest

Answer

Adds random noise to the logits

Question 12

Why is cross-entropy loss connected to the idea of information and 'surprise'?

Accepted Answer

−log(p) measures surprise; low surprise on true labels means low loss

Answer

It counts the number of bits in the input image

Answer

It measures the entropy of the model's weights

Answer

It is unrelated to information theory

Question 13

What is the relationship between minimizing cross-entropy and maximizing likelihood?

Accepted Answer

Minimizing cross-entropy equals maximizing the data likelihood

Answer

Minimizing cross-entropy minimizes the number of parameters

Answer

They are unrelated optimization goals

Answer

Cross-entropy maximizes the variance of predictions

Question 14

What is the hinge loss and in which classic algorithm is it used?

Accepted Answer

max(0, 1 − y·ŷ), used in Support Vector Machines

Answer

−log(ŷ), used in logistic regression

Answer

(y − ŷ)², used in linear regression

Answer

|y − ŷ|, used in decision trees

Question 15

Why must a loss function used with gradient descent generally be differentiable?

Accepted Answer

Gradient descent needs derivatives to update parameters

Answer

Differentiability makes the loss values smaller

Answer

Non-differentiable losses always overfit

Answer

It is only required for classification tasks

Question 16

Why can't classification accuracy be used directly as a loss function for gradient-based training?

Accepted Answer

Its gradient is zero almost everywhere, giving no learning signal

Answer

It is too slow to compute on large datasets

Answer

It can only be used for regression problems

Answer

It always equals the cross-entropy loss

Question 17

What is the difference between a loss function and an evaluation metric?

Accepted Answer

Loss is minimized in training; a metric judges quality and need not be differentiable

Answer

They are always identical quantities

Answer

A metric is minimized in training; loss only judges quality

Answer

Metrics must be differentiable but losses need not be

Question 18

What is a convex loss function, and why is convexity desirable for optimization?

Accepted Answer

It has one global minimum, so gradient descent reliably converges

Answer

It has many local minima that trap optimization

Answer

It is always non-differentiable

Answer

It can never be minimized by gradient descent

Question 19

In gradient descent, how is the loss function used to update the model's parameters?

Accepted Answer

Parameters move opposite to the loss gradient, scaled by the learning rate

Answer

Parameters move in the same direction as the gradient

Answer

Parameters are set to the gradient value directly

Answer

The loss replaces the parameters each step

Question 20

What is regularization, and how does it modify the loss function to reduce overfitting?

Accepted Answer

It adds a weight-size penalty to the loss to discourage complexity

Answer

It removes the loss function entirely

Answer

It increases the learning rate automatically

Answer

It deletes training examples that cause high loss

Question 21

How do L1 and L2 regularization penalties differ in their effect on a model's weights?

Accepted Answer

L1 drives some weights to exactly zero (sparse); L2 shrinks all weights smoothly

Answer

L2 produces sparsity; L1 keeps all weights large

Answer

Both always set every weight to zero

Answer

Neither has any effect on the weights

Question 22

What is Root Mean Squared Error (RMSE) and how does it relate to MSE?

Accepted Answer

The square root of MSE, expressed in the target's units

Answer

The same as MSE but doubled

Answer

MSE minus MAE

Answer

The square of MSE

Question 23

Why is it numerically risky to compute log(p) directly in cross-entropy, and how is this usually handled?

Accepted Answer

log(0) is undefined, so p is clipped or a stable log-softmax is used

Answer

Logarithms are too slow, so they are skipped

Answer

The probabilities are rounded to integers first

Answer

Cross-entropy never uses logarithms

Question 24

What is Kullback-Leibler (KL) divergence and how is it related to cross-entropy?

Accepted Answer

KL measures distribution mismatch; minimizing cross-entropy minimizes KL

Answer

KL is the square root of cross-entropy

Answer

KL can be negative, unlike cross-entropy

Answer

They are completely unrelated concepts

Question 25

How should the choice of loss function differ between a regression task and a classification task?

Accepted Answer

Regression uses MSE/MAE; classification uses cross-entropy/hinge

Answer

Both should always use MSE

Answer

Both should always use cross-entropy

Answer

The loss choice does not depend on the task

Question 26

What is the focal loss, and what problem in classification was it designed to address?

Accepted Answer

It down-weights easy examples to handle severe class imbalance

Answer

It removes all misclassified examples from training

Answer

It is a loss only for regression problems

Answer

It increases the loss from easy examples

Question 27

How can class weighting be incorporated into a loss function to deal with imbalanced datasets?

Accepted Answer

Give larger loss weights to rare classes so they get more attention

Answer

Give larger loss weights to common classes

Answer

Remove all rare-class examples

Answer

Set every class weight to exactly one

Question 28

What is label smoothing, and why is it applied to cross-entropy loss?

Accepted Answer

Softening one-hot targets to reduce overconfidence and improve generalization

Answer

Deleting incorrect labels from the dataset

Answer

Converting probabilities back to logits

Answer

Increasing the learning rate over time

Question 29

For a single example with per-example loss (ŷ - y)², what is the gradient of the loss with respect to the prediction ŷ?

Accepted Answer

2(ŷ − y)

Answer

(ŷ − y)²

Answer

(ŷ − y) / 2

Answer

1 / (ŷ − y)

Question 30

Why does combining softmax with cross-entropy produce an especially simple and stable gradient?

Accepted Answer

The gradient simplifies to (predicted probability − true label)

Answer

The gradient becomes exactly zero everywhere

Answer

The gradient requires computing a matrix inverse

Answer

The gradient is always equal to one

Test yourself