Activation functions quiz

Question 1

What is an activation function in a neural network?

Accepted Answer

A function applied to a neuron's weighted input to add non-linearity

Answer

A function that randomly drops neurons during training

Answer

A function that initializes the network weights

Answer

A function that measures prediction error

Question 2

Why do neural networks need non-linear activation functions?

Accepted Answer

So the network can model complex non-linear relationships

Answer

To make the network train faster

Answer

To reduce the number of parameters

Answer

To prevent the weights from being negative

Question 3

What is the output range of the sigmoid function?

Accepted Answer

(0, 1)

Answer

(-1, 1)

Answer

(0, infinity)

Answer

(-infinity, infinity)

Question 4

Write the mathematical formula for the sigmoid activation function.

Accepted Answer

sigma(x) = 1 / (1 + e^(-x))

Answer

sigma(x) = max(0, x)

Answer

sigma(x) = e^x / (1 + e^x)^2

Answer

sigma(x) = (e^x - e^(-x)) / (e^x + e^(-x))

Question 5

What is the maximum value of the derivative of the sigmoid function, and where does it occur?

Accepted Answer

0.25 at x = 0

Answer

1.0 at x = 1

Answer

0.5 at x = 0

Answer

0.25 at x = 1

Question 6

What is the vanishing gradient problem and how is it related to activation functions?

Accepted Answer

Gradients shrink toward zero, stalling learning in early layers

Answer

Gradients become too large and the network diverges

Answer

The loss function has no minimum

Answer

Weights are initialized to zero

Question 7

What is the output range of the tanh (hyperbolic tangent) activation function?

Accepted Answer

(-1, 1)

Answer

(0, 1)

Answer

(0, infinity)

Answer

(-infinity, infinity)

Question 8

Why is tanh often preferred over sigmoid for hidden layers?

Accepted Answer

Tanh is zero-centered, giving more balanced gradients

Answer

Tanh never saturates

Answer

Tanh always outputs positive values

Answer

Tanh has no derivative

Question 9

What is the mathematical relationship between tanh and sigmoid?

Accepted Answer

tanh(x) = 2*sigmoid(2x) - 1

Answer

tanh(x) = sigmoid(x) - 1

Answer

tanh(x) = sigmoid(x)^2

Answer

tanh(x) = 1 - sigmoid(x)

Question 10

Define the ReLU (Rectified Linear Unit) activation function.

Accepted Answer

f(x) = max(0, x)

Answer

f(x) = 1 / (1 + e^(-x))

Answer

f(x) = min(0, x)

Answer

f(x) = x^2

Question 11

What is the output range of the ReLU activation function?

Accepted Answer

[0, infinity)

Answer

(-1, 1)

Answer

(0, 1)

Answer

(-infinity, infinity)

Question 12

Why is ReLU computationally cheaper than sigmoid or tanh?

Accepted Answer

It only needs a comparison, avoiding expensive exponentials

Answer

It uses exponentials which are fast

Answer

It has more parameters to optimize

Answer

It requires matrix inversion

Question 13

What is the 'dying ReLU' problem?

Accepted Answer

Neurons get stuck outputting 0 with zero gradient and stop learning

Answer

ReLU outputs explode to infinity

Answer

ReLU forgets previously learned weights

Answer

ReLU outputs become negative

Question 14

What is the derivative of ReLU for positive and negative inputs?

Accepted Answer

1 for x>0 and 0 for x<0

Answer

0 for x>0 and 1 for x<0

Answer

Always 1

Answer

x for x>0 and 0 otherwise

Question 15

How does Leaky ReLU differ from standard ReLU?

Accepted Answer

It allows a small negative slope instead of flat zero

Answer

It squashes outputs into (0,1)

Answer

It removes all negative inputs entirely

Answer

It is identical to sigmoid

Question 16

What is the purpose of the softmax activation function?

Accepted Answer

To turn raw scores into a probability distribution over classes

Answer

To normalize inputs before the first layer

Answer

To add regularization to the loss

Answer

To compute the gradient of the loss

Question 17

What key property do the outputs of softmax always satisfy?

Accepted Answer

They sum to 1 and are all non-negative

Answer

They are all between -1 and 1

Answer

They sum to 0

Answer

They are all equal

Question 18

Which activation function is typically used in the output layer for binary classification?

Accepted Answer

Sigmoid

Answer

ReLU

Answer

Tanh

Answer

Softmax

Question 19

Why is softmax, not sigmoid, generally used for multi-class (single-label) classification output?

Accepted Answer

Softmax makes class probabilities compete and sum to 1

Answer

Softmax is faster than sigmoid

Answer

Sigmoid cannot output probabilities

Answer

Softmax has fewer parameters

Question 20

What would happen if a deep network used only linear (identity) activation functions in all layers?

Accepted Answer

It would become equivalent to a single linear model

Answer

It would learn faster but the same way

Answer

It would overfit any dataset perfectly

Answer

It would output only zeros

Question 21

What is the step (Heaviside) activation function used in the classic perceptron?

Accepted Answer

Outputs 1 above a threshold and 0 below it

Answer

Outputs the input value scaled by 2

Answer

Outputs a probability between 0 and 1

Answer

Outputs the maximum of inputs

Question 22

What does it mean for an activation function to 'saturate'?

Accepted Answer

The output flattens and the gradient approaches zero

Answer

The output grows without bound

Answer

The function becomes linear

Answer

The output oscillates rapidly

Question 23

What is the Swish (SiLU) activation function?

Accepted Answer

f(x) = x * sigmoid(x)

Answer

f(x) = max(0, x)

Answer

f(x) = tanh(x) + 1

Answer

f(x) = x^2

Question 24

What is the main advantage of ELU (Exponential Linear Unit) over standard ReLU?

Accepted Answer

It allows smooth negative values, easing the dying-neuron issue

Answer

It is always faster to compute than ReLU

Answer

It outputs probabilities

Answer

It has no negative region at all

Question 25

In a typical deep network for image classification, which activation is most common in hidden layers?

Accepted Answer

ReLU

Answer

Sigmoid

Answer

Softmax

Answer

Step function

Question 26

How do activation functions affect backpropagation?

Accepted Answer

Their derivatives are multiplied into the gradient via the chain rule

Answer

They are ignored during backpropagation

Answer

They only affect the forward pass

Answer

They replace the loss function

Question 27

Compute the sigmoid of 0. What is the result?

Accepted Answer

0.5

Answer

0

Answer

1

Answer

Undefined

Question 28

Apply ReLU to the input vector [-3, 0, 2, -1, 5]. What is the output?

Accepted Answer

[0, 0, 2, 0, 5]

Answer

[-3, 0, 2, -1, 5]

Answer

[3, 0, 2, 1, 5]

Answer

[0, 0, 1, 0, 1]

Question 29

Why is numerical stability a concern when implementing softmax, and how is it handled?

Accepted Answer

Subtract the maximum logit before exponentiating

Answer

Add 1 to every input

Answer

Divide all inputs by their sum first

Answer

Use only positive inputs

Question 30

What is the Parametric ReLU (PReLU)?

Accepted Answer

A ReLU whose negative slope is learned during training

Answer

A ReLU with a fixed slope of 0.01 for negatives

Answer

A ReLU that outputs probabilities

Answer

A ReLU applied only to the output layer

Test yourself