Entropy and information gain quiz

Question 1

What does entropy measure in information theory?

Accepted Answer

The average uncertainty or unpredictability of outcomes

Answer

The total number of data points in a dataset

Answer

The accuracy of a trained model

Answer

The correlation between two features

Question 2

For a binary classification with class probabilities p and (1-p), what is the formula for Shannon entropy (in bits)?

Accepted Answer

H = -p·log2(p) - (1-p)·log2(1-p)

Answer

H = p² + (1-p)²

Answer

H = p·log2(p) + (1-p)·log2(1-p)

Answer

H = -p·(1-p)

Question 3

What is the maximum entropy of a binary variable, and when does it occur?

Accepted Answer

1 bit, when p = 0.5

Answer

2 bits, when p = 0.25

Answer

0 bits, when p = 0.5

Answer

1 bit, when p = 1.0

Question 4

What is the entropy of a perfectly pure node where all examples belong to a single class?

Accepted Answer

0

Answer

1

Answer

0.5

Answer

Equal to the number of classes

Question 5

What is information gain in the context of decision trees?

Accepted Answer

The reduction in entropy after splitting on a feature

Answer

The total entropy of the dataset

Answer

The number of branches created by a split

Answer

The increase in dataset size after a split

Question 6

How is information gain calculated mathematically for a split?

Accepted Answer

IG = H(parent) - weighted sum of H(children)

Answer

IG = H(parent) + sum of H(children)

Answer

IG = sum of H(children) - H(parent)

Answer

IG = H(parent) × H(children)

Question 7

Why does a decision tree algorithm prefer the feature with the highest information gain?

Accepted Answer

It produces the purest child nodes and reduces uncertainty most

Answer

It always creates the fewest branches

Answer

It guarantees zero training error

Answer

It minimizes the number of features used

Question 8

What is the Gini impurity, and how does it differ from entropy?

Accepted Answer

Gini = 1 - Σ p_i², cheaper since it avoids logarithms

Answer

Gini is identical to entropy in every way

Answer

Gini uses base-2 logarithms while entropy does not

Answer

Gini can be negative while entropy cannot

Question 9

Compute the entropy of a node with 3 positive and 3 negative examples.

Accepted Answer

1 bit

Answer

0 bits

Answer

0.5 bits

Answer

2 bits

Question 10

What is the general formula for Shannon entropy of a discrete variable with k classes?

Accepted Answer

H = -Σ p_i·log2(p_i)

Answer

H = Σ p_i²

Answer

H = -Σ log2(p_i)

Answer

H = Σ p_i·log2(1/p_i²)

Question 11

What is the maximum entropy of a variable with k equally likely outcomes?

Accepted Answer

log2(k)

Answer

k

Answer

1/k

Answer

k²

Question 12

In the ID3 decision tree algorithm, which splitting criterion is used?

Accepted Answer

Information gain

Answer

Mean squared error

Answer

Gini gain only

Answer

Variance reduction

Question 13

What problem does information gain have with high-cardinality features, and what fixes it?

Accepted Answer

Bias toward many-valued features; fixed by gain ratio

Answer

Bias toward binary features; fixed by Gini

Answer

It ignores rare classes; fixed by oversampling

Answer

It overfits numeric features; fixed by scaling

Question 14

How is the gain ratio defined?

Accepted Answer

Information Gain divided by Split Information

Answer

Information Gain multiplied by entropy

Answer

Split Information divided by Information Gain

Answer

Information Gain plus Gini impurity

Question 15

Why is the logarithm base 2 commonly used in entropy calculations?

Accepted Answer

So entropy is measured in bits

Answer

Because base 2 is faster to compute always

Answer

Because it makes entropy always an integer

Answer

Because probabilities are always powers of 2

Question 16

A dataset has 9 'Yes' and 5 'No' labels. What is its entropy (approximately)?

Accepted Answer

≈ 0.940 bits

Answer

≈ 0.500 bits

Answer

≈ 1.000 bits

Answer

≈ 0.643 bits

Question 17

What does it mean if a split produces an information gain of 0?

Accepted Answer

The split does not reduce uncertainty at all

Answer

The split perfectly separates the classes

Answer

The dataset is empty

Answer

The feature has the highest predictive power

Question 18

What is conditional entropy H(Y|X)?

Accepted Answer

The remaining uncertainty in Y after knowing X

Answer

The total uncertainty in X and Y combined

Answer

The uncertainty in X ignoring Y

Answer

The probability that X equals Y

Question 19

How is information gain related to mutual information?

Accepted Answer

Information gain equals the mutual information I(X;Y)

Answer

They are unrelated concepts

Answer

Information gain is the square of mutual information

Answer

Mutual information is always larger than information gain

Question 20

Can entropy ever be negative?

Accepted Answer

No, discrete entropy is always ≥ 0

Answer

Yes, when probabilities are small

Answer

Yes, for balanced datasets

Answer

Only when using natural logarithms

Question 21

In scikit-learn's DecisionTreeClassifier, which parameter selects entropy-based splitting?

Accepted Answer

criterion='entropy'

Answer

split='entropy'

Answer

impurity='entropy'

Answer

metric='entropy'

Question 22

A node splits into two children: one pure (entropy 0) with 4 samples and one with entropy 1 with 4 samples. If the parent entropy is 1, what is the information gain?

Accepted Answer

0.5 bits

Answer

1.0 bits

Answer

0 bits

Answer

0.25 bits

Question 23

What is the entropy of a fair six-sided die roll?

Accepted Answer

≈ 2.585 bits

Answer

6 bits

Answer

1 bit

Answer

3 bits

Question 24

Why is entropy a good measure of impurity for classification but not directly for regression?

Accepted Answer

It needs discrete class probabilities; regression uses variance/MSE

Answer

Entropy works equally well for both tasks

Answer

Regression uses entropy of the predictions

Answer

Entropy cannot be computed for more than two classes

Question 25

What happens to entropy as a class distribution becomes more skewed (one class dominating)?

Accepted Answer

Entropy decreases toward 0

Answer

Entropy increases toward infinity

Answer

Entropy stays constant

Answer

Entropy becomes negative

Question 26

In the 'surprise' interpretation, how much information does an event with probability 1 carry?

Accepted Answer

0 bits

Answer

1 bit

Answer

Infinite bits

Answer

Equal to the entropy

Question 27

What is the self-information (surprisal) of an event with probability p?

Accepted Answer

I = -log2(p)

Answer

I = p·log2(p)

Answer

I = 1 - p

Answer

I = log2(1+p)

Question 28

How does entropy connect to the concept of 'expected surprisal'?

Accepted Answer

Entropy is the expected value of self-information

Answer

Entropy is the maximum self-information

Answer

Entropy is the minimum self-information

Answer

Entropy is unrelated to surprisal

Question 29

When comparing two candidate splits, which one should a greedy decision tree pick?

Accepted Answer

The split with higher information gain

Answer

The split with more branches

Answer

The split with lower information gain

Answer

The split that uses the fewest samples

Question 30

What is cross-entropy, and how is it used in machine learning?

Accepted Answer

A loss comparing the true and predicted distributions

Answer

The entropy of two independent dice

Answer

A measure that is always 0

Answer

The sum of two separate entropies

Test yourself