Naive Bayes and probability models

The 'naive' assumption is that all features (predictors) are conditionally independent of one another given the class label. This means the presence or value of one feature is assumed not to affect any other feature once the class is known. Although this is rarely strictly true in real data, the classifier still often performs surprisingly well.

Bayes' theorem states P(class | features) = P(features | class) * P(class) / P(features). Here P(class) is the prior, P(features | class) is the likelihood, and P(features) is the evidence (a normalizing constant). The classifier picks the class that maximizes the numerator P(features | class) * P(class).

The prior P(class) is the probability of a class before observing any feature evidence, usually estimated as the fraction of training examples belonging to that class. It encodes our initial belief about how common each class is. The likelihood from the data then updates this prior into the posterior.

P(features) is the same constant for every class because it does not depend on the class label, so it does not change which class has the largest posterior. Since we only need to compare classes and pick the maximum, we can drop this normalizing term. We therefore compare P(features | class) * P(class) across classes.

Under conditional independence, the joint likelihood factorizes into a product of individual feature likelihoods: P(x1,...,xn | class) = P(x1|class) * P(x2|class) * ... * P(xn|class). This turns a hard-to-estimate joint distribution into many simple one-feature distributions. It is the key simplification that makes Naive Bayes fast and easy to train.

likelihood = 1.0
for xi in features:
    likelihood *= P_xi_given_class[xi]

Laplace smoothing adds a small constant (often 1) to every count when estimating probabilities, so no probability is ever exactly zero. It is used to avoid the zero-frequency problem, where an unseen feature value would make the entire product likelihood zero. This keeps the classifier robust to feature values not present in the training data for a class.

# Laplace smoothing for a categorical feature
P = (count_xy + 1) / (count_y + num_values)

Multinomial Naive Bayes is most appropriate for word-count (frequency) features, such as how many times each word appears in a document. It models the data as draws from a multinomial distribution over the vocabulary. It is the standard choice for tasks like spam detection and topic classification using bag-of-words counts.

Gaussian Naive Bayes assumes that the continuous features within each class follow a normal distribution. For each class it estimates a mean and variance per feature, then uses the Gaussian probability density to compute likelihoods. It is suited to continuous numeric data like measurements of height or sensor readings.

Bernoulli Naive Bayes works with binary (0/1) features that indicate the presence or absence of an attribute. In text classification it uses whether a word appears in a document, not how many times. Each feature is modeled with a Bernoulli distribution conditioned on the class.

Multiplying many small probabilities can cause numerical underflow, where the product becomes too tiny for the computer to represent accurately. Taking logarithms converts the product into a sum of log probabilities, which is numerically stable and does not change which class has the maximum score. This is why implementations add log P(xi|class) instead of multiplying P(xi|class).

import math
score = math.log(prior)
for xi in features:
    score += math.log(P_xi_given_class[xi])

Compute the posterior numerators: spam gives 0.4 * 0.02 = 0.008, and ham gives 0.6 * 0.005 = 0.003. Since 0.008 > 0.003, the larger score belongs to spam. Therefore Naive Bayes classifies the email as spam.

spam = 0.4 * 0.02   # 0.008
ham  = 0.6 * 0.005  # 0.003
print('spam' if spam > ham else 'ham')

A generative model learns the joint distribution P(features, class) and can model how the data is generated, while a discriminative model directly learns the decision boundary P(class | features). Naive Bayes is a generative model because it models P(features | class) and P(class). Logistic regression, by contrast, is discriminative.

The zero-frequency problem occurs when a feature value never appears with a particular class in the training data, giving it an estimated probability of zero. Because likelihoods are multiplied together, a single zero makes the whole product zero, wiping out all other evidence. Smoothing techniques like Laplace smoothing are used to prevent this.

This equation is the definition of statistical independence: the joint probability equals the product of the individual probabilities. It means knowing that one event occurred gives no information about whether the other occurred. Naive Bayes assumes this kind of independence holds for features given the class.

Conditional probability P(A | B) is the probability that event A occurs given that event B has occurred. It is defined as P(A | B) = P(A and B) / P(B), provided P(B) > 0. It rescales probabilities to the subset of outcomes where B is true.

Using Bayes' theorem, P(disease|positive) = (0.99*0.01) / (0.99*0.01 + 0.01*0.99) = 0.0099 / 0.0198 = 0.5, about 50%. Despite the test being 99% accurate, the low base rate of 1% means about half of positives are false positives. This illustrates how the prior strongly affects the posterior.

p = (0.99*0.01) / (0.99*0.01 + 0.01*0.99)
print(p)  # 0.5

The class sklearn.naive_bayes.MultinomialNB implements Multinomial Naive Bayes. It is commonly used for text classification with count or TF-IDF features. You fit it with model.fit(X, y) and predict with model.predict(X).

from sklearn.naive_bayes import MultinomialNB
model = MultinomialNB()
model.fit(X_train, y_train)

Naive Bayes only needs the correct class to receive the highest posterior score, not perfectly calibrated probabilities. Even when features are correlated, the ranking of classes is frequently still correct, so classification accuracy stays high. This robustness is why it remains a strong, fast baseline despite an unrealistic assumption.

The likelihood P(features | class) measures how probable the observed features are assuming a given class, while the posterior P(class | features) measures how probable the class is given the observed features. Bayes' theorem converts the likelihood (times the prior) into the posterior. Classification decisions are based on the posterior.

It is estimated as the count of that word in all spam documents divided by the total number of words in spam documents, usually with Laplace smoothing added. This is the maximum-likelihood estimate of the word's probability under the spam class. The same is done for the ham class to compare likelihoods.

P_word_spam = (word_count_in_spam + 1) / (total_spam_words + vocab_size)

When all classes share the same per-feature variance, the quadratic terms cancel and Gaussian Naive Bayes produces a linear decision boundary. With different variances per class, the boundary becomes quadratic (curved). This connects Naive Bayes to linear classifiers under special conditions.

The law of total probability says P(features) = sum over all classes of P(features | class) * P(class). It combines the likelihood and prior across every possible class to get the overall probability of the observed features. This term is the denominator in Bayes' theorem.

Because Naive Bayes treats features as independent, a duplicated feature is counted twice, doubling its influence (its log-likelihood contribution is added twice). This can overweight that piece of evidence and skew the posterior. Removing redundant or highly correlated features often improves the classifier.

The MAP rule selects the class with the highest posterior probability P(class | features), which is equivalent to maximizing P(features | class) * P(class). It balances how well the data fits each class (likelihood) against how common each class is (prior). This is the standard prediction rule for Naive Bayes.

The ML rule chooses the class that maximizes only the likelihood P(features | class), ignoring the prior. The MAP rule additionally multiplies by the prior P(class), so it accounts for how frequent each class is. ML and MAP give the same answer only when all class priors are equal.

It uses the Gaussian probability density function: P(x|class) = (1 / sqrt(2*pi*var)) * exp(-(x - mean)^2 / (2*var)), where mean and var are estimated per class. The value x is plugged into this density to get the likelihood. The class-specific mean and variance are learned from training data.

import math
def gauss(x, mean, var):
    return (1/math.sqrt(2*math.pi*var))*math.exp(-(x-mean)**2/(2*var))

P(first red) = 3/5, and given a red was drawn, P(second red) = 2/4. Multiplying gives (3/5)*(2/4) = 6/20 = 3/10 = 0.3. So the probability both balls are red is 0.3.

p = (3/5) * (2/4)
print(p)  # 0.3

Because it estimates each feature's probability independently, Naive Bayes needs only a small amount of data per feature and scales linearly with the number of features. This avoids the explosion of parameters that joint models require in high dimensions. It is therefore fast to train and effective on thousands of word features in text.

The classes should be mutually exclusive (an example belongs to exactly one class) and exhaustive (every example belongs to some class), so that their priors sum to 1. This ensures the law of total probability correctly computes the evidence P(features). It also guarantees the posteriors across classes sum to 1.

A bag-of-words represents a document by the counts (or presence) of each word from a vocabulary, ignoring word order and grammar. Each document becomes a vector of word frequencies. This representation pairs naturally with Multinomial or Bernoulli Naive Bayes.

When all priors are equal, the prior term is a constant that does not affect the comparison, so MAP reduces to maximizing the likelihood P(features | class). This is exactly the maximum likelihood (ML) decision rule. The prediction then depends only on how well the features fit each class.

Because of the unrealistic independence assumption, the predicted class probabilities from Naive Bayes are often poorly calibrated, frequently being pushed very close to 0 or 1. The class ranking and chosen label can still be accurate, but the numeric confidence values should not be trusted as true probabilities. Calibration techniques can be applied if reliable probabilities are needed.

P(A)=3/6=0.5, P(B)=3/6=0.5, and A and B = {4,6} so P(A and B)=2/6=1/3. Since P(A)*P(B)=0.25 but P(A and B)=0.333, they are not equal, so the events are not independent. The dependence shows knowing B changes the probability of A.

PA, PB = 3/6, 3/6
P_AB = 2/6
print('independent' if abs(PA*PB - P_AB) < 1e-9 else 'not independent')