The Naive Bayes classifier

Naive Bayes

The Naive Bayes posterior

Bayes theorem

Consider Bayes’ theorem


Here, \(y\) is the label, and \(x_1,x_2,...,x_n\) is the evidence. We want to know the probability of each label given evidence.

The denominator, \(P(x_1,x_2,...,x_n)\), is the same for all, so we only need to identify:

\(P(y|x_1,x_2,...,x_n)\propto P(x_1,x_2,...,x_n|y)P(y)\)

The assumption of Naive Bayes

We assume each \(x\) is independent. Therefore:


\(P(y|x_1,x_2,...,x_n)\propto P(x_1|y)P(x_2|y)...P(x_n|y)P(y)\)

The Naive Bayes classifier

Calculating the Naive Bayes estimator

With the Naive Bayes assumption we have:

\(P(y|x_1,x_2,...,x_n)\propto P(x_1|y)P(x_2|y)...P(x_n|y)P(y)\)

We now choose \(y\) which maximises this.

This is easier to calculate, as there is less of a sample restriction.

This is used when evidence is also in classes, as the chance of any individual outcome on a continuous probability is \(0\).

Estimating \(P(y)\)

We can easily calculate \(P(y)\), by looking at the frequency across the sample.

Estimating \(P(x_1|y)\)

Normally, \(P(x_1|y)=\dfrac{n_c}{n_y}\), where:

  • \(n_c\) is the number of instances where the evidence is \(c\) and the label is \(y\).

  • \(n_y\) is the number of instances where the label is \(y\).

Regularising the Naive Bayes estimator

To reduce the risk of specific probabilities being zero, we can adjust them, so that:

\(P(x_1|y)=\dfrac{n_c+mp}{n_y+m}\), where:

  • \(p\) is the prior probability. If this is unknown, use \(\dfrac{1}{k}\), where \(k\) is the number of classes.

  • \(m\) is a parameter called the equivilant sample size.

Text classification using Naive Bayes

Naive Bayes and text classification

Naive Bayes can be used to classify text documents. The \(x\) variables can be appearances of each word, and \(y\) can be the document classification.