Conditional probability and Bayes’ theorem


Conditional probability

We define conditional probability

\(P(E_i|E_j):=\dfrac{P(E_i\land E_j)}{P(E_j)}\)

We can show this is between \(0\) and \(1\).

\(P(E_j)=P(E_i\land E_j)+P(\bar{E_i}\land E_j)\)

\(P(E_i|E_j):=\dfrac{P(E_i\land E_j)}{ P(E_i\land E_j)+P(\bar{E_i}\land E_j)}\)

We know:

\(P(x_i|y_j):=\dfrac{P(x_i \land y_j)}{P(y_j)}\)

\(P(y_j|x_i):=\dfrac{P(x_i \land y_j)}{P(x_i)}\)


\(P(x_i|y_j)P(y_j)=P(y_j|x_i) P(x_i)\)

\(P(x_i|y_j)=\dfrac{P(y_j|x_i) P(x_i)}{P(y_j)}\)

Note that this is undefined when \(P(y_j)=0\)

Note that for the same event,

\(P(x_i|x_j)=\dfrac{P(x_i\land x_j)}{P(x_j)}\)


For the same outcome:

\(P(x_i|x_i)=\dfrac{P(x_i\land x_i)}{P(x_i)}\)



Bayes’ theorem

From the definition of conditional probability we know that:

\(P(E_i|E_j):=\dfrac{P(E_i\land E_j)}{P(E_j)}\)

\(P(E_j|E_i):=\dfrac{P(E_i\land E_j)}{P(E_i)}\)


\(P(E_i\land E_j)=P(E_i|E_j)P(E_j)\)

\(P(E_i\land E_j)=P(E_j|E_i)P(E_i)\)



Independent variables

Events are independent if:


Note that:

\(P(E_i\land E_j)=P(E_i|E_j)P(E_j)\)

And so for independent events:

\(P(E_i\land E_j)=P(E_i)P(E_j)\)

Conjugate priors

If the prior \(P(\theta)\) and the posterior \(P(\theta | X)\) are in the same family of distributions (eg both Gaussian), then the prior and posterior are conjugate distributions