We want to generate the probability distribution of \(\theta\) given the evidence \(X\).
We can transform this using Bayes rule.
\(P(\theta | X)=\dfrac{P(X|\theta )P(\theta )}{P(X)}\)
Here we have:
Our prior - \(P(\theta )\)
Our likelihood function - \(P(X|\theta )\)
Our posterior - \(P(\theta | X)\)
If our prior is a normal distribution then:
\(P(\theta )=\dfrac{1}{\sqrt {(2\pi )^n|\Sigma_0|}}e^{-\dfrac{1}{2}(x-\mu )^T\Sigma_0^{-1}(x-\mu)}\)
Similarly, if our likelihood function \(P(X|\theta )\) is a normal distriubtion then:
\(P(X|\theta )=\dfrac{1}{\sqrt {2\pi \sigma^2}}e^{-\dfrac{(x-\mu)^2}{2\sigma ^2}}\)
We can now plug these into Bayes rule:
\(P(\theta |X)=\dfrac{1}{P(X)}\dfrac{1}{\sqrt {2\pi \sigma_0^2}}e^{-\dfrac{(\theta-\mu_0)^2}{2\sigma_0^2}}\dfrac{1}{\sqrt {2\pi \sigma^2}}e^{-\dfrac{(x-\mu)^2}{2\sigma ^2}}\)
\(P(\theta |X)\propto e^{-\dfrac{1}{2}[\dfrac{(\theta-\mu_0)^2}{\sigma_0^2}+\dfrac{(x-\mu)^2}{\sigma ^2}]}\)
We can then set this an a new Gaussian:
\(P(\theta |X)=\dfrac{1}{\sqrt {(2\pi )^{n}|\Sigma|}^{\dfrac{1}{2}}} e^{-\dfrac{1}{2}[\dfrac{(\theta-\mu_0)^2}{\sigma_0^2}+\dfrac{(x-\mu)^2}{\sigma ^2}]}\)
We can calculate the posterior probability for \(\theta\), but we need a prior \(P(\theta )\).
\(P(\theta | X)=\dfrac{P(X|\theta )P(\theta )}{P(X)}\)
With empirical Bayes we get our prior from the data.
We have \(P(X|\theta )\)
And \(P(\theta |\rho )\)
We observe \(X\) and want to estimate \(\theta\).
\(P(\theta |X)=\dfrac{P(X|\theta)P(\theta )}{P(X)}=\dfrac{P(X|\theta)}{P(X)}\int P(\theta | \rho )P(\rho )d\rho\)
Our prior predictive distribution for \(X\) depends on our prior for \(\theta\).
\(P(\mathbf x)=\int_\Theta P(\mathbf x|\theta)P(\theta )d\theta\)
Once we have calculated \(P(\theta |X)\), we can calculate a posterior probability distribution for \(X\).
\(P(\mathbf x|\mathbf X)=\int_\Theta P(\mathbf x|\theta)P(\theta |\mathbf X)d\theta\)
Risk and Bayes risk.