Bayesian parameter estimation

Bayesian parameter estimation

Bayesian parameter estimation

Bayes rule

We want to generate the probability distribution of \(\theta \) given the evidence \(X\).

We can transform this using Bayes rule.

\(P(\theta | X)=\dfrac{P(X|\theta )P(\theta )}{P(X)}\)

Here we have:

  • Our prior - \(P(\theta )\)

  • Our likelihood function - \(P(X|\theta )\)

  • Our posterior - \(P(\theta | X)\)

Normal priors and posteriors

If our prior is a normal distribution then:

\(P(\theta )=\dfrac{1}{\sqrt {(2\pi )^n|\Sigma_0|}}e^{-\dfrac{1}{2}(x-\mu )^T\Sigma_0^{-1}(x-\mu)}\)

Similarly, if our likelihood function \(P(X|\theta )\) is a normal distriubtion then:

\(P(X|\theta )=\dfrac{1}{\sqrt {2\pi \sigma^2}}e^{-\dfrac{(x-\mu)^2}{2\sigma ^2}}\)

We can now plug these into Bayes rule:

\(P(\theta |X)=\dfrac{1}{P(X)}\dfrac{1}{\sqrt {2\pi \sigma_0^2}}e^{-\dfrac{(\theta-\mu_0)^2}{2\sigma_0^2}}\dfrac{1}{\sqrt {2\pi \sigma^2}}e^{-\dfrac{(x-\mu)^2}{2\sigma ^2}}\)

\(P(\theta |X)\propto e^{-\dfrac{1}{2}[\dfrac{(\theta-\mu_0)^2}{\sigma_0^2}+\dfrac{(x-\mu)^2}{\sigma ^2}]}\)

We can then set this an a new Gaussian:

\(P(\theta |X)=\dfrac{1}{\sqrt {(2\pi )^{n}|\Sigma|}^{\dfrac{1}{2}}} e^{-\dfrac{1}{2}[\dfrac{(\theta-\mu_0)^2}{\sigma_0^2}+\dfrac{(x-\mu)^2}{\sigma ^2}]}\)

Empirical Bayes

Bayes rule

We can calculate the posterior probability for \(\theta \), but we need a prior \(P(\theta )\).

\(P(\theta | X)=\dfrac{P(X|\theta )P(\theta )}{P(X)}\)

Empirical Bayes

With empirical Bayes we get our prior from the data.

We have \(P(X|\theta )\)

And \(P(\theta |\rho )\)

We observe \(X\) and want to estimate \(\theta \).

\(P(\theta |X)=\dfrac{P(X|\theta)P(\theta )}{P(X)}=\dfrac{P(X|\theta)}{P(X)}\int P(\theta | \rho )P(\rho )d\rho \)

Prior and posterior predictive distributions

Prior predictive distribution

Our prior predictive distribution for \(X\) depends on our prior for \(\theta \).

\(P(\mathbf x)=\int_\Theta P(\mathbf x|\theta)P(\theta )d\theta \)

Posterior predictive distribution

Once we have calculated \(P(\theta |X)\), we can calculate a posterior probability distribution for \(X\).

\(P(\mathbf x|\mathbf X)=\int_\Theta P(\mathbf x|\theta)P(\theta |\mathbf X)d\theta \)

Bayesian risk

Risk and Bayes risk.