# Bayesian parameter estimation

## Bayesian parameter estimation

### Bayesian parameter estimation

#### Bayes rule

We want to generate the probability distribution of \(\theta \) given the evidence \(X\).

We can transform this using Bayes rule.

\(P(\theta | X)=\dfrac{P(X|\theta )P(\theta )}{P(X)}\)

Here we have:

Our prior - \(P(\theta )\)

Our likelihood function - \(P(X|\theta )\)

Our posterior - \(P(\theta | X)\)

#### Normal priors and posteriors

If our prior is a normal distribution then:

\(P(\theta )=\dfrac{1}{\sqrt {(2\pi )^n|\Sigma_0|}}e^{-\dfrac{1}{2}(x-\mu )^T\Sigma_0^{-1}(x-\mu)}\)

Similarly, if our likelihood function \(P(X|\theta )\) is a normal distriubtion then:

\(P(X|\theta )=\dfrac{1}{\sqrt {2\pi \sigma^2}}e^{-\dfrac{(x-\mu)^2}{2\sigma ^2}}\)

We can now plug these into Bayes rule:

\(P(\theta |X)=\dfrac{1}{P(X)}\dfrac{1}{\sqrt {2\pi \sigma_0^2}}e^{-\dfrac{(\theta-\mu_0)^2}{2\sigma_0^2}}\dfrac{1}{\sqrt {2\pi \sigma^2}}e^{-\dfrac{(x-\mu)^2}{2\sigma ^2}}\)

\(P(\theta |X)\propto e^{-\dfrac{1}{2}[\dfrac{(\theta-\mu_0)^2}{\sigma_0^2}+\dfrac{(x-\mu)^2}{\sigma ^2}]}\)

We can then set this an a new Gaussian:

\(P(\theta |X)=\dfrac{1}{\sqrt {(2\pi )^{n}|\Sigma|}^{\dfrac{1}{2}}} e^{-\dfrac{1}{2}[\dfrac{(\theta-\mu_0)^2}{\sigma_0^2}+\dfrac{(x-\mu)^2}{\sigma ^2}]}\)

### Empirical Bayes

#### Bayes rule

We can calculate the posterior probability for \(\theta \), but we need a prior \(P(\theta )\).

\(P(\theta | X)=\dfrac{P(X|\theta )P(\theta )}{P(X)}\)

#### Empirical Bayes

With empirical Bayes we get our prior from the data.

We have \(P(X|\theta )\)

And \(P(\theta |\rho )\)

We observe \(X\) and want to estimate \(\theta \).

\(P(\theta |X)=\dfrac{P(X|\theta)P(\theta )}{P(X)}=\dfrac{P(X|\theta)}{P(X)}\int P(\theta | \rho )P(\rho )d\rho \)

### Prior and posterior predictive distributions

#### Prior predictive distribution

Our prior predictive distribution for \(X\) depends on our prior for \(\theta \).

\(P(\mathbf x)=\int_\Theta P(\mathbf x|\theta)P(\theta )d\theta \)

#### Posterior predictive distribution

Once we have calculated \(P(\theta |X)\), we can calculate a posterior probability distribution for \(X\).

\(P(\mathbf x|\mathbf X)=\int_\Theta P(\mathbf x|\theta)P(\theta |\mathbf X)d\theta \)

### Bayesian risk

Risk and Bayes risk.