# Bayesian parameter estimation

## Bayesian parameter estimation

### Bayesian parameter estimation

#### Bayes rule

We want to generate the probability distribution of $$\theta$$ given the evidence $$X$$.

We can transform this using Bayes rule.

$$P(\theta | X)=\dfrac{P(X|\theta )P(\theta )}{P(X)}$$

Here we have:

• Our prior - $$P(\theta )$$

• Our likelihood function - $$P(X|\theta )$$

• Our posterior - $$P(\theta | X)$$

#### Normal priors and posteriors

If our prior is a normal distribution then:

$$P(\theta )=\dfrac{1}{\sqrt {(2\pi )^n|\Sigma_0|}}e^{-\dfrac{1}{2}(x-\mu )^T\Sigma_0^{-1}(x-\mu)}$$

Similarly, if our likelihood function $$P(X|\theta )$$ is a normal distriubtion then:

$$P(X|\theta )=\dfrac{1}{\sqrt {2\pi \sigma^2}}e^{-\dfrac{(x-\mu)^2}{2\sigma ^2}}$$

We can now plug these into Bayes rule:

$$P(\theta |X)=\dfrac{1}{P(X)}\dfrac{1}{\sqrt {2\pi \sigma_0^2}}e^{-\dfrac{(\theta-\mu_0)^2}{2\sigma_0^2}}\dfrac{1}{\sqrt {2\pi \sigma^2}}e^{-\dfrac{(x-\mu)^2}{2\sigma ^2}}$$

$$P(\theta |X)\propto e^{-\dfrac{1}{2}[\dfrac{(\theta-\mu_0)^2}{\sigma_0^2}+\dfrac{(x-\mu)^2}{\sigma ^2}]}$$

We can then set this an a new Gaussian:

$$P(\theta |X)=\dfrac{1}{\sqrt {(2\pi )^{n}|\Sigma|}^{\dfrac{1}{2}}} e^{-\dfrac{1}{2}[\dfrac{(\theta-\mu_0)^2}{\sigma_0^2}+\dfrac{(x-\mu)^2}{\sigma ^2}]}$$

### Empirical Bayes

#### Bayes rule

We can calculate the posterior probability for $$\theta$$, but we need a prior $$P(\theta )$$.

$$P(\theta | X)=\dfrac{P(X|\theta )P(\theta )}{P(X)}$$

#### Empirical Bayes

With empirical Bayes we get our prior from the data.

We have $$P(X|\theta )$$

And $$P(\theta |\rho )$$

We observe $$X$$ and want to estimate $$\theta$$.

$$P(\theta |X)=\dfrac{P(X|\theta)P(\theta )}{P(X)}=\dfrac{P(X|\theta)}{P(X)}\int P(\theta | \rho )P(\rho )d\rho$$

### Prior and posterior predictive distributions

#### Prior predictive distribution

Our prior predictive distribution for $$X$$ depends on our prior for $$\theta$$.

$$P(\mathbf x)=\int_\Theta P(\mathbf x|\theta)P(\theta )d\theta$$

#### Posterior predictive distribution

Once we have calculated $$P(\theta |X)$$, we can calculate a posterior probability distribution for $$X$$.

$$P(\mathbf x|\mathbf X)=\int_\Theta P(\mathbf x|\theta)P(\theta |\mathbf X)d\theta$$

### Bayesian risk

Risk and Bayes risk.