Multiple hypothesis testing

Multiple hypothesis testing

P-hacking

Likely to see some significant results from random chance.

Family-Wise Error Rate (FWER)

What is the chance of making at least one false positive result?

Number of tests: $$m$$

Number of false positive results: $$V$$

$$FWER=P(V>0)$$

False Discovery Rate (FDR)

The proportion of false discoveries is:

$$Q=\frac{V}{V+S}$$

Where: $$V$$ is the number of false positives

$$S$$ is the number of true positives

The FRD is $$E[Q]$$.

The Bonferroni correction

We change the significance level.

reject if $$p\le \frac{\alpha }{m}$$

If $$m=1$$ this is the standard test.