What can go wrong with a significance test?

On this page we'll add some critical analysis to our previous section on hypothesis testing. Mainly, we'll analyze what kinds of errors we can make, and what the consequences of those errors might be. We know that in significance tests we choose some confidence level, but ought to be mindful that, by definition, that level allows for errors to sneak through, however unlikely. Recall, for example, that a 95% condfidence level means that about 95% of all samples will capture the true mean of a distribution to within the sample mean plus or minus the margin of error. The other 5% will not.

Type I errors

A Type I error occurs when our significance test rejects the null hypothesis, $H_o$, when it is actually true. There are various ways to describe a Type I error, including a "false positive" (in the sense of the hypothesis test), or having a "hair trigger," a perhaps oversized propensity to reject.

The probability of making a Type I error is the conditional probability that, given that $H_o$ is true, we then conclude that te alternate hypothesis, $H_a$ is true instead,

$$P(H_a|H_o)$$

Here are a couple of examples.

• Consider a capital case in a court in the United States, where capital punishment (the death penalty) is still normal. The status quo in the U.S. system is that the accused is not guilty of the crime until proven to be so in a court — that's $H_o$. A trial is a test of the significance of the evidence against the null hypothesis. In U.S. courts, juries must reject the null hypothesis under the "beyond a reasonable doubt" standard. If $H_o$ is rejected but the null hypothesis (innocence) is actually true, then a grave error has been made and an accused person may be deprived of life by the government, which, in the U.S., is the citizens.

Type II errors

A Type II error occurs when our significance test fails to reject a null hypothesis which was in fact false. As a conditional probability, it is:

$$P(!H_a|!H_o)$$

Consider the capital murder case from above. A Type II error in that instance will let a guilty person go rather than convict, when in fact, the null hypothesis — presumed innocence — was false. Many would consider this the preferred outcome if an error was to be made at all, making a Type II error preferable to a Type I error – in that case.

Now think about the pharmaceutical example. In that case, if we assume that the null hypothesis is false — that the drug is better than the existing treatment or no treatment at all — and we miss that, then we've incurred an opportunity cost, a chance to improve treatment and for patients to feel better.

Here is a more visual way to look at these significance decisions.

Type I errors

Look at the red square on the chart. It represents a Type I error, for which the null hypothesis was true, but was rejected by the test. The probability of that rejecttion mdash; of that error — is $\alpha$.

You can see that from the figure on the right (we'll simplify just to one-tailed tests), that the probability of rejecting $H_o$ is $\alpha$. That figure shows two probability distributions, the one on the left assumes that the null hypothesis holds, and the one on the right that the alternative is true.

We can view the probability of making a Type I error as a conditional probability,

$$P(H_a | H_o) = \alpha.$$

$H_o$ true, correct decision

Below the red square is the outcome of a correct decision from the data. The null hypothesis is true and the test fails to reject it. We can use conditional probabilities to find that probability, too. The conditional probability that, given that $H_o$ is true, then $H_a$ is rejected can be written as $P(!H_a | H_o)$. Now we have

$$ \begin{align} P(!H_a|H_o) + P(H_a|H_o) &= 1 \\[5pt] P(!H_a)|H_o) &= 1 - P(H_a|H_o) \\[5pt] P(!H_a|H_o) &= 1 - \alpha \end{align}$$

Statistical power

The power of a statistical test is the probability that it correctly rejects the null hypothesis, $H_o$, when some alternative hypothesis, $H_a$ is true. According to our table above, that's $1 - \beta$. Like any probability, statistical power is between 0 and 1. As the power gets closer to one, $\beta$, the probability of making a Type II error, decreases. So increasing the power of a test decreases the Type II error probability.

We can't really calculate a statistical power in most situations. This is because of the nature of hypotheses (see box above). A null and alternative hypothesis are usually sometthing like $H_o: \; \mu = a, \phantom{00} H_a: \mu \ne a$.

Things that affect the resolving power of a significance test

1. Significance level (α)

   If all other things are held constant, then as $\alpha$ increases, so does the power of the test. Consider the distribution below, in which two $\alpha$ levels are marked off:

   

   As $\alpha$ increases, the probability of rejecting the null hypothesis increases, therefore the power of the test increases. BUT: As the area of the rejection region (red) increases, so does the probability of rejecting $H_o$ when it is actually true — the probability of a Type I error).

2. Sample size, $n$

   As $n$ increases, the width of the distribution of the test statistic is reduced by a factor of $1/\sqrt{n}$. The hypothesized distribution and the true distribution of the test statistic (we're assuming that $H_o$ is false) then become more separated, and it's easier to tell whether the observed statistic comes from one or the other.