Probability distributions

A probability distribution is any graph or function that represents the likelihood of obtaining a result from some probability experiment – such as a random-chance event, a survey of randomly-chosen individuals, an observational study. Any probability distribution has some domain and yields a range of values.

Probability distributions can be discrete, representing a countable domain, or continuous, representing an uncountable domain with an infinite number of possible values.

Here are a couple of examples of discrete probability distributions. In the first, we'll imagine that we have an "unfair" coin, one that comes up heads 75% of the time and tails the rest. Here's the discrete distribution:

Notice that the probabilities of the only two possible outcomes of this experiment (we omit the very unlikely event that the coin lands on its edge) add to 1. In other words, it's absolutely certain (probability = 1 or 100%) that one of the two outcomes will occur.

The second example is the distribution of the six possible outcomes of rolling a six-sided die. The set of possible outcomes is $\Omega = \{1, 2, 3, 4, 5, 6\}$. The capital Greek letter omega, $\Omega$, is often used to represent the set of all possible outcomes of a probability experiment. Here's the distribution:

Both of these illustrate the two characteristics that are essential for any probability distribution:

1. The probability of any individual outcome has to be postive and less than 1:

   $$0 \le P(x) \le 1$$

2. The sum of all probabilities in the distribution is 1.

   $$\sum_{i=1}^N P(x_i) = 1$$

Discrete → continuous

Often there's no sharp dividing line between a discrete and a continuous distribution – or at least one we can treat as continuous. This example shows the results of a U.S. Census Bureau study of the heights of American women ages 30-39. The actual discrete data is represented by the bar graph: the higher the bar, the more women (as a percent of all women 30-39) of that height. That is, the higher the bar, the greater the probability of a woman you meet aged 30-39 being that height.

Describing a distribution

Probability distributions come in all shapes and sizes. We need to be able to describe and classify them. In general, we want to know four things about any distribution:

• Shape – consider symmetry and skewness
• Center – often the mean or median The center gives us the location of the distribution in the domain.
• Spread – what extent does the data cover – what are the minimum and maximum? The spread of a distribution is usually characterized by one or both of two measures, the standard deviation $(\sigma)$ and the interquartile range (IQR).
• Outliers – are any points suspiciously far from the rest? Might we have reason not to consider them in our analysis of the data?

The acronyms SCSO or SOCS might help you remember these features.

The most common distribution of data or measurements with variations that occur by random chance is the Normal or Gaussian distribution, shown above. We'll cover its mathematical form on another page.

The height of the normal curve represents the probability of some event occuring or of finding some value of the data. High means likely and low means unlikely. The standard Gaussian curve is left-right symmetric, and the mean and median of the data are the same – right in the middle. Gaussian distributions can be very narrow or very spread out, representing high or low precision of the data, respectively.

Not all distributions are symmetric. Some are skewed one direction or another. We say that the distribution below is "skewed to the left" because it looks like it's being dragged to the left by its left-most point or "tail."

Shape of a distribution: The center

We measure the center of a distribution in two ways, the mean and the median. When a distribution is symmetric from left to right, these are the same. It's when that symmetry fails that the median can be more useful.

Here's a simple example: Let's say that there are 25 homes in a certain community. Their values, to the nearest \$50,000, are displayed in the dot plot below.

Now, as shown on the graph, the mean has shifted to a value of \$536,000, which is higher than the price of most houses in the area. In this case, the mean might not be representative of the true housing situation for a family looking to buy a home in the area.

The median home value, however, didn't change. It's the average of the 13^th and 14^th values, which remains \$350,000. So for a family looking to move into the neighborhood, they can be aware of that \$5-million home, but also that the price they will pay for a home will more likley be in the \$350,000 range.

This is precisely why we commonly use these two measures of the center of a distribution. When the mean and median are the same, we usually have a left-right symmetric distribution. When the mean is higher than the median, the distribution is skewed to the right, as is our wide home-value graph above. That \$5-million home really stretches the distribution out in that direction. When the mean is smaller than the median, the distribution is usually skewed to the left.

The median is more resistant to change produced by outliers – data points far outside the more clustered data.

Shape of a distribution: The width

The width of a distribution corresponds to the precision of the data. A distribution that is narrow means that there is a low probability of values far from the mean or median. A wider distribution means that data points far from the middle are more probable. Consider these two distributions of the distance of each dart from the bullseye of a dart board:

The player on the left is very good. She throws near the center of the board on every throw. The player on the right isn't as good. His throws sometimes hit the bullseye, but on average they're more widely scattered.

We use two methods to measure and report the width of a distribution, the standard deviation and the interquartile range. Both are discussed in other sections (standard deviation, interquartile range), so we won't go into them in great detail here.

Standard deviation

Strictly speaking, the standard deviation (denoted by the Greek lower case "s", $\sigma$) applies only to the Normal or Gaussian distribution. It is defined as the distance along the domain axis from the mean to one of the two inflection points of the curve. An inflection point is a point on the graph where the curvature changes from concave upward to concave downward, or vice versa).

The standard deviation is the average distance of the squares of the distances of each member of a data set from the mean of that set. Here is the forumula in summation notation:

$$\sigma = \sqrt{\frac{1}{N-1} \sum_{i=1}^N (x_i - \bar x)^2}$$

We use the squares of the distances, $x_i - \bar x$, to avoid having the errors of a wide distribution cancel to a very small width, one that doesn't comport with the precision of the data. We further divide not by $N$, the number of data points – as we would when calculating a mean – but $N - 1$ instead. This is because we've essentially already used one of our $N$ pieces of data to calculate the mean, leaving only $N - 1$ left for other calculations. For more about the standard deviation, go here.

What we're doing here is calculating the sum of the squares of all of the distances of each point from the center, then dividing by the number of data points (minus one), then taking the square root to undo the earlier squaring. We could have taken an absolute value rather than squaring (after all, $|x| = \sqrt{x^2}$), but it turns out that the squaring trick makes a lot of the mathematics of statistics easier as things get more complicated.

Outliers

Often when we collect data there are one or more points that just don't seem to fit. For example, let's say we're measuring reaction times in a chemistry experiment. We run the same reaction under the same conditions ten times, finding an average time of 8.2 ± 1.1 minutes for nine of the measurements, but a time of 21 minutes for the tenth. That last measurement just doesn't look right — it's an outlier, and we might be correct to think about what might have gone wrong with our technique. We might be tempted to throw that datum out of our data set. Would we be justified?

It's important to have some objective measure of what constitutes an outlier, some criterion that we all agree upon. One such criterion is the 1.5 IQR rule. It works like this:

• A data point is an outlier to the left if it is less than 1.5 times the IQR to the left of the first quartile (Q1).

  $x \lt \text{Q1} - 1.5 \cdot \text{IQR} $

• A data point is an outlier to the right if it is greater than 1.5 times the IQR to the right of the third quartile (Q3).

  $x \gt \text{Q3} + 1.5 \cdot \text{IQR} $

Example 1

Consider the dot plot showing the distribution of 52 scores on a chemistry test. Describe the distribution in terms of its shape, center, spread and outliers (SCSO).

Solution: The shape of this dot plot seems roughly symmetric and Normal (that is, has the shape of the Normal or Gaussian curve). If anything, it is skewed a bit to the right just a bit.

The center of the distribution can be measured in two ways: by calculating the mean and/or by finding the median. To find the mean, we just sum the values and divide by their number to get 70.9. The median is the average of the 26^th and 27^th data points, for a value of 70. The mean is just a bit larger than the median, confirming just a little bit of rightward skew.

The first and third quartiles are the averages of the 13^th and 14^th points (Q1 = 62.5) and of the 39^th and 40^th points (Q3 = 80). The IQR is then 80 - 62.5 = 17.5. We can check for outliers:

Uniform probability distribution

Note: the math after the heading "Average and variance" goes a little bit beyond what you might have encountered so far. If you can't follow it, just move on; that's OK.

The uniform probability distribution is one in which the probability of any outcome of a probability "experiment" is the same. A good example is the rolling of a single fair die, where "fair" means an equal chance of rolling 1, 2, 3, 4, 5 or 6.

The probability of rolling a 1 is the same as that for rolling a 2 or 3, 4, 5 or 6. For each there is a 1 in 6, or ⅙ chance. We can sketch a graph of these discrete outcomes like this:

Now any probability distribution must also capture the idea of an assured outcome, that is, that when the experiment (tossing the die) is performed, one outcome is assured. The die will come up with a result that is in the set of possible outcomes.

The total probability must sum to 1. In this case, we have six possible outcomes, each with a ⅙ probability, so the total area of our rectangular probability distribution graph (below) is 1. We would refer to this as a normalized distribution.

In mathematical notation, we'd write the sum of all six of the probabilities as

$$\sum_{i=1}^6 \, P_i = 1$$

Coin tossing is another example of a probability experiment with a uniform distribution of outcomes. There is a ½ probability of tossing heads and a ½ probability of tossing tails, the only two possible outcomes (we can approximate the probability of a coin landing on its edge to zero). The two probabilities sum to 1.

Example 2 – uniform distribution

Let's say we have a shoe store that sells pairs of shoes with a uniform probability of a sale throughout a seven-day week. That is, it's equally probable that a pair of shoes is sold on Monday as on Tuesday. The minimum number of pairs sold per week is 50, and the max is 250. Calculuate the probability of selling between 100 and 150 pairs of shoes. Calculate the mean and standard deviation, $\sigma = \sqrt{\sigma^2}$ of the distribution.

Solution: First, the probability is

$$P_{100-150} = (150 - 50) \cdot \frac{1}{250 - 50} = \frac{100}{200} = \frac{1}{2}$$

The mean of the distribution is

$$\bar{x} = \frac{a + b}{2} = \frac{50 + 250}{2} = 150 \; \text{pairs}$$

Mean & variance of a continuous distribution

The

The

First we ought to normalize our PDF. That means ensuring that the area underneath it over its domain is equal to 1. ere's the process:

$$ \begin{align} f(x) &= a(1 - x^2) \tag{1} \\[5pt] a \int_{-1}^1 \, (1 - x^2) dx &= 1 \tag{2} \\[5pt] a \bigg[ x - \frac{x^3}{3} \bigg]_{-1}^1 &= 1 \\[5pt] 2a \bigg[ x - \frac{x^3}{3} \bigg]_0^1 \tag{3} \\[5pt] 2a \bigg[ 1 - \frac{1}{3} \bigg] &= 1 \\[5pt] 2a \bigg[ \frac{2}{3} \bigg] &= 1 \\[5pt] \frac{4}{3} a &= 1 \\[5pt] a &= \frac{3}{4} \end{align}$$

So our normalized PDF is

$$f(x) = \frac{3}{4} (1 - x^2)$$

$$ \begin{align} \mu &= E(x) \\[5pt] &= \int_a^b \, x f(x) \, dx \\[5pt] &= \int_a^b \, x(1 - x^2) \, dx \\[5pt] &= \int_a^b \, (x - x^3) \, dx \\[5pt] &= \bigg[ \frac{x^2}{2} - \frac{x^4}{4} \bigg]_a^b \\[5pt] &= \frac{b^2}{2} - \frac{b^4}{4} - \frac{a^2}{2} + \frac{a^4}{4} \\[5pt] \mu &= \frac{b^2 - a^2}{2} - \frac{b^4 - a^4}{4} \end{align}$$

Just to check, let's think about a symmetric interval, with $a = b$. Then we have

$$\mu = \frac{a^2 - a^2}{2} - \frac{a^4 - a^4}{4} = 0,$$

which is what we expect for the mean of our distribution on a symmetric interval around zero.

The Bernoulli & Binomial distributions

Often we encounter situations in which our data answers a yes-or-no question. For a coin flip, for example, the result can either be heads or "not heads." It's a binary situation, at least if we disregard the exceedingly-small possibility of the coin landing on its edge. The Bernoulli^* and binomial distributions are suitable for modeling probability experiments that ask such yes-or-no questions.

For our coin flip, we might write the probability of tossing heads as P(H), then the probability of tails is P(!H), which we read as "not H," or "not heads." Further,

$$P(H) + P(!H) = 1.$$

The Bernoulli distribution need not represent 50/50 probabilities. It could also be used to model the behavior of a non-fair coin like one that came up heads 75% of the time, in which

$$ \begin{align} P(H) &= 0.75 \\[5pt] P(!H) &= 0.25 \\[5pt] P(H) + P(!H) &= 1 \end{align}$$

Here's that distribution as a bar graph:

The normal or Gaussian distribution

The curve under the bar graph above has a familiar "bell" shape. It's often referred to as a bell curve, but more often as the normal distribution or the Gaussian distribution, after Carl Friedrich Gauss.

The curve is a probability distribution. You can always read its meaning by imagining that the vertical axis is a measure of the relative probability or likelihood of something happening and that all of the somethings are arrayed in order along the x-axis.

The Gaussian curve is aways symmetric on either side of its maximum, and the maximum is the mean or average value. Whatever value or event is in the middle is the most likely. That "event" in our women's height example would be the "event" of being 5'-4" tall. Out in the "wings," probability is the lowest: There are far fewer very short and very tall women, and the probability of being short or tall is lower than being of more average height.

If we add up all of the probability under a Gaussian curve, we should get one (or 100%), the probability that something — anything at all — happened. Often we scale a Gaussian curve so that its total area – the area under the curve – is one. That's called "normalizing" the distribution.

Here's another example before we move on. The graph below shows the results of 5000 simulated throws of two dice. The sum of both dice is shown. Notice that because there are more ways to come up with a total of 7 (6+1, 5+2, 4+3),it's the most probable throw. After 5000 throws, the dice-total distribution looks pretty "normal."

So where does that curve come from?

That's a tricky question. It comes from modeling random chance, but the functional form of the curve has to be derived using calculus. In particular, it is derived using the second fundamental theorem of calculus. We don't need to go there just yet, though; the result will serve our needs just fine. Here's what the Guassian function / Normal function looks like, with some explanation of its parameters.

This function might look complicated, but think of the first part as a constant prefactor. Then the exponential function is a symmetric bell-shaped curve that's translated by -h units along the x-axis and scaled by the 2σ² in the denominator.