The basic flow of the probability pages goes like this:
- Definitions
- Set operations
- Probability axioms (this page)
- Counting: Permutations & combinations
- Conditional probability
- Independence of events
- Bayesian probability
Doing the math
In this section we'll assert the three basic axioms of probability theory. Then we'll put those together to construct some further useful rules.
In mathematics, we always begin with the fewest axioms necessary to build up the whole theory. It may seem like our three axioms are a little spare, but hang in there until we develop all of the rules.
The probability axioms
In mathematics, we want to derive as much as possible from the simplest set of axioms or assumptions. There are three essential probability axioms. They are
1. P(A) ≥ 0
The probability of an event is always positive. We agree that a negative probability is not necessary.
2. P(Ω) = 1
This means that the total probability for some part of the sample space to occur is 1. If the sample space includes all possible outcomes, then one of them has to happen. Consider the coin-flipping sample space:
If a coin is flipped, one of the two possible outcomes must occur. The total probability that one of them happens, regardless of which, is 1. This property of probability spaces is also called the normalization property, and we would say that any such space is normalized.
3. P(A ∪ B) = P(A) + P(B) for disjoint sets
Recall that disjoint sets are sets that have no elements in common. In a Venn diagram, they look like this:
The axiom says that the probability of the union of these two sets is the sum of the probabilities of each set. This is the probability that either an outcome in set A will occur or one in set B will occur. Often we interpret the union sign, $\cup,$ as the word "or": $A \cup B$ means "A or B."
Now we'll use these axioms to expand outward and derive some other useful rules that we'll eventually use to solve probability problems.
0 ≤ P(A) ≤ 1
Consider a probability space divided into two regions:
The two regions represent an event or set of events, A, occuring or not occuring (!A). Recall that we can also call !A the complement of A, or AC.
Now the union of these two sets is the entire space:
$$A \cup !A = \Omega$$
Because these two sets have no overlap (they are disjoint, they have no intersection:
$$A \cap !A = \varnothing,$$
where $\varnothing$ is the empty or null set, { }.
Now if we use our third axiom, we can write
$$P(A \cup !A) = P(A) + P(!A)$$
Now $P(A \cup !A) = P(\Omega) = 1$ by our second axiom, so
$$ \begin{align} 1 &= P(A) + P(!A) \\[5pt] P(A) &= 1 - P(!A) \end{align}$$
Now $P(!A) \gt 0,$ so we conclude that $P(A) \le 1.$ That is, probablities are always in the interval [0, 1], or $0 \le P(A) \le 1.$
Probability of the null set, $\varnothing,$ { }
The probability of any set plus its complement is
$$P(A) + P(!A) = 1$$
So this must also be true for an entire sample space:
$$P(\Omega) + P(!\Omega) = 1$$
Now we've already said that the probability that some element of a sample space will occur is 1, or $P(\Omega) = 1,$ so we have
$$1 + P(!\Omega) = 1,$$
and $P(!\Omega) = P(\varnothing),$
So we have
$$P(\varnothing) = 0.$$
The probability of the null set is zero.
Adding more than one disjoint set
Consider a probability sample space, Ω, with three possible outcome sets, A, B & C, shown here in a Venn diagram:
The three sets are disjoint (do not overlap or share any elements), and we'd like to come up with a formula for their union, $A \cup B \cup C,$ using our third axiom. To do that, we can first join sets A and B together as (A ∪ B)., then we can string all three together as
$$ \begin{align} P(A \cup B \cup C) &= P((A \cup B) \cup C) \\[5pt] &= P(A) + P(B) + P(C) \end{align}$$
It's not too difficult to think about extending this procedure to come up with a general rule for the uniton of n disjoint sets:
$$ \begin{align} P(A_1 \cup A_2 &\cup \dots \cup A_n) \\[5pt] &= P(A_1) + P(A_2) + \dots + P(A_n) \end{align}$$
We can write this rule in summation shorthand like this:
$$P(A_1 \cup A_2 \cup \dots \cup A_n) = \sum_{i=1}^n \, P(A_i)$$
When sets are not disjoint, we need to make a small modification to this procedure, which we'll do below.
Relations among subsets
We'd like to prove that if $A \subset B,$ then $P(A) \le P(B).$ The symbol $\subset$ means "is a subset of." Here's the Venn diagram of the situation:
Notice that set B can be written as the union of set A (the hashed area), and the intersection of everything that's not A, and B (the plain yellow area) like this:
$$B = A \cup (B \cap !A),$$
Then we have
$$P(B) = P(A) + P(B \cap !A) \ge P(A)$$
Notice that if A is a subset of B, and A ≠ B, then $P(A) \lt P(B).$
Unions of non-disjoint sets
Now let's combine sets that overlap. Here's the picture:
We can't just add the sets here as we did for disjoint sets, because we'd add the overlap, $(A \cap B),$ twice. That is, we'd overcount. Here's one way to think about how to do it correctly. We'll divide the regions into three parts:
Now the regions are
- $a = P(A \, \cap !B)$ everything in A and not in B. That is, this region is set A minus the overlap, c.
- $b = P(A \cap B)$ everything in both A and B. This is the overlap region, c.
- $c = P(B \, \cap !A)$ everything in B and not in A. That is, this region is the set B minus the overlap, c.
So if we add all three regions, $a + b + c,$ we get
$$a + b + c = P(A \cap !B) + P(A \cap B) = P(B \cap !A)$$
Now notice that
$$ \require{cancel} \begin{align} P(A) + P(B) &- P(A \cap B) = (a + c) + (b + c) - c \\[5pt] &= a + c + b + \cancel{c} - \cancel{c} \\[5pt] &= a + b + c \end{align}$$
So we have a formula for finding the probability of the union of non-disjoint sets. Notice that it also works for disjoint sets because in that case $P(A \cap B) = 0.$ If we go a little bit further, we note that $P(A \cap B) \le 1,$ which means that $P(A \cup B) \le P(A) + P(B),$ which is sometimes called the union bound. It says that the greatest a union of probabilities can be is the union of disjoint sets.
Sometimes this idea is called the inclusion-exclusion principle.
Example: Union of non-disjoint sets
Let's do a quick example of a probability problem requiring the union of non-disjoint sets: Let's calculate the probability of rolling a sum of seven OR at least one even number on two distinguishable cubic dice.
There are six possibilities for rolling a total of seven (we'll call that event 7 and its probability P(7)). Those are boxed in the table below and show up on the diagonal, and all of the possibilites for ending up with at least one even number (we'll call that event E and its probability P(E)) are highlighted in green. Notice that these sets overlap.
In a Venn diagram, it looks like this:
The gray elements are in the sample space for throwing two dice, but they are not in either set {E} or {7}.
Notice that all of the elements of the set $\{7\} = \{16, 25, 34, 43, 52, 61\}$ are also contained in the set E. So if we combine all of the elements of {E} and {7} (remember, we're looking for the union of the sets), we get
$$ \begin{align} |E \cup 7| &= |E| + |7| \\[5pt] &= 27 + 6 = 33 \end{align}$$
The problem here is that we've overcounted. We've actually counted the intersection of the sets, $E \cap 7,$ twice. We don't want to do that. To correct for our error, we can simply subtract the intersection of the two sets:
$$A \cup B = A + B - A \cap B.$$
This would ensure that, in terms of numbers of elements in the sets,
$$ \begin{align} |A \cup B| &= |A| + |B| - |A \cap B| \\[5pt] &= 27 + 6 - 6 = 27. \end{align}$$
So the probability of rolling either a seven or having at least one even number is 27/36, or ¾.
Inclusion-exclusion principle
When counting the elements of the union of two mutually-exclusive sets (sets that share no elements), the number of elements is
$$P(A \cup B) = P(A) + P(B)$$
If the sets are not mutually-exclusive (they share one or more members), then the number of elements in the union is
$$P(A \cup B) = P(A) + P(B) - P(A \cap B)$$
Any union of probability sets has an upper bound:
$$P(A \cup B) \le P(A) + P(B),$$
Discrete probabilities
The probability of occurence of an event (call it event x) with a discrete number of equally-probable* outcomes (n) is the inversely proportional to the number of outcomes:
$$P(x) \propto \frac{1}{n}$$
The probability is equal to the number of ways of obtaining a given outcome (k) divided by the total number of outcomes:
$$P(x) = \frac{k}{n}$$
For example, the probability of rolling a 1 on a single role of a six-sided die is
$$P(1) = \frac{1}{6},$$
where we have only one way to roll a six, and the sample space is $\Omega = \{1, 2, 3, 4, 5, 6\}.$
The probability of rolling an even number on a single roll of a six-sided die is the number of ways of rolling an even number, $\{2, 4, 6\},$ or three ways, divided by the six possible outcomes, so $P(\text{even}) = \frac{3}{6} = \frac{1}{2}.$
The probability of rolling two cubic dice and having the sum be seven can be calculated in the same way. The ways of obtaining a sum of 7 are
$$ \begin{Bmatrix} 1,6 & 6,1 \\ 2,5 & 5,2 \\ 3,4 & 4,3 \end{Bmatrix}$$
The total sample space is
$$ \Omega = \begin{Bmatrix} 1,1 & 1,2 & 1,3 & 1,4 & 1,5 & 1,6 \\ 2,1 & 2,2 & 2,3 & 2,4 & 2,5 & 2,6 \\ 3,1 & 3,2 & 3,3 & 3,4 & 3,5 & 3,6 \\ 4,1 & 4,2 & 4,3 & 4,4 & 4,5 & 4,6 \\ 5,1 & 5,2 & 5,3 & 5,4 & 5,5 & 5,6 \\ 6,1 & 6,2 & 6,3 & 6,4 & 6,5 & 6,6 \end{Bmatrix}$$
So the probability of rolling a sum of seven is
$$P(7) = \frac{6}{36} = \frac{1}{6}.$$
*Equally-probable is important here. The rules change if we're working, for example, with a pair of loaded (weighted) dice that land with 5 showing more often than any other number.
Multiplying probabilities
We've shown before that the probability of rolling a six on a single cubic die is ⅙. What about the probability of rolling two sixes in a row? In this case, we simply multiply probabilities. The probability of rolling two sixes in a row is
$$P(6,6) = \frac{1}{6} \cdot \frac{1}{6} = \frac{1}{36}.$$
Let's do another couple of examples. Consider flipping a two-sided coin. What is the probability of tossing three "heads" in a row? We know that there are two equally likely outcomes, $\Omega = \{H, T\}.$ The probability of tossing three heads in a row is
$$ \begin{align} P(HHH) &= P(H) \cdot P(H) \cdot P(H) \\[5pt] &= \frac{1}{2} \frac{1}{2} \frac{1}{2} = \frac{1}{8}. \end{align}$$
Now consider a jar containing 100 red marbles and 20 blue marbles. What is the probability that four marbles, pulled consecutively from the jar without looking, will all be blue? First, the probability of drawing a blue marble is 100/120 = 5/6. Now the probability of doing this four times in a row is
$$P(4B) = \left( \frac{5}{6} \right)^4 = 0.48$$
So there's a 48% chance (about 50-50) of drawing four blue marbles in a row given this setup.
Here's another way to look at our first experiment, rolling two sixes in a row on a cubic die, using a tree diagram:
For the first roll, there are six possibilities. For the second, each of these is followed by six more. After two rolls, there are 36 possible outcomes, 6 × 6, so the total probability of rolling two sixes in a row is ⅙ ยท ⅙ = 1/36.
Adding probabilities
Probabilities are added when we are talking about alternative ways of obtaining the same outcome. For example, think about tossing a fair coin (50-50 chance of obtaining heads or tails) three times. What is the probability of obtaining tails exactly once?
The sample space for this experiment is
$$ \Omega = \begin{Bmatrix} HHH & \color{#E90F89}{HHT} & \color{#E90F89}{HTH} \\ \color{#E90F89}{THH} & TTH & HTT \\ THT & TTT & \end{Bmatrix}$$
The probability is 3 possibilities out of 8, or 3/8.
We can caclulate this probability by realizing that, first, the probability of getting two heads and one tails is
$$ \begin{align} P(HHT) &= P(T) \cdot P(T) \cdot P(H) \\[5pt] &= \frac{1}{2} \cdot \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{8}. \end{align}$$
Now there are three ways to toss and obtain two tails and a head, so we add this probability three times:
$$P(1T) = \frac{1}{8} + \frac{1}{8} + \frac{1}{8} = \frac{3}{8}.$$
Example: Adding probabilities
What is the probability of rolling either a 3 or a 4 with a cubic (six-sided) die?
In this situation, the tree diagram is very helpful.
The probability of rolling a 3 or a 4 is clearly two chances out of six, or ⅓.
Notice that this is the same as adding the sum of the individual probabilities,
$$P(3 \text{ or } 4) = \frac{1}{6} + \frac{1}{6} = \frac{1}{3}.$$
Multiplying vs. adding
When we are calculating the probability of consecutive events, we multiply probabilities. Think of consecutive events, like flipping a coin to heads, then doing it again, as increasingly rare. Probabilities are numbers ≤ 1, so multiplying them gives a smaller result. A key word often present when multiplication is necessary is "and"
When one outcome has more than one mutually-exclusive path to get to it (like flipping HHT, HTH or THH to obtain exactly one tails after flipping three times), we add those individual probabilities to get the final probability. A key word often present when multiplication is necessary is "or"
Practice problems
| 1. |
Which of the following are valid probabilities? Why or why not?
Solutions(a) The highest-possible probability is P = 1, or 100%. (b) This is a valid probability. Probabilities must be in the range $0 \le P \le 1.$ (c) 38/37 > 1, so this is not a valid probability. (d) Probabilities can't be negative. |
||||||||||||
| 2. |
Let P(A) and P(B) be probabilities of events A and B. Write probability statements for each of these events:
Solutionsa. $P(A \cup B)$ b. $P(A \cap B)$ c. $P(!A)$ d. $P(A) \cdot P(!B)$ e. $P(!A) \cdot P(!B)$ f. $P(A) + P(B)$ g. The probability that only one event occurs is the converse of the probability that both occur: $P = 1 - P(A \cap B).$ |
||||||||||||
| 3. |
Consider the Venn diagram showing sets Ω, A and B.
Calculate these probabilities:
Solutionsa. There are ten elements in the universal set Ω and four in A, so the probability is $$P(A) = \frac{3}{10} = 30 \%$$ b. there are four elements in B so $$P(B) = \frac{4}{10} = \frac{2}{5} = 30 \%$$ c.$P(!A) = 1 - P(A) = \frac{7}{10}.$ We can also notice that of the elements in Ω, seven are not in A, and we get the same result. d. $P(!B) = 1 - P(B) = \frac{3}{5}.$ e. The set $A \cup B$ consists of all elements inside sets A and B. There are six of them, so $$P(A \cup B) = \frac{6}{10} = \frac{3}{5} = 60\%$$ f. $P(!(A \cup B)) = 1 - P(A \cup B) = \frac{2}{5}.$ g. $P(A \cap B)$ consists of all elements in both sets A and B. There is only one, { 9 }, so the probability is $P(A \cap B) = \frac{1}{10}.$ h. $P(!(A \cap B)) = 1 - P(A \cap B) = \frac{9}{10}.$ i. $P(!A) \cap P(!B)$ is the probability of the intersection of elements not in sets A and B. That set is {1, 3, 5, 8}, so the probability is $P(!A) \cap P(!B) = \frac{2}{5}.$ We could also get there using one of DeMorgan's laws: $$P(!A) \cap P(!B) = P(!A \cup !B) = P(!(A \cup B)$$ This is just the set of everything outside of $(A \cup B)$, which is {1, 3, 5, 8}. j. The union $P(!A) \cup P(!B)$ is everything not in sets A and B. Because the element 9 is in both A and B, it is the only thing excluded from this set, for a probability of $$P(!A) \cup P(!B) = \frac{9}{10}$$ We can also use the other version of DeMorgan's laws to get there: $$P(!A) \cup P(!B) = P(!(A \cap B))$$ Now $A \cap B$ is the set { 9 }, so the converse is a set with nine elements, for the same result. |
||||||||||||
| 4. |
A seller of electric vehicles makes the following repairs in one year (see table). Calculate the probability that if a customer were to buy a car (based on last year's information), they would need a major or a minor repair [P(maj), P(min), respectively].
SolutionsFrom the table we can tell that 360 cars were sold, so that's the denominator in our probability calculations. The probability of minor repairs is $$P(min) = \frac{110}{360} = \frac{11}{36} \approx 31 \%.$$ The probability of a major repair is $$P(maj) = \frac{30}{360} = \frac{1}{12} \approx 8 \%$$ That's a pretty high incidence of major repairs, actually. I made the data up, though. Hopefully car manufacturers do much better! |
||||||||||||
| 5. |
A bin of mechanical components at a factory contains parts that are, on average, about 5% defective. From a bin of 300 such parts, calculate the probabilities that if four parts are removed,
SolutionsOf 300 parts, a 5% defective rate means that there are 15 defective parts in the bin. We assume that the parts aren't replaced after they're taken out. a. The probability that we select four defective parts, P(D), should be very small. It is 15/300 for the first part, then 14/299 for the second (one defective part is gone), and so on. So the probability is $$ \begin{align} P(DDDD) &= \frac{30}{300} \cdot \frac{29}{299} \cdot \frac{28}{298} \cdot \frac{27}{297} \\[5pt] &= 0.008 \% \end{align}$$ b. To calculate the probability of drawing two good and two defective parts, we're getting a little ahead of ourselves in terms of the ordering of these pages, but let's give it a try. First, the number of ways of finding a defective part is 15 for the first one, and 14 for the second. The number of ways of choosing a good part is 285 for the first one, and 284 for the second. So the total number of ways of selecting these parts – the number of combinations – is: $$n = 15 \cdot 14 \cdot 285 \cdot 284 = 16,997,400$$ But, there's a hitch: It doesn't really matter what order we choose our parts. In fact if we designate defective parts with D and good parts with G, we could choose DDGG, DGGD, GGDD, DGDG , GDGD or GDDG — six different ways that are all really the same. So we need to divide our large number of ways by 6: $$n = \frac{16,997,400}{6} = 2,832,900.$$ Now the probability of drawing four defective parts at once is 1 in that large number, or $$P(DDDD) = \frac{1}{2,832,900} = 3.5 \times 10^{-5} \%$$ c. The probability that none of the parts are defective follows from part (a): $$ \begin{align} P(GGGG) &= \frac{285}{300} \cdot \frac{284}{299} \cdot \frac{283}{298} \cdot \frac{282}{297} \\[5pt] &= 0.81 \end{align}$$ So the probability of grabbing four non-defective parts at once is just over 80%. We'd probably like to do better than that in manufacturing, especially if a life might depend on that part! |
||||||||||||
| 6. |
Two bad lightbulbs are mixed up with six good, new ones. Calculate the probability that if two bulbs are taken at random, they will both be bad. SolutionThe probability of picking one defective bulb is $P(D) = \frac{2}{8}.$ The probability of choosing the remaining bad bulb is $\frac{1}{7},$ for a total probability of $$ \begin{align} P(DD) &= \frac{2}{8} \cdot \frac{1}{7} = \frac{2}{56}\\[5pt] &= \frac{1}{28} \approx 3.5 \% \end{align}$$ |
||||||||||||
| 7. |
A high school has 475 students, in classes given in the table. Calculate P(F), P(S), P(J) and P(R).
SolutionThere are 475 students, so that will be our denominator. Our probabilities are $$P(F) = \frac{121}{475} = 25.5 \%$$ $$P(S) = \frac{130}{475} = 27.4 \%$$ $$P(J) = \frac{118}{475} = 24.8 \%$$ $$P(R) = \frac{106}{475} = 22.3 \%$$ Notice that the sum of these probabilities is 100%. |
||||||||||||
| 8. |
If a coin is flipped three times,
Solutions
|
||||||||||||
| 9. |
Suppose that in a four-child family, the probability that all four children are boys is 0.07. Calculate the probability that a four-child family includes at least one girl. SolutionThe only family that wouldn't count in this situation would be one of four boys. The probability that such a family has at least one girl child [P(1-4G) "the probability of having from 1 to 4 girls"] is the converse of the probability of having all boys: $$ \begin{align} P(\text{1-4G}) &= 1 - P(\text{BBBB}) \\[5pt] &= 1 - 0.7 \\[5pt] &= 0.3 = 30\%. \end{align}$$ |
||||||||||||
| 10. |
Two cards are chosen at ramdom from a 52-card deck. Calculate the probability that the cards are:
Solutionsa. In this problem, we get the first card for "free." That is, it doesn't matter what suit we draw, only that the second card is not of the same suit. For the second card, there are 39 cards of a different suit left out of 51. We have to account for the card we'e already drawn, so there are now 51 cards.The probability is $$P = \frac{39}{51} = 76.5 \%.$$ b. The probability of drawing one ace is 4/52 or 1/13. We aren't replacing the first card in the deck, so the chance of drawing a second one is 3 (the number of aces remaining in the deck) out of 51 (the number of cards left), 3/51. The overall probability is the product of the two: $$ \begin{align} P(AA) &= \frac{4}{52} \cdot \frac{3}{51} \\[5pt] &= \frac{12}{2652} = 0.45 \% \end{align}$$ |
||||||||||||
| 11. |
Two dice are rolled. Calculate the probability that:
Solutions
a. The diagram shows the 36 possibilities for rolls of two dice. Of those, 26 rolls have a sum of 6 or greater, for a probability of $$P(\ge 6) = \frac{26}{36} = \frac{13}{18} = 72 \%$$ b. Half of all of the possibilities in the table have two even numbers showing, so the probability is $$P(EE) = \frac{1}{2} = 50 \%$$ |
||||||||||||
| 12. |
The four DNA bases, adenine, cytosine, guanine and thymine are usually just abbreviated A, C, G and T. These bases are linked on a phosphate-sugar chain to form a strand of DNA. On DNA, every three bases forms a "codon," a three-letter code that encodes one of the 20 amino acids that will later form a protein chain. Assuming that A, C, G and T are equally abundant in a cell, calculate these probabilities:
That any of the codons GGG, GGA, GGC or GGT (all of which encode the amino acid glycine) will be formed. Solutionsa. The probability of forming or finding the ATG sequence in a DNA chain is $$P(ATG) = \frac{1}{4} \cdot \frac{1}{4} \cdot \frac{1}{4} = \frac{1}{64}$$ b. Building the sequence AAAAA requires five independent events to occur in a row, each with probability P(A) = ¼;. So $$P(AAAAA) = P(A)^5 = \frac{1}{4^5} = \frac{1}{1024}$$ It would be rare (~0.1 chance — but not impossible) in a DNA sequence to find so many of one base in a row. c. We showed in A that the chance of getting the ATG sequence is $\frac{1}{64}.$. It's the same probability for obtaining every three-letter sequence, so for this probability we sum the individual probabilities: $$ \begin{align} P(GGG) &+ P(GGA) + P(GGC) + P(GGT) \\[5pt] &= \frac{1}{64} + \frac{1}{64} + \frac{1}{64} + \frac{1}{64} \\[5pt] &= \frac{4}{64} = \frac{1}{16} \end{align}$$ These are all of the codons that encode for glycine, so that means there is a 1/16 chance that an amino acid in a protein chain is a glycine. |
Axiom
An axiom is a statement that is self-evidently true, accepted or long-established, but which can't necessarily be proven so. A famous axiom is Euclid's first postulate, also known as the transitive property: "Things that are equal to the same thing are equal to each other," or If a = b and c = b, then a = c.
Induction
When we reason inductively, or use induction to reason, we extend a pattern. The sun has risen every day of my life, so I'm reasonably confident it will rise again tomorrow. That's induction. To extend a series like 2, 4, 6, 8, x, by concluding that x = 10 is also induction.
xaktly.com by Dr. Jeff Cruzan is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. © 2012, Jeff Cruzan. All text and images on this website not specifically attributed to another source were created by me and I reserve all rights as to their use. Any opinions expressed on this website are entirely mine, and do not necessarily reflect the views of any of my employers. Please feel free to send any questions or comments to jeff.cruzan@verizon.net.