Unit 5: Probability

1. Introduction: Basic Concepts

Probability is the measure of the likelihood that an event will occur. It is quantified as a number between 0 and 1 (where 0 indicates impossibility and 1 indicates certainty).

Random Experiment

An experiment or process for which the outcome cannot be predicted with certainty, but all possible outcomes are known.

• Example: Tossing a coin, rolling a die, drawing a card from a deck.

Sample Point

A single possible outcome of a random experiment.

• Example: When rolling a die, "4" is a sample point.
• Example: When tossing a coin, "Heads" is a sample point.

Sample Space (S)

The set of all possible outcomes (sample points) of a random experiment.

• Example (Rolling a die): S = {1, 2, 3, 4, 5, 6}
• Example (Tossing two coins): S = {HH, HT, TH, TT}

Event (E)

A subset of the sample space. It is a collection of one or more sample points.

• Example (Rolling a die):
  • Event A (Getting an even number): A = {2, 4, 6}
  • Event B (Getting a number > 4): B = {5, 6}

2. Algebra of Events and Types of Events

Events can be combined using set operations.

Algebra of Events (Set Operations)

• A or B (A ∪ B): The event that *at least one* of A or B occurs.
  • A = {2, 4, 6}, B = {5, 6} => A ∪ B = {2, 4, 5, 6}
• A and B (A ∩ B): The event that *both* A and B occur simultaneously.
  • A = {2, 4, 6}, B = {5, 6} => A ∩ B = {6}
• Not A (A' or A^c): The complement of A. The event that A does *not* occur.
  • S = {1, 2, 3, 4, 5, 6}, A = {2, 4, 6} => A' = {1, 3, 5}

Types of Events

1. Mutually Exclusive (or Disjoint) Events

Two events that cannot occur at the same time. They have no sample points in common.

• Condition: A ∩ B = ∅ (the empty set)
• Example: When rolling a die, the events "Getting a 1" and "Getting a 6" are mutually exclusive. You can't get both on a single roll.

2. Exhaustive Events

A set of events that covers the entire sample space. When the experiment is performed, at least one of these events must occur.

• Condition: A₁ ∪ A₂ ∪ ... ∪ A_k = S
• Example: When rolling a die, the events E1="Even" {2,4,6} and E2="Odd" {1,3,5} are exhaustive, because their union is {1,2,3,4,5,6} = S.

3. Equally Likely Events

Outcomes that have the same chance of occurring.

• Example: In a fair coin toss, "Heads" and "Tails" are equally likely.
• Example: In a fair die roll, {1}, {2}, {3}, {4}, {5}, {6} are all equally likely.

3. Classical (Statistical) Definition of Probability

This is the first and most intuitive definition of probability.

If a random experiment has 'n' mutually exclusive, exhaustive, and equally likely outcomes, and 'm' of these outcomes are favorable to an event A, then the probability of event A is:

Example:

Experiment: Drawing one card from a standard 52-card deck.

Event A: Drawing a King.

• Total outcomes (n) = 52 (all cards are equally likely)
• Favorable outcomes (m) = 4 (there are 4 Kings)
• P(A) = 4 / 52 = 1 / 13

Limitations of Classical Definition:

• It only works if outcomes are equally likely. (e.g., cannot be used for a biased coin).
• It only works if the sample space is finite (n must be a finite number).

4. Axiomatic Definition of Probability

This is the modern, mathematical definition of probability (developed by Kolmogorov). It does not say *how* to calculate probability, but states the *rules* (axioms) that any probability measure P must follow.

Given a sample space S, a probability P(A) is a number assigned to every event A that satisfies these three axioms:

Axiom 1 (Non-negativity): For any event A, P(A) ≥ 0.
(Probability can never be negative).

Axiom 2 (Certainty): The probability of the entire sample space is 1.
P(S) = 1.
(Something *must* happen).

Axiom 3 (Additivity): If A and B are two mutually exclusive events (A ∩ B = ∅), then the probability of their union is the sum of their individual probabilities.
P(A ∪ B) = P(A) + P(B)

From these 3 simple axioms, all other probability rules can be derived, such as:

• P(∅) = 0 (Probability of an impossible event is 0)
• P(A') = 1 - P(A) (Probability of the complement)
• 0 ≤ P(A) ≤ 1 (Probability is always between 0 and 1)

5. Laws of Addition and Multiplication

1. Addition Law of Probability

This rule is used to find the probability of (A or B).

General Rule (for *any* two events):

(We subtract P(A ∩ B) because it was counted twice - once in P(A) and once in P(B)).

Special Rule (for *Mutually Exclusive* events):

If A and B are mutually exclusive, then P(A ∩ B) = 0. The formula simplifies to:

2. Multiplication Law of Probability

This rule is used to find the probability of (A and B). It is derived from the definition of conditional probability.

A special case for *Independent* events is covered below.

6. Conditional Probability

Conditional Probability, P(A | B): The probability of event A occurring, given that event B has already occurred.

It "updates" the probability of A based on new information (that B happened). We are restricting our sample space from S down to just B.

Formula:

Example:

Experiment: Roll a fair die (S = {1, 2, 3, 4, 5, 6}).

Event A: Getting a "4". P(A) = 1/6.

Event B: Getting an "even number". B = {2, 4, 6}. P(B) = 3/6.

Question: What is the probability of getting a 4, *given that* we know the result was an even number? We want to find P(A | B).

• A ∩ B (Getting a 4 AND an even number) = {4}. So, P(A ∩ B) = 1/6.
• P(A | B) = P(A ∩ B) / P(B) = (1/6) / (3/6) = 1/3.

This makes sense: if we know the outcome is even {2, 4, 6}, the chance of it being the "4" is 1 out of 3.

7. Independent Events

Independent Events: Two events A and B are independent if the occurrence of one event does not affect the probability of the other event occurring.

Formal Definition:

A and B are independent if and only if:

This also means:

• P(A | B) = P(A) (Knowing B happened doesn't change the probability of A)
• P(B | A) = P(B) (Knowing A happened doesn't change the probability of B)

Example:

Experiment: Toss a fair coin twice.

Event A: Get Heads on the 1st toss. P(A) = 1/2.

Event B: Get Heads on the 2nd toss. P(B) = 1/2.

Event (A ∩ B): Get Heads on both. S = {HH, HT, TH, TT}. P(A ∩ B) = 1/4.

Are they independent? Let's check:
Does P(A ∩ B) = P(A) * P(B)?
1/4 = (1/2) * (1/2)
1/4 = 1/4. Yes. The events are independent.

8. Theorem of Total Probability

This theorem is used to find the probability of an event (B) by considering all the possible ways it can happen. It relies on partitioning the sample space.

Let's say the sample space S is partitioned into 'k' mutually exclusive and exhaustive events (A₁, A₂, ..., A_k). This means one (and only one) of the A_i events must occur.

Now, to find the total probability of an event B, we sum up the probabilities of B happening *within each partition*:

Example:

Two urns, A1 and A2.

• Urn A1 has 2 White, 3 Black balls.
• Urn A2 has 4 White, 1 Black ball.

We choose an urn at random (P(A1) = 0.5, P(A2) = 0.5) and then draw one ball (Event B = "Ball is White"). What is P(B)?

• P(B | A1) = Probability of White, given Urn A1 = 2/5
• P(B | A2) = Probability of White, given Urn A2 = 4/5

Using the theorem:

P(B) = P(B | A1)P(A1) + P(B | A2)P(A2)

P(B) = (2/5)(0.5) + (4/5)(0.5) = (1/5) + (2/5) = 3/5

The total probability of getting a white ball is 3/5.

9. Bayes' Theorem and Its Applications

Bayes' Theorem is one of the most important concepts in probability. It allows us to "reverse" the conditional probability.

If we know P(B | A), Bayes' Theorem helps us find P(A | B).

• Prior Probability, P(A): Our initial belief about the probability of A *before* we get new evidence.
• New Evidence, B: We observe that event B has happened.
• Posterior Probability, P(A | B): Our *updated* belief about the probability of A, *given* that B has happened.

The Theorem

Let A₁, A₂, ..., A_k be a partition of the sample space. Let B be any event. We want to find the probability of a specific partition, A_i, given that B has occurred.

We can expand the denominator P(B) using the Theorem of Total Probability:

Bayes' Theorem (Full Form):
P(A_i | B) = [ P(B | A_i)P(A_i) ] / [ Σ P(B | A_j)P(A_j) ]

Application / Example (Classic Disease Test):

A disease (D) affects 1% of the population. P(D) = 0.01.
The other 99% are healthy (H). P(H) = 0.99.

There is a test (T).
- If you have the disease, the test is positive 95% of the time. (P(T+ | D) = 0.95)
- If you are healthy, the test is positive 2% of the time (a "false positive"). (P(T+ | H) = 0.02)

Question: You test positive (T+). What is the probability you *actually have the disease*?
We want to find P(D | T+).

Step 1: Identify the pieces.
- A₁ = D (Disease), A₂ = H (Healthy)
- B = T+ (Test is positive)
- P(D) = 0.01 (Prior)
- P(H) = 0.99 (Prior)
- P(T+ | D) = 0.95 (Likelihood)
- P(T+ | H) = 0.02 (Likelihood)

Step 2: Calculate the numerator: P(T+ | D) * P(D)
(0.95) * (0.01) = 0.0095

Step 3: Calculate the total denominator, P(T+), using Total Probability.
P(T+) = P(T+ | D)P(D) + P(T+ | H)P(H)
P(T+) = (0.95)(0.01) + (0.02)(0.99)
P(T+) = 0.0095 + 0.0198
P(T+) = 0.0293

Step 4: Divide Numerator by Denominator.
P(D | T+) = 0.0095 / 0.0293 ≈ 0.324