Unit 5: Theory of Probability

1. Basic Concepts
2. Algebra of Events
3. Approaches to Probability
4. Theorems of Probability (Addition & Multiplication)
5. Conditional Probability
6. Independent Events
7. Bayes' Theorem and Its Applications

1. Basic Concepts

Random Experiment

An experiment or process for which the outcome cannot be predicted with certainty, but all possible outcomes are known.

Example: Tossing a coin, rolling a die.

Sample Point

A single possible outcome of a random experiment.

Example: When rolling a die, "4" is a sample point.

Sample Space (S)

The set of all possible outcomes (sample points) of a random experiment.

Example (Rolling a die): S = {1, 2, 3, 4, 5, 6}
Example (Tossing two coins): S = {HH, HT, TH, TT}

Event (E)

A subset of the sample space. It is a collection of one or more sample points.

Example (Rolling a die): Event A (Getting an even number): A = {2, 4, 6}

2. Algebra of Events

Events can be combined using set operations.

A or B (A ∪ B): The event that *at least one* of A or B occurs.
A and B (A ∩ B): The event that *both* A and B occur simultaneously.
Not A (A' or A^c): The complement of A. The event that A does *not* occur.
Mutually Exclusive Events: Two events that cannot occur at the same time. (A ∩ B = ∅, the empty set).
Exhaustive Events: A set of events that covers the entire sample space. (A₁ ∪ A₂ ∪ ... = S).

3. Approaches to Probability

How we define and calculate probability.

1. Classical (or 'a priori') Definition

This definition assumes all outcomes are equally likely.

If a random experiment has 'n' mutually exclusive, exhaustive, and equally likely outcomes, and 'm' of these outcomes are favorable to an event A:

P(A) = m / n = (Number of favorable outcomes) / (Total number of possible outcomes)

Example: P(Rolling a 3 on a die) = 1 / 6.
Limitation: Fails if outcomes are not equally likely (e.g., a biased coin).

2. Statistical (or Empirical / 'a posteriori') Definition

This definition is based on relative frequency from actual experiments.

If an experiment is repeated 'n' times and event A occurs 'f' times, the probability of A is the limit of this relative frequency as 'n' becomes infinitely large.

P(A) = lim (n→∞) [ f / n ]

Example: If we toss a coin 10,000 times and get 5,030 heads, we estimate P(Heads) ≈ 0.503.

3. Axiomatic Definition

This is the modern, mathematical definition. It states the *rules* (axioms) that probability must follow, without saying how to calculate it.

Axiom 1 (Non-negativity): For any event A, P(A) ≥ 0.

Axiom 2 (Certainty): The probability of the entire sample space is 1. P(S) = 1.

Axiom 3 (Additivity): If A and B are mutually exclusive events, then P(A ∪ B) = P(A) + P(B).

4. Theorems of Probability (Addition & Multiplication)

1. Addition Law of Probability

Used to find the probability of (A or B).

General Rule (for any two events):

P(A ∪ B) = P(A) + P(B) - P(A ∩ B)

(We subtract P(A ∩ B) because it was counted twice).

Special Rule (for Mutually Exclusive events):

Since P(A ∩ B) = 0, the formula simplifies to:

P(A ∪ B) = P(A) + P(B)

2. Multiplication Law of Probability

Used to find the probability of (A and B). It is derived from the definition of conditional probability.

P(A ∩ B) = P(A) * P(B | A)
(Probability of A, times the probability of B *given that* A has already happened)

5. Conditional Probability

Conditional Probability, P(A | B): The probability of event A occurring, given that event B has already occurred.

It "updates" the probability of A based on new information. We are restricting our sample space from S down to just B.

Formula:

P(A | B) = P(A ∩ B) / P(B) (provided P(B) > 0)

Example:

Roll a die. S = {1, 2, 3, 4, 5, 6}.
Event A = "Get a 4". P(A) = 1/6.
Event B = "Get an even number" = {2, 4, 6}. P(B) = 3/6.
Event (A ∩ B) = "Get a 4 AND an even number" = {4}. P(A ∩ B) = 1/6.

P(A | B) = P(A ∩ B) / P(B) = (1/6) / (3/6) = 1/3.

6. Independent Events

Independent Events: Two events A and B are independent if the occurrence of one event does not affect the probability of the other event occurring.

Formal Definition:

A and B are independent if and only if:

P(A ∩ B) = P(A) * P(B)

This also means:

P(A | B) = P(A)
P(B | A) = P(B)

Common Mistake: Do NOT confuse "Mutually Exclusive" with "Independent".
- Mutually Exclusive: If A happens, B *cannot* happen. P(A ∩ B) = 0. They are strongly *dependent*.
- Independent: If A happens, it tells you *nothing* about B. P(A ∩ B) = P(A)P(B).

7. Bayes' Theorem and Its Applications

Bayes' Theorem allows us to "reverse" the conditional probability. If we know P(B | A), Bayes' Theorem helps us find P(A | B).

It is used to update our belief about a "cause" (A) given a new "effect" (B).

P(A): Prior Probability (our initial belief about A)
P(B | A): Likelihood (the chance of seeing effect B if cause A is true)
P(A | B): Posterior Probability (our updated belief about A after seeing B)

The Theorem

For two events A and B:

P(A | B) = [ P(B | A) * P(A) ] / P(B)

More generally, if the sample space is partitioned by events A₁, A₂, ..., A_k:

Bayes' Theorem (Full Form):
P(A_i | B) = [ P(B | A_i)P(A_i) ] / [ Σ P(B | A_j)P(A_j) ]

The denominator is just P(B), calculated using the Law of Total Probability.

Application (Example):

A factory has two machines, A1 and A2.
- A1 produces 60% of items (P(A1) = 0.6).
- A2 produces 40% of items (P(A2) = 0.4).
- 2% of items from A1 are defective (P(D | A1) = 0.02).
- 1% of items from A2 are defective (P(D | A2) = 0.01).

Question: An item is selected at random and found to be defective (D). What is the probability it came from machine A1?

We want to find P(A1 | D).

Step 1: Numerator: P(D | A1) * P(A1)
(0.02) * (0.6) = 0.012

Step 2: Denominator (Total Probability of D):
P(D) = P(D | A1)P(A1) + P(D | A2)P(A2)
P(D) = (0.02)(0.6) + (0.01)(0.4)
P(D) = 0.012 + 0.004 = 0.016

Step 3: Divide.
P(A1 | D) = 0.012 / 0.016 = 12 / 16 = 0.75

Result: There is a 75% chance the defective item came from machine A1.