v1 · padrão canônico

Lesson 79 — Deep Dive into Bayes' Theorem

Priors, posteriors, and sequential updating. Odds form, Beta-binomial conjugate prior, base rate fallacy, Naive Bayes. Applications in medical diagnosis, spam filtering, and ML.

Used in: Stochastik LK alemão · H2 Math Statistics singapurense · Math B japonês · Equiv. AP Statistics EUA

P(H \mid E) = \frac{P(E \mid H)\,P(H)}{P(E)}

Bayes' theorem is the rule for rational belief updating. The prior $P(H)$ represents what we believe before seeing the evidence; the likelihood $P(E \mid H)$ measures how much the evidence favors the hypothesis; the posterior $P(H \mid E)$ is the updated belief after observing $E$ . The denominator $P(E)$ normalizes the result so that the probability sums to 1.

Choose your door

Rigorous notation, full derivation, hypotheses

Definitions and theorems

Conditional probability

"The conditional probability $P(E \mid F)$ , the probability of $E$ given $F$ , expresses the probability of $E$ when we know that $F$ has occurred. It can be computed using the formula $P(E \mid F) = P(EF)/P(F)$ , assuming $P(F) > 0$ ." — Grinstead & Snell, Introduction to Probability, §4.1

Law of total probability

Bayes' theorem

"Bayes' Theorem is just a formula that comes from the definition of conditional probability. Yet it is extremely powerful, and is the key to understanding what it means to rationally revise your beliefs in light of new evidence." — OpenIntro Statistics 4e, §3.2

Odds form

Sequential updating

Beta-binomial conjugate prior

Definition· Conjugate prior

The prior $\pi(\theta)$ is conjugate to the likelihood $L(\theta \mid \mathbf{x})$ if the posterior $\pi(\theta \mid \mathbf{x})$ belongs to the same parametric family as the prior.

For the Bernoulli model: if $X_1, \ldots, X_n \overset{\text{iid}}{\sim} \text{Bernoulli}(\theta)$ and $k = \sum X_i$ , with prior $\theta \sim \text{Beta}(\alpha, \beta)$ :

$\theta \mid k \sim \text{Beta}(\alpha + k,\; \beta + n - k)$

Prior Beta(1,1) = uniform on $[0,1]$ (non-informative prior). Posterior mean: $(\alpha + k)/(\alpha + \beta + n)$ .

SVG — Bayes diagram in 2×2 table

Absolute frequency diagram. The PPV (Positive Predictive Value) is the Bayesian posterior P(sick | positive test). When prevalence is low, false positives outweigh true positives even with a high-quality test.

Solved examples

Example— 1· Direct calculation with law of total probability (application)

Problem: A factory has three production lines: A (40% of production), B (35%), and C (25%). Defect rates are: A: 2%, B: 3%, C: 5%. A part is randomly selected and is defective. What is the probability it was produced by line B?

Strategy: Apply the law of total probability to calculate $P(\text{defect})$ , then use Bayes' theorem to obtain $P(B \mid \text{defect})$ .

Resolution:

Define events: $H_A$ , $H_B$ , $H_C$ = part comes from line A, B, C. $D$ = defective part.
Priors: $P(H_A) = 0.40$ , $P(H_B) = 0.35$ , $P(H_C) = 0.25$ .
Likelihoods: $P(D \mid H_A) = 0.02$ , $P(D \mid H_B) = 0.03$ , $P(D \mid H_C) = 0.05$ .
Law of total probability: $P(D) = 0.02 \times 0.40 + 0.03 \times 0.35 + 0.05 \times 0.25 = 0.008 + 0.0105 + 0.0125 = 0.031$
Bayes for line B: $P(H_B \mid D) = \frac{0.03 \times 0.35}{0.031} = \frac{0.0105}{0.031} \approx 0.339$

Verification: Let's also calculate for A and C: $P(H_A \mid D) = 0.008/0.031 \approx 0.258$ ; $P(H_C \mid D) = 0.0125/0.031 \approx 0.403$ . Sum: $0.258 + 0.339 + 0.403 = 1.000$ . Correct.

Source. Grinstead & Snell — Introduction to Probability §4.1 — GNU FDL. (Problem adapted from the structure of Example 4.11, three causes with different likelihoods.)

Example— 2· Base rate fallacy and PPV (application)

Problem: Disease X affects 0.5% of the population. Diagnostic test: 95% sensitivity, 90% specificity. A patient tests positive. What is the positive predictive value?

Strategy: Use absolute frequencies in a sample of 10,000 people — method recommended by OpenIntro Statistics to avoid intuition errors.

Resolution:

In 10,000 people:

Sick: $10,000 \times 0.005 = 50$ .
True positives (TP): $50 \times 0.95 = 47.5 \approx 48$ (rounded).
Healthy: $10,000 - 50 = 9,950$ .
False positives (FP): $9,950 \times (1 - 0.90) = 9,950 \times 0.10 = 995$ .
Total positives: $48 + 995 = 1,043$ .
$\text{PPV} = 48/1043 \approx 4.6\%$ .

Verification: Via direct formula: $\text{PPV} = \frac{0.95 \times 0.005}{0.95 \times 0.005 + 0.10 \times 0.995} = \frac{0.00475}{0.00475 + 0.0995} = \frac{0.00475}{0.10425} \approx 4.6\%$

Matches. The intuition of "95% chance of being sick" is wrong by a factor of 20 — classic illustration of the base rate fallacy.

Source. OpenIntro Statistics 4e §3.2 — CC-BY-SA. (Structure of Example 3.10, medical diagnosis with low prevalence.)

Example— 3· Urn with two causes — classic form (application)

Problem: Urn A contains 3 red balls and 2 blue. Urn B contains 1 red ball and 4 blue. An urn is chosen at random (50%-50%) and a ball is drawn. The ball is red. What is the probability the chosen urn was A?

Strategy: Two hypotheses ( $H_A$ and $H_B$ ), uniform prior, evidence = red ball. Apply Bayes directly.

Resolution:

Priors: $P(H_A) = P(H_B) = 0.5$ .
Likelihoods: $P(R \mid H_A) = 3/5 = 0.60$ ; $P(R \mid H_B) = 1/5 = 0.20$ .
Total probability of red ball: $P(R) = 0.60 \times 0.50 + 0.20 \times 0.50 = 0.30 + 0.10 = 0.40$
Posterior: $P(H_A \mid R) = \frac{0.60 \times 0.50}{0.40} = \frac{0.30}{0.40} = 0.75$

Verification: $P(H_B \mid R) = 0.10/0.40 = 0.25$ . Sum $= 1$ . Urn A has red balls in a proportion 3 times higher than Urn B, so it makes sense that the posterior of A is 3 times that of B ( $0.75 = 3 \times 0.25$ ).

Source. Grinstead & Snell — Introduction to Probability §4.1 — GNU FDL. (Adapted from Exercise 4.1.1 on two urns.)

Example— 4· Sequential updating with two tests (intermediate)

Problem: Disease prevalence: 2%. Two independent tests: Test 1 with 90% sensitivity and 95% specificity; Test 2 with 85% sensitivity and 92% specificity. Both are positive. What is the posterior after both positive results?

Strategy: Apply Bayes sequentially: the posterior of Test 1 becomes the prior for Test 2.

Resolution:

Step 1 — after Test 1 positive: $P(D \mid T_1^+) = \frac{0.90 \times 0.02}{0.90 \times 0.02 + 0.05 \times 0.98} = \frac{0.018}{0.018 + 0.049} = \frac{0.018}{0.067} \approx 0.269$

Step 2 — after Test 2 positive (prior = 0.269): $P(D \mid T_1^+, T_2^+) = \frac{0.85 \times 0.269}{0.85 \times 0.269 + 0.08 \times 0.731} = \frac{0.229}{0.229 + 0.0585} = \frac{0.229}{0.287} \approx 0.798$

Verification by odds form:

Prior odds: $0.02/0.98 \approx 0.0204$ .
$\text{LR}_1^+ = 0.90/0.05 = 18$ ; $\text{LR}_2^+ = 0.85/0.08 = 10.625$ .
Posterior odds: $0.0204 \times 18 \times 10.625 = 3.898$ .
Posterior: $3.898/(1 + 3.898) \approx 79.6\%$ . Confirms the calculation above.

Source. OpenIntro Statistics 4e §3.3 — CC-BY-SA. (Extension of Example 3.13 on sequential updating with two independent tests.)

Example— 5· Beta-binomial conjugate prior (advanced)

Problem: A quality control uses prior Beta(2, 8) for the defect rate $\theta$ of a production line (equivalent to "we historically observed 2 defects in 10 inspections"). In a new batch, 20 parts are inspected and 4 are found defective. Determine: (a) the posterior, (b) the posterior mean, (c) an approximate 90% credible interval.

Strategy: Use the Beta-Binomial conjugation property. Posterior = Beta( $\alpha + k$ , $\beta + n - k$ ). For the interval, use the normal approximation of the Beta distribution for moderate parameters.

Resolution:

(a) Posterior: Prior Beta(2, 8), $n = 20$ , $k = 4$ . $\theta \mid 4 \sim \text{Beta}(2 + 4,\; 8 + 20 - 4) = \text{Beta}(6, 24)$

(b) Posterior mean: $\mu = \alpha/(\alpha + \beta) = 6/(6 + 24) = 6/30 = 0.20$ .

Compare with prior: $\mu_{\text{prior}} = 2/10 = 0.20$ . The MLE would be $k/n = 4/20 = 0.20$ . In this case they coincide because the prior was constructed with the same proportions.

(c) 90% credible interval: The Beta(6, 24) distribution has standard deviation $\sigma = \sqrt{\alpha\beta/((\alpha+\beta)^2(\alpha+\beta+1))} = \sqrt{6 \times 24/(900 \times 31)} \approx \sqrt{144/27900} \approx 0.0718$ .

Approximate interval $\mu \pm 1.645\sigma$ : $[0.20 - 0.118, 0.20 + 0.118] = [0.082, 0.318]$ . (The exact interval, by quantiles of Beta(6,24), is approximately $[0.090, 0.338]$ .)

Verification: As $n \to \infty$ , the influence of the prior decreases and the posterior concentrates on the MLE. With prior Beta(2,8) and 200 parts observed with 40 defects, the posterior would be Beta(42, 168) with mean $42/210 \approx 0.20$ and standard deviation $\approx 0.028$ — much more concentrated.

Source. OpenIntro Statistics 4e §3.4 — CC-BY-SA. (Structure of the exercise on Bayesian inference with conjugate prior, introductory Bayesian inference section.)

Exercise list

40 exercises · 10 with worked solution (25%)

Application 18Understanding 4Modeling 10Challenge 5Proof 3

Sources

Grinstead, C.M. & Snell, J.L. — Introduction to Probability (2nd ed.) · GNU FDL · Dartmouth College. Chapter 4 (§4.1): Conditional probability, independence, Bayes' theorem — primary source for most urn, coin, and proof exercises in this lesson.
Diez, D.M., Çetinkaya-Rundel, M., Barr, C.D. — OpenIntro Statistics (4th ed.) · CC-BY-SA · OpenIntro. Sections §3.2–3.4: conditional probability, Bayes, frequency tables, and Bayesian updating — source for PPV, sequential updating, and conjugate prior exercises.
Illowsky, B. & Dean, S. — Statistics (OpenStax) · CC-BY · OpenStax. Section §3.4 (Contingency Tables and Probability Trees): medical diagnosis, spam filtering, and probability trees — basis for Naive Bayes and fraud exercises.