v1 · padrão canônico

Lesson 109 — Introductory Bayesian statistics

Prior, likelihood, posterior. Bayes's rule. Beta-Bernoulli conjugates. MAP versus MLE. Credible interval. Introduction to inference under the Bayesian paradigm.

Used in: Stochastik LK (Alemanha, Klasse 12) · H2 Math Statistics (Singapura) · AP Statistics (EUA)

P(\theta \mid D) = \frac{P(D \mid \theta)\,P(\theta)}{P(D)}

Bayes's rule updates our belief about the parameter $\theta$ after observing data $D$ . The posterior $P(\theta \mid D)$ is proportional to the product of the likelihood $P(D \mid \theta)$ and the prior $P(\theta)$ , normalized by the evidence $P(D)$ .

Choose your door

Rigorous notation, full derivation, hypotheses

Rigorous definition

Bayes's theorem

Definition· Bayes's rule (general form)

Let $\theta$ be a parameter (or hypothesis) and $D$ the observed data. Definitions:

$P(\theta)$ — prior: probability distribution of the parameter before observing data.
$P(D \mid \theta)$ — likelihood: probability of the data given the parameter.
$P(D)$ — marginal evidence: $P(D) = \int P(D \mid \theta) P(\theta)\, d\theta$ (or discrete sum).
$P(\theta \mid D)$ — posterior: updated distribution after observing $D$ .

Bayes's rule:

P(\theta \mid D) = \frac{P(D \mid \theta)\, P(\theta)}{P(D)} \propto P(D \mid \theta)\, P(\theta)

what this means · The posterior is proportional to the likelihood times the prior. The denominator P(D) is merely a normalization constant.

"Bayes's theorem is a basic result of conditional probability, but its interpretation changes everything: it offers a formal recipe for updating beliefs in light of evidence." — OpenIntro Statistics §3.6

Conjugate priors: the Beta-Bernoulli case

Definition· Beta-Bernoulli conjugate family

When $X_1, \ldots, X_n \stackrel{\text{iid}}{\sim} \text{Bernoulli}(\theta)$ and the prior is $\theta \sim \text{Beta}(\alpha, \beta)$ , the posterior has closed form:

\theta \mid D \;\sim\; \text{Beta}(\alpha + s,\; \beta + n - s)

what this means · After s successes in n trials, the posterior is another Beta with updated parameters.

where $s = \sum_i X_i$ is the number of successes. The Beta is conjugate for the Bernoulli: prior and posterior belong to the same family.

The Beta distribution has density:

f(\theta; a, b) = \frac{\theta^{a-1}(1-\theta)^{b-1}}{B(a,b)}, \quad \theta \in [0, 1]

what this means · Density of Beta(a,b): defined on [0,1], controlled by parameters a and b.

with $B(a, b) = \Gamma(a)\Gamma(b)/\Gamma(a+b)$ .

Point estimators

Definition· MAP and MLE

Given the posterior $P(\theta \mid D)$ :

MAP (maximum a posteriori): $\hat\theta_{\text{MAP}} = \arg\max_\theta P(\theta \mid D)$ . Maximizes the posterior — includes prior information.
MLE (maximum likelihood): $\hat\theta_{\text{MLE}} = \arg\max_\theta P(D \mid \theta)$ . Maximizes the likelihood — ignores the prior.
Posterior mean (quadratic Bayes estimator): $E[\theta \mid D]$ .

Relationship: With uniform prior ( $P(\theta) = c$ ), MAP = MLE. With $n \to \infty$ , the posterior concentrates at $\hat\theta_{\text{MLE}}$ — the prior becomes irrelevant as data increases.

For Beta-Bernoulli:

\hat\theta_{\text{MAP}} = \frac{\alpha + s - 1}{\alpha + \beta + n - 2}, \quad E[\theta \mid D] = \frac{\alpha + s}{\alpha + \beta + n}

what this means · MAP and posterior mean for the Beta-Bernoulli case. Compare with MLE = s/n (empirical proportion).

Credible interval

Bayes factor

Bayesian flow: prior × likelihood → posterior. The posterior becomes the new prior when more data arrives.

Worked examples

Example— 1· Discrete Bayes's rule — diagnosis (basic)

Problem. A disease affects 2% of the population. A test has sensitivity 90% and specificity 85%. A patient tests positive. What is the probability that they have the disease?

Strategy. Apply Bayes's formula with partition $\{D, \neg D\}$ and calculate $P(D \mid +)$ .

Solution.

$P(D) = 0.02$ , $P(\neg D) = 0.98$ , $P(+ \mid D) = 0.90$ , $P(+ \mid \neg D) = 0.15$ .

Total evidence: $P(+) = P(+ \mid D)\,P(D) + P(+ \mid \neg D)\,P(\neg D)$ $= 0.90 \times 0.02 + 0.15 \times 0.98 = 0.018 + 0.147 = 0.165$

Posterior: $P(D \mid +) = \frac{0.90 \times 0.02}{0.165} = \frac{0.018}{0.165} \approx 0.109 = 10.9\%$

Check. With only 2% prevalence, even a relatively good test generates many false positives. The answer $\approx 11\%$ makes sense: most positives come from the huge healthy population.

Source. OpenIntro Statistics §3.6, medical diagnosis example — CC-BY-SA.

Example— 2· Sequential Beta-Bernoulli update (intermediate)

Problem. An urn has unknown proportion $\theta$ of red balls. Prior: $\theta \sim \text{Beta}(2, 2)$ . Sampling with replacement: 1st sample — 3 red out of 5 draws; 2nd sample — 4 red out of 6 draws. Calculate the posterior after each sample and the final posterior mean.

Strategy. Beta-Bernoulli: after $s$ successes in $n$ trials, $\text{Beta}(\alpha, \beta) \to \text{Beta}(\alpha+s, \beta+n-s)$ . Apply iteratively.

Solution.

Prior: $\text{Beta}(2, 2)$ , mean $= 2/4 = 0.50$ .

After 1st sample ( $s = 3$ , $n = 5$ ): $\text{Beta}(2+3,\; 2+5-3) = \text{Beta}(5, 4), \quad E[\theta] = \frac{5}{9} \approx 0.556$

After 2nd sample ( $s = 4$ , $n = 6$ ): $\text{Beta}(5+4,\; 4+6-4) = \text{Beta}(9, 6), \quad E[\theta] = \frac{9}{15} = 0.600$

Check. Total data: 7 red out of 11 draws, sample proportion = $7/11 \approx 0.636$ . Posterior mean 0.60 is between the prior (0.50) and sample proportion — makes sense. With weak prior, the posterior converges to the MLE as $n$ grows.

Source. Think Bayes §3 — Allen Downey — CC-BY-NC-SA.

Example— 3· MAP vs MLE vs posterior mean (intermediate)

Problem. For the Beta-Bernoulli model with $\alpha = 3$ , $\beta = 3$ , after 6 successes in 10 trials, calculate the MLE, the MAP, and the posterior mean. Interpret the difference.

Strategy. MLE maximizes the likelihood; MAP maximizes the posterior; posterior mean is $E[\theta \mid D]$ .

Solution.

MLE: $\hat\theta_{\text{MLE}} = s/n = 6/10 = 0.600$ .

Posterior: $\text{Beta}(3+6, 3+4) = \text{Beta}(9, 7)$ .

MAP (mode of Beta $(a, b)$ with $a, b > 1$ is $(a-1)/(a+b-2)$ ): $\hat\theta_{\text{MAP}} = \frac{9 - 1}{9 + 7 - 2} = \frac{8}{14} \approx 0.571$

Posterior mean: $E[\theta \mid D] = \frac{9}{9 + 7} = \frac{9}{16} = 0.5625$

Check. Ordering: mean (0.5625) between MAP (0.571) and the mode of Beta(9,7). MLE (0.60) is largest — the prior "pulls" toward 0.5 (prior symmetric around 0.5). With large $n$ , all three converge to the MLE.

Source. Think Bayes §4, §6 — Allen Downey — CC-BY-NC-SA.

Example— 4· 95% credible interval Beta (advanced)

Problem. After 12 successes in 20 trials with prior $\text{Beta}(1, 1)$ (uniform), calculate the central 95% credible interval for $\theta$ .

Strategy. Posterior $\text{Beta}(13, 9)$ . The central 95% interval is given by the 2.5th and 97.5th percentiles of the Beta distribution.

Solution.

Posterior: $\text{Beta}(1+12, 1+8) = \text{Beta}(13, 9)$ .

Posterior mean: $13/(13+9) = 13/22 \approx 0.591$ .

By table or software (R: qbeta(c(0.025, 0.975), 13, 9)):

2.5th percentile: $\approx 0.376$ . 97.5th percentile: $\approx 0.779$ .

95% credible interval: $(0.376;\; 0.779)$ .

Check. Direct interpretation: "given the uniform prior and the data, the probability that $\theta$ lies between 0.376 and 0.779 is 95%". Note that the interval is not centered at 0.6 — it is asymmetric because Beta is asymmetric in this case.

Source. Introduction to Probability §4.1 — Grinstead & Snell — GNU FDL.

Example— 5· Bayes factor — hypothesis comparison (proof)

Problem. To test whether $\theta = 0.5$ (fair coin) versus $\theta = 0.7$ (biased coin), with equal prior ( $P(H_0) = P(H_1) = 0.5$ ), calculate the Bayes factor and the posterior probability of $H_1$ after 8 heads in 10 flips.

Strategy. Calculate $P(D \mid H_i)$ for each hypothesis, then apply Bayes.

Solution.

$P(D \mid H_0) = \binom{10}{8}(0.5)^{10} = 45 \times \frac{1}{1024} \approx 0.0439$

$P(D \mid H_1) = \binom{10}{8}(0.7)^8(0.3)^2 = 45 \times 0.05765 \times 0.09 \approx 0.2335$

Bayes factor: $BF_{10} = \frac{0.2335}{0.0439} \approx 5.32$

With prior $P(H_0) = P(H_1) = 0.5$ : $P(H_1 \mid D) = \frac{BF_{10}}{1 + BF_{10}} = \frac{5.32}{6.32} \approx 0.842$

Check. $BF_{10} \approx 5.32$ — moderate to strong evidence for $H_1$ (Jeffreys scale: between 3 and 10 is "moderate"). The posterior probability of biased coin increased from 50% to 84%. Coherent with the data (8 of 10 favors $\theta = 0.7$ ).

Source. OpenIntro Statistics §3.7 — Diez, Çetinkaya-Rundel, Barr — CC-BY-SA.

Exercise list

40 exercises · 10 with worked solution (25%)

Application 29Understanding 3Modeling 5Challenge 1Proof 2

Sources

Think Bayes — Allen B. Downey · CC-BY-NC-SA · Greenteapress · Chapters 1–9.
Introduction to Probability — Grinstead & Snell · GNU FDL · Dartmouth · §4.1.
OpenIntro Statistics — Diez, Çetinkaya-Rundel, Barr · CC-BY-SA · OpenIntro · §3.6–3.7.