Lesson 15: Significance Tests for a ProportionAP

A significance test asks: if nothing interesting were true, how surprising is our sample? If the data would be very unlikely under the boring hypothesis, we reject the boredom. CED Unit 6 formalises this for proportions: hypotheses about a parameter, a test statistic standardising the estimate, and a p-value measuring surprise — plus the error vocabulary (Type I, Type II, power) that keeps conclusions honest.

Hypotheses and the test statistic

The null hypothesis H0 states the parameter equals a claimed value (p = p0) — the "no effect / no difference / claim is true" position. The alternative Ha states the parameter lies in a direction: p < p0, p > p0, or p ≠ p0 (chosen *before* seeing data, from the question's wording — "evidence that the rate has *decreased*" means Ha: p < p0). Hypotheses are always about the parameter, never about p̂.

The test statistic measures how far the estimate sits from the claim, in standard-error units computed under H0: z = (p̂ − p0) / √(p0(1−p0)/n). Note the standard error uses p0, not p̂ — the test asks "in the world where H0 is true, how variable would p̂ be?" This is the mirror image of the confidence interval's standard error (p̂), and mixing them up is the Unit 6 signature error. Conditions mirror Lesson 13 but with p0: Random; Normal via np0 ≥ 10 and n(1−p0) ≥ 10; Independent (10% condition).

The p-value is the probability, *assuming H0 is true*, of obtaining a sample proportion at least as extreme as the one observed, in the direction of Ha. Small p-value → the data are surprising under H0 → reject H0 when p-value < α (the significance level, usually 0.05). Large p-value → fail to reject H0 (never "accept" or "prove" H0).

Worked example 1. A manufacturer claims 90% of its batteries pass a stress test. An inspector samples 200 batteries and 168 pass (p̂ = 0.84). Test the claim against Ha: p < 0.90.

• State: H0: p = 0.90 vs Ha: p < 0.90, where p = true proportion of all batteries that pass.
• Plan: one-proportion z-test; conditions — random sample (stated), np0 = 180 ≥ 10 and n(1−p0) = 20 ≥ 10 ✓, sample under 10% of production ✓.
• Do: SE = √(0.90 × 0.10/200) = 0.0212; z = (0.84 − 0.90)/0.0212 ≈ −2.83; p-value ≈ 0.0023.
• Conclude: 0.0023 < 0.05, so reject H0. There is convincing evidence that the true pass rate is less than the claimed 90% — in context, with the direction, every time.

Worked example 2 (errors in context). For the battery test: a Type I error would be rejecting H0 when it is true — concluding the pass rate is below 90% when it really is 90% (consequence: the manufacturer is wrongly accused, production is needlessy overhauled). A Type II error would be failing to reject when the rate truly is below 90% (consequence: a defective production process keeps shipping weak batteries). The probability of a Type I error is α, set by us. Power = 1 − P(Type II) is the chance the test detects a real shortfall; power grows with larger n, larger true discrepancy, and larger α.

Real-world application

Clinical trials are significance tests with lives at stake: H0 says a drug works no better than placebo, and regulators demand p < 0.05 *and* replication before approval, precisely because a Type I error (approving a useless drug) and a Type II error (shelving a useful one) both harm people. Online A/B testing runs the same logic thousands of times a day — and suffers the same temptation to peek early, which is just choosing Ha (and α) after seeing data.

Common mistakes

• Concluding "accept H0" or "H0 is proven." Failing to reject means the evidence was insufficient, not that the claim is true — a small sample fails to reject almost everything.
• Defining the p-value as "the probability H0 is true." It is a probability about the *data*, computed *assuming* H0: P(data at least this extreme | H0).
• Using p̂ in the test's standard error (see above) — or p0 in a confidence interval's. Keep the two SEs straight: tests assume the claim; intervals estimate from data.
• Choosing the alternative after inspecting the data, or reporting significance without a conclusion in context ("reject H0" with no sentence about batteries).
• Equating "statistically significant" with "large/important." With n = 1,000,000, a trivially small difference is highly significant; significance is about evidence against H0, not effect size.

AP Exam Experience

Question 1. A school claims 75% of its students graduate on time. A sceptical board member suspects the true rate is lower. The correct hypotheses are:

• (A) H0: p̂ = 0.75 vs Ha: p̂ < 0.75
• (B) H0: p = 0.75 vs Ha: p < 0.75
• (C) H0: p = 0.75 vs Ha: p ≠ 0.75
• (D) H0: p < 0.75 vs Ha: p = 0.75
• (E) H0: μ = 0.75 vs Ha: μ < 0.75

Answer: (B). Hypotheses concern the parameter p (never p̂, ruling out A; never μ for a proportion, ruling out E). "Suspects lower" fixes the one-sided direction before data are examined, ruling out (C); H0 always carries the equality, ruling out (D).

Question 2. A test of H0: p = 0.5 gives a p-value of 0.03. Which interpretation is correct?

• (A) There is a 3% probability that H0 is true.
• (B) If H0 is true, the probability of a sample result at least as extreme as ours is 3%.
• (C) There is a 3% probability the result occurred by chance, so H0 is false.
• (D) The probability of a Type I error is 0.03.
• (E) We can be 97% confident H0 is false.

Answer: (B). That is the definition, word for word. (A) and (E) attach probabilities to the hypothesis; (C) smuggles in a conclusion ("so H0 is false") the p-value does not deliver; (D) confuses the p-value with α, the pre-set Type I error rate.

Question 3. In a test of a new allergy drug's effectiveness (H0: the drug is no better than placebo), a Type I error would mean:

• (A) Concluding the drug works when it actually does not
• (B) Concluding the drug does not work when it actually does
• (C) Approving the drug because the sample was too small
• (D) The drug works, and we correctly conclude it works
• (E) The sample proportion differs from the population proportion

Answer: (A). Type I = rejecting a true H0: we announce an effect that is not really there. (B) is the Type II error. Errors must always be stated *in the context of the hypotheses* — generic definitions without the drug earn partial credit at best.

Free-response question. A district reports that 75% of its students graduate on time. A journalist takes a random sample of 120 students and finds that 82 graduated on time. Test, at the 5% level, whether there is convincing evidence that the district's true graduation rate differs from 75%.

Scoring guideline.

• State: Essentially correct if H0: p = 0.75 and Ha: p ≠ 0.75 with p defined as the true proportion of all students in the district who graduate on time. (Two-sided, because the prompt says "differs".)
• Plan: Essentially correct if a one-proportion z-test is named AND conditions verified: random sample (stated); np0 = 120 × 0.75 = 90 ≥ 10 and n(1−p0) = 30 ≥ 10; 120 is under 10% of the district's students. Partially correct if the test is named but a condition check is missing or uses p̂ counts.
• Do: Essentially correct if p̂ = 82/120 ≈ 0.683, SE = √(0.75 × 0.25/120) ≈ 0.0395 (using p0 — the point of the exercise), z ≈ (0.683 − 0.75)/0.0395 ≈ −1.69, and the two-sided p-value ≈ 0.092. Partially correct if SE uses p̂ (0.0425) but the rest of the machinery is right.
• Conclude: Essentially correct if it says: since 0.092 > 0.05, fail to reject H0; there is NOT convincing evidence that the district's true graduation rate differs from 75%. Incorrect if it says the journalist "proved the rate is 75%" or "accepts H0" — those phrases are automatic conclusion deductions.

Deeper dive: the p-value in the prosecutor's fallacy and beyond

Lesson 6 warned against swapping P(data | H0) with P(H0 | data); significance testing institutionalises the first quantity and *refuses* the second — that refusal is principled, not a gap. We can compute how surprising the data are under a fully specified null model because the model fixes every probability. We cannot compute P(H0 | data) without a prior probability for H0, which the classical framework deliberately declines to invent. The practical consequences for your exam answers: never translate a p-value into a probability that a claim is true or false; translate it into the strength of evidence *against* H0, on the accepted scale — p < 0.01 "very convincing", around 0.05 "convincing" at the usual threshold, 0.08 "suggestive but not convincing at the 5% level", 0.40 "no real evidence against H0". This graded language, rather than a binary significant/not, is what strong FRQ conclusions sound like, and what scientists increasingly demand after the replication crisis exposed how a hard 0.05 line invited p-hacking — repeatedly testing until noise crosses the threshold, a sin of process no single p-value can confess.

Case file: why the direction of Ha is fixed in advance

The journalist's graduation test (the lesson's FRQ) produced z ≈ −1.69 and a two-sided p-value ≈ 0.092: not significant at 5%. Watch the temptation: the *one-sided* p-value for the same data is ≈ 0.046 — significant! A researcher who peeks at the data, notices the sample fell below the claim, and then declares a one-sided test has quietly doubled the false-alarm rate: extreme results in *either* direction would have triggered the same switch, so the honest Type I error rate is the two-sided 5%, while the reported one pretends to be 5% having spent 10%. This is why the State step fixes Ha from the *research question* ("differs from" → two-sided; "evidence the rate has dropped" → one-sided) before any numbers appear, and why readers treat a suspiciously convenient one-sided test as an integrity issue, not a style choice. The same discipline governs α and stopping rules in real trials: pre-registration exists because the garden of forking paths is paved with p = 0.046.