Lesson 16: Statistics

Statistics is the science of learning from data, and in the age of big data it is civic self-defence: the citizen who cannot interrogate an average will be ruled by those who can. China has counted itself for two thousand years — the Han census recorded some 60 million people, and the 2020 census counted 1.41 billion — but counting everyone was never the hard part; summarising honestly is. This lesson covers the summaries (mean, median, mode; variance and standard deviation), the pictures (histograms, scatter plots), sampling and the normal curve, and ends with Simpson's paradox and the five questions that disarm most statistical manipulation.

Summarising data

The mean (sum ÷ count) is the balance point: the signed deviations from it sum to zero. The median is the middle value of the ordered data; the mode is the most frequent. For symmetric data they coincide; for skewed data they diverge, and the divergence itself is information. The weighted mean Σwᵢxᵢ/Σwᵢ handles grades (40% midterm, 60% final) and GPAs (credits as weights).

Spread: the range is crude; the interquartile range (IQR = Q3 − Q1) covers the middle half; the professional measure is the variance σ² = (1/N)Σ(xᵢ − μ)² and its square root, the standard deviation σ, which returns to the original units. Two classes can share a mean of 80 with σ = 4 (uniform) or σ = 15 (polarised) — same average, different classrooms, different teaching decisions.

Worked examples

Worked example 1 — mean versus median. Five salaries (¥): 5,000; 6,000; 7,000; 8,000; 100,000. Mean: 126,000/5 = ¥25,200. Median: the middle value ¥7,000. The mean claims a "typical" salary nobody earns — four of five workers earn less than a quarter of it. The single outlier (the boss) dragged the mean; the median resisted. For income-like (right-skewed) data, the median is the honest summary; companies quoting "average salary" choose whichever flatters.

Worked example 2 — variance by hand. Data: 2, 4, 4, 4, 5, 5, 7, 9 (N = 8). Mean: 40/8 = 5. Deviations: −3, −1, −1, −1, 0, 0, 2, 4. Squares: 9, 1, 1, 1, 0, 0, 4, 16 — sum 32. Variance σ² = 32/8 = 4; standard deviation σ = 2. Computing shortcut (verify it): Σx²/N − μ² = (4 + 16 + 16 + 16 + 25 + 25 + 49 + 81)/8 − 25 = 232/8 − 25 = 29 − 25 = 4 ✓. The shortcut is faster; the deviation table is clearer. In examinations, clarity wins.

Worked example 3 — weighted mean. A course weights the midterm 40% (score 85) and the final 60% (score 92): final grade = 0.4 × 85 + 0.6 × 92 = 34 + 55.2 = 89.2. Note the weights sum to 1; if given as raw weights (credits 3 and 5), divide by their sum: Σwx/Σw.

Worked example 4 — z-scores and poll margins. (a) A score of 130 on a test with μ = 100, σ = 15 has z = (130 − 100)/15 = 2: two standard deviations up, roughly the top 2.5% for bell-shaped data (the empirical rule: 68% within 1σ, 95% within 2σ, 99.7% within 3σ). Gaokao standardised scores （标准分） are built on exactly this idea, making different years comparable. (b) A poll of n = 1000 has margin of error ≈ 1/√n ≈ 3.2%. Quadrupling n only halves the margin — precision is expensive, and a poll of 100 (±10%) proves almost nothing.

Worked example 5 — spotting the lie. A chart shows a school's average score "soaring" from 80 to 82 — but the y-axis runs from 78 to 84, so the bar heights visually triple. Truncated axes are the commonest graphical lie. Defences: always read the axis before the bars; mentally re-extend to zero; ask for the numbers behind the picture. A histogram's shape is real information (symmetric, skewed, bimodal — the last often means two populations were mixed); a bar chart with a doctored axis is theatre.

Challenge problem — Gaokao standard

Problem (Simpson's paradox). Two treatments are compared. Treatment A: among 100 easy cases, 90 recover (90%); among 100 hard cases, 30 recover (30%). Treatment B: among 1000 easy cases, 870 recover (87%); among 100 hard cases, 20 recover (20%). Which treatment is better?

Solution. Subgroup by subgroup, A wins: 90% > 87% on easy cases and 30% > 20% on hard cases. But overall: A recovers (90 + 30)/200 = 60%; B recovers (870 + 20)/1100 ≈ 80.9%. B's overall rate is far higher despite losing every subgroup. The resolution: the mixtures differ. B's patients were mostly easy cases (1000 of 1100), A's were half hard. The aggregate compares different populations, not different treatments; the subgroup comparison — like with like — is the valid one, and it favours A. The real-world original (Berkeley admissions, 1973) showed apparent bias overall that vanished department by department, because applicants of one group applied disproportionately to harder departments. Aggregates lie when compositions differ; stratify first, conclude second.

Common mistakes

• Quoting the mean for skewed data. Income, house prices, city sizes: right-skewed, mean inflated by the tail (Worked example 1). Ask "mean or median?" reflexively.
• Correlation read as causation. Ice-cream sales correlate with drowning — both are driven by summer. A correlation coefficient measures co-movement, never mechanism; the mechanism needs an experiment or a causal argument.
• Relative risk without the base. "Cuts risk by 50%" from 2% to 1% (absolute change: 1 percentage point) versus from 40% to 20% (absolute: 20 points). The relative figure is identical; the meaning is not (Lesson 15's base-rate lesson in a new costume).
• Tiny samples, grand claims. Margin ≈ 1/√n (Worked example 4b). A "survey" of 30 customers carries ±18% — noise wearing a lab coat.
• Confusing σ and σ². Variance is in squared units (¥² is meaningless); standard deviation is in the data's units. Report σ for interpretation, σ² for computation.

Exam Experience

Multiple choice. The median of 3, 7, 5, 9, 11 is: A. 5 B. 7 C. 9 D. 35 Answer: B. Order first: 3, 5, 7, 9, 11; the middle value is 7. (Option D is the sum — the trap for the hurried.)

Fill in the blank. The (population) variance of the data 4, 6, 8 is ____. Answer: 8/3. Mean 6; squared deviations 4, 0, 4; variance (4 + 0 + 4)/3 = 8/3.

Solution question (10 marks). Two classes take the same test. Class A (3 sampled scripts) scores 76, 80, 84. Class B scores 60, 80, 100. (1) Compute each class's mean and variance. (2) A report says "the classes performed identically". Critique this claim. Marking scheme. (1) Both means: (76 + 80 + 84)/3 = 80 and (60 + 80 + 100)/3 = 80 — 2 marks. Class A variance: deviations −4, 0, 4 → (16 + 0 + 16)/3 = 32/3 ≈ 10.7 — 2 marks. Class B variance: deviations −20, 0, 20 → (400 + 0 + 400)/3 = 800/3 ≈ 266.7 — 2 marks. [6 marks] (2) The means agree but the spreads differ by a factor of 25 (σ ≈ 3.3 versus ≈ 16.3): Class A is uniformly near 80, Class B is polarised between weak and perfect scripts — 2 marks. Conclusion: identical means conceal opposite teaching situations; a mean without a spread is half a fact — 2 marks. [4 marks] Total: 10 marks.

Going deeper — regression to the mean and a first regression line

Regression to the mean. Galton found that tall fathers have sons who are tall but, on average, less tall than themselves — not because of any force pulling towards mediocrity, but because an extreme measurement mixes signal with luck, and luck does not inherit. The mechanism: any observation = true level + noise. Selecting the extreme observations selects extreme noise too; on re-measurement the noise re-randomises and values drift towards the mean. Consequences are everywhere: the "Sports Illustrated jinx" (cover athletes, selected for a career-best performance, decline), students whose worst test triggers tutoring and then "improvement" (how much is the tutoring?), a harsh manager whose scolding "works" after a bad week. Whenever an intervention follows an extreme event, regression is the null hypothesis to beat. Lesson 15's base rates and this phenomenon are the same lesson in two dialects: context determines what a number means.

A first regression line. For paired data, the least-squares line passes through (x̄, ȳ) with slope b = Σ(xᵢ − x̄)(yᵢ − ȳ)/Σ(xᵢ − x̄)². Worked: points (1, 2), (2, 3), (3, 5) (study hours versus quiz score, say). x̄ = 2, ȳ = 10/3. Numerator: (−1)(2 − 10/3) + 0 × (…) + (1)(5 − 10/3) = (−1)(−4/3) + (1)(5/3) = 4/3 + 5/3 = 3. Denominator: 1 + 0 + 1 = 2. Slope b = 3/2. Line: y − 10/3 = (3/2)(x − 2), i.e. y = 1.5x + 1/3. Predictions: at x = 4, y ≈ 6.3 — an extrapolation beyond the data (risky; regression lines are local summaries, and the Gaokao will ask you to say so). The five skeptical questions guard the exit: (1) compared to what? (2) says who, and how measured? (3) where is the denominator? (4) mean or median — and what is the spread? (5) could regression, composition (Simpson), or a confounder explain it instead? Carry them everywhere.

Second look — the five-number summary and outlier fences

Data (test scores): 3, 5, 6, 8, 9, 11, 14. The five-number summary: minimum 3; first quartile Q1 = 5 (the median of the lower half 3, 5, 6); median 8; third quartile Q3 = 11 (median of 9, 11, 14); maximum 14. IQR = 11 − 5 = 6. Outlier fences sit at Q1 − 1.5 × IQR = 5 − 9 = −4 and Q3 + 1.5 × IQR = 11 + 9 = 20; no data point lies outside, so the set has no outliers by the standard rule — the box plot is a plain box from 5 to 11 with whiskers to 3 and 14. Now change the 14 to 40: Q1, median, Q3 and the IQR do not move at all (they are order statistics, insensitive to how extreme the extreme is), the fences stay at −4 and 20, and 40 is flagged as an outlier — correctly: the summary resists the contamination while reporting it. This pair of properties (resistance plus detection) is why medians, quartiles and box plots are the professional default for messy real data, and why Worked example 1's salary mean failed so badly: the mean offers neither.