Lesson 10: Experiments — Design PrinciplesAP

A survey can show that two variables move together; only an experiment can show that one causes the other. The difference is manipulation: in an experiment, researchers impose a treatment on units and observe the response. Everything in CED Unit 3's experiment topics — control, random assignment, replication, blinding — exists to kill alternative explanations for the results, because killing alternative explanations is what "cause" means.

Observational studies and confounding

In an observational study, nothing is imposed; we watch. Observational studies are indispensable (we cannot randomly assign smoking), but they suffer from confounding: a lurking variable mixed up with the explanatory variable so their effects cannot be separated. Coffee drinkers might develop more heart disease — or they might smoke more, work longer hours, and sleep less. No amount of sample size separates tangled variables after the fact; only design does. Retrospective (past data) and prospective (follow forward) describe timing, not quality — both can be confounded.

The principles of experimental design

• Comparison: use at least two treatment groups (or a treatment and a control), so the treatment effect is measured against something.
• Random assignment: allocate experimental units to treatments by chance. This is the magic ingredient: it balances the groups on *all* variables — known, unknown, unmeasurable — in the long run, so a later difference in response can be attributed to the treatment.
• Control: keep everything else as constant as possible — same environment, same procedures — so only the treatment differs systematically.
• Replication: use enough units in each group that chance variation averages out and the effect, if real, can be detected.

Refinements: a placebo (fake treatment) controls for the placebo effect — improvement from believing you are being treated. Blinding hides assignments: single-blind (subjects don't know) or double-blind (neither subjects nor those measuring the response know), preventing both expectation effects and measurement bias. In a matched pairs design, units are paired by similarity and each pair is split between treatments (or each unit receives both treatments in random order) — comparing within pairs removes a huge source of variability. In a randomised block design, units are grouped into blocks similar within (like strata, but for experiments), and treatments are randomly assigned within each block.

Worked example 1 (designing a clean experiment). A farmer wants to test whether a new fertiliser increases tomato yield; she has 24 similar garden plots. Design: randomly assign 12 plots to the new fertiliser and 12 to her standard fertiliser using numbered slips drawn from a hat (or a random number generator); plant the same tomato variety, water identically, and measure yield per plot at season's end; compare the groups' mean yields. Each principle is visible: comparison (standard fertiliser control), random assignment (slips), control (same variety, watering, location management), replication (12 plots per group, not 1 — one plot per group would make treatment and plot indistinguishable, a fatal design the exam presents often).

Worked example 2 (matched pairs). Thirty runners each test two insole brands for foot pain over two weeks, wearing Brand A in one week and Brand B in the other, with the order randomised for each runner. Every runner is their own control, so differences between runners (weight, gait, training volume) cancel within pairs. The analysis compares the 30 within-runner differences. Blocking/pairing works because variability *between* runners is typically much larger than the treatment effect we are hunting.

Scope of inference — the 2×2 the exam loves

Two design choices decide what you may conclude:

• Random sample from a population? If yes, results generalise to that population; if the subjects were volunteers/convenience, they do not.
• Random assignment to treatments? If yes, cause-and-effect conclusions are permitted; if not, only association.

A study with randomly sampled subjects *and* random assignment earns both. The classic trap: volunteers randomly assigned to treatments — causally valid, but it generalises only to people like the volunteers.

Real-world application

The COVID-19 vaccine trials are the gold-standard case study: tens of thousands of volunteers, randomly assigned, double-blind, placebo-controlled — which is why their efficacy claims could be causal. Tech companies run the same design as "A/B tests": users randomly assigned to page versions, so conversion differences are attributable to the design, not to which users happened to show up.

Common mistakes

• Confusing random assignment with random selection. Assignment creates comparable treatment groups inside the study; selection creates a representative sample from a population. They solve different problems, and FRQ conclusions must name the right one.
• One unit per treatment. If one classroom gets the new curriculum and another does not, "classroom" and "curriculum" are perfectly confounded — the experiment has no replication.
• Claiming causation from an observational study because the association is strong. Strength does not substitute for design.
• Forgetting blinding where expectations matter (pain, effort, taste): unblinded subjective responses invite the placebo effect into the data.

AP Exam Experience

Question 1. Researchers recruit 200 volunteers from a gym and randomly assign them to a new training programme or the standard programme. After 12 weeks the new programme shows greater strength gains. Which conclusion is best supported?

• (A) The new programme causes greater gains for all gym members everywhere.
• (B) The new programme causes greater gains, but the result generalises only to people similar to the volunteers.
• (C) The study shows association only, because volunteers were used.
• (D) No conclusion is possible because the subjects were not randomly selected.
• (E) The new programme is proven better for the general population.

Answer: (B). Random assignment licenses cause-and-effect; the convenience sample of volunteers limits generalisation to similar people. Choice (A)/(E) overstate the population reach; (C) understates what random assignment buys.

Question 2. In a taste test, each of 40 participants tries two cola formulations and states a preference, with the order of tasting randomised. This design is best described as:

• (A) A completely randomised design with 80 subjects
• (B) A matched pairs design
• (C) A stratified design
• (D) An observational study
• (E) A cluster sample

Answer: (B). Each participant experiences both treatments, forming a pair with themselves; randomising the order controls for order/learning effects. There are 40 experimental units, not 80 — the two tastings per person are paired observations, a distinction the analysis (and the exam) insists on.

Question 3. Why is double-blinding used in a drug trial?

• (A) To make random assignment unnecessary
• (B) To prevent both subjects' expectations and evaluators' biases from affecting the measured response
• (C) To increase the sample size
• (D) To ensure the results generalise to the whole population
• (E) To eliminate sampling variability

Answer: (B). Blinding protects the *measurement* process: subjects who know they received the drug may report improvement (placebo effect), and evaluators who know assignments may measure or record differently. It does nothing for (A), (C), (D) or (E).

Free-response question. A horticulturist wants to determine whether a new plant food increases the growth of tomato plants. She has 30 similar plants and two growing areas: a sunny bench and a shaded bench, with 15 plants on each. She suspects light level affects growth. (a) Describe a randomised block design for this experiment, identifying the blocks, treatments, response variable, and exactly how randomisation is used. (b) Explain why blocking by bench is preferable to a completely randomised design here. (c) After 6 weeks she will measure growth. State one additional feature she should keep constant, and why.

Scoring guideline.

• Part (a): Essentially correct if: blocks = sunny bench and shaded bench (15 plants each); within EACH block, plants are randomly assigned (for example, numbered 1–15 per bench, random digits select 7 or 8) to new food vs standard food; response = growth (height gain or mass) after the period; comparison of mean growth between treatments within/across blocks. Partially correct if randomisation is described only once for all 30 plants (that is a completely randomised design, ignoring the block structure). Incorrect if treatments are assigned by bench — that confounds food with light completely, and readers treat it as the central error.
• Part (b): Essentially correct if it states that light affects growth and varies between benches, so blocking removes bench-to-bench variability from the treatment comparison, making a real food effect easier to detect (like pairing). Incorrect if it says blocking "makes the sample representative" — blocks are about variability within the experiment, not generalisation.
• Part (c): Essentially correct for any sensible constant — same watering schedule, same pot size/soil, same plant variety/age, same measurement method — with the reason that otherwise that factor could confound or add variability. Partially correct if the feature is named with no reason.

Deeper dive: why random assignment works — the long-run balance argument

Sceptics ask how shuffling can possibly control variables nobody has measured. The answer is probabilistic, not magical: random assignment makes the treatment groups *exchangeable*. Any lurking variable — age, motivation, a gene nobody knows about — is distributed between the groups purely by chance, so with enough units the groups end up similar on *every* variable simultaneously, and any systematic difference in the response is left with only two explanations: the treatment, or an unlucky shuffle. The second explanation is quantified by the p-value machinery of Units 6–7 (that is literally what a p-value measures here). Small experiments can still be unlucky — with 6 subjects, a coin-flip assignment can put all three strongest in one group — which is why replication and blocking exist: blocking forces balance on the variable you most fear, and randomisation handles the rest. Notice the division of labour, because FRQ parts ask for it: *blocking is insurance on known risks; randomisation is insurance on unknown ones.*

Case file: the ginkgo memory experiment, fully specified

A researcher tests whether ginkgo supplements improve memory. Forty adult volunteers are recruited. Design, annotated: each volunteer takes a baseline memory test and is paired with another volunteer of similar baseline score and age, forming 20 pairs (blocking on the strongest known predictor of the response — baseline memory). Within each pair, a coin flip assigns one volunteer to ginkgo and the other to an identical-looking placebo (random assignment *within blocks*); neither volunteers nor the testers who score the final test know the assignments (double-blind). After eight weeks, everyone retakes the test, and the analysis compares score gains within pairs. Every Unit 3 defence is deployed: comparison (placebo group absorbs the "practice makes you better at tests" effect and the placebo effect), control (identical pills, identical schedule), replication (20 pairs, not 1), blinding (against expectation effects on a subjective-ish outcome), and blocking (removing baseline variability from the comparison). Scope check, stated before any results: causation, if found, extends only to adults like these volunteers — the sample was not random from any population. Designing this completely, in this order, is exactly the skill the experiment-design FRQ scores.