Lesson 19: Data Visualization and an Introduction to Data Science

"Data science" sounds grand, but its daily reality is the skills of this course aimed at collections of measurements: summarise them honestly (mean, median, spread), picture them clearly (charts), and — in the simplest machine-learning model there is — classify new points by their nearest neighbours. This lesson builds each piece from scratch on one small dataset, then shows the library versions you will use from here on.

Summaries that tell the truth

Given ten quiz scores, three numbers describe the centre — and they are not interchangeable. Building them by hand shows exactly what each one counts:

scores = [78, 85, 92, 85, 70, 88, 85, 95, 72, 80]

def mean(numbers):
    return sum(numbers) / len(numbers)

def median(numbers):
    ordered = sorted(numbers)
    n = len(ordered)
    mid = n // 2
    if n % 2 == 1:
        return ordered[mid]
    return (ordered[mid - 1] + ordered[mid]) / 2

def mode(numbers):
    counts = {}
    for n in numbers:
        counts[n] = counts.get(n, 0) + 1
    return max(counts, key=counts.get)

def pstdev(numbers):
    avg = mean(numbers)
    return (sum((x - avg) ** 2 for x in numbers) / len(numbers)) ** 0.5

print("Mean:", mean(scores))
print("Median:", median(scores))
print("Mode:", mode(scores))
print("Std dev:", round(pstdev(scores), 2))

Output:

Mean: 83.0
Median: 85.0
Mode: 85
Std dev: 7.66

The mean (830 ÷ 10 = 83.0) balances the total; the median (the middle of the sorted list — with ten values, the average of the 5th and 6th, both 85) marks the halfway student; the mode (85, appearing three times, tallied with Lesson 5's counting pattern) names the most typical score. The standard deviation (~7.66) measures *spread*: on average, scores sit about 7.7 points from the mean. Computing it by hand once — squared distances, averaged, square-rooted — demystifies it permanently: it is a typical distance from the centre, in the data's own units. The statistics module provides all four (mean, median, mode, pstdev) and you verified its answers match in Lesson 12's Try it; from here on, use the library and spend your attention on *which* summary a question needs.

Worked example 1 — why the median exists

Add one extreme value and watch the summaries disagree:

incomes = [30, 32, 35, 38, 40, 42, 1000]  # thousands per year; one outlier
print("Mean:", round(mean(incomes), 1))
print("Median:", median(incomes))

Output:

Mean: 173.9
Median: 38

The mean claims the "average" income is about £174k — in a street where six of seven households earn under £42k. The single 1000 dragged it (the mean feels every value; that sensitivity is sometimes the point, as with total spending). The median, 38, is immune to the outlier's size. Honest reporting chooses deliberately: incomes, house prices and anything with rare extremes are usually summarised by medians, which is why official statistics quote median earnings. Choosing a summary to mislead is the oldest trick in data — you now know how to spot it.

Worked example 2 — a text bar chart

No graphics window? Charts still work: scale values to a row of characters (the same scaling maths as Lesson 18's turtle chart — one idea, two renderers):

def bar_chart(data, width=40):
    biggest = max(data.values())
    for label, value in data.items():
        bars = "#" * round(value / biggest * width)
        print(f"{label}: {bars} {value}")

grades = {"A": 13, "B": 30, "C": 40, "D": 10, "F": 3}
bar_chart(grades)

Output:

A: ############# 13
B: ############################## 30
C: ######################################## 40
D: ########## 10
F: ### 3

C owns the full 40 characters; every other bar is its honest fraction (A: 13/40 of the width = 13 characters; F: 3). The distribution's shape — most students in the middle bands, few at the extremes — is visible at a glance, which is the entire purpose of a chart: pre-attentive comparison. For publication-quality figures, the matplotlib library takes over; the pattern is identical (data in, geometry computed, rendered out):

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

days = ["Mon", "Tue", "Wed", "Thu", "Fri"]
temps = [21, 24, 19, 23, 26]
plt.plot(days, temps, marker="o")
plt.title("Temperature this week")
plt.ylabel("Degrees C")
plt.savefig("temps.png")
print("Chart saved to temps.png")

Output: Chart saved to temps.png — and the file it names is a real line chart of the week (the two use("Agg") lines direct output to a file instead of a window, so the snippet runs anywhere). Same temperatures as Lesson 13's weather data: spot the cold Wednesday dip in both representations.

Worked example 3 — a first machine-learning model: k-NN

Classification by neighbours: to label a new point, find the k closest training points and take a majority vote of their labels. Distances are Pythagoras; the training points are labelled examples; k=3 smooths noise:

def distance(a, b):
    return ((a[0] - b[0]) ** 2 + (a[1] - b[1]) ** 2) ** 0.5

def predict(point, train, k=3):
    by_distance = sorted(train, key=lambda item: distance(point, item[0]))
    neighbours = by_distance[:k]
    votes = {}
    for _, label in neighbours:
        votes[label] = votes.get(label, 0) + 1
    return max(votes, key=votes.get)

train = [((2, 3), "A"), ((3, 4), "A"), ((8, 9), "B"), ((9, 8), "B")]
print(predict((2, 4), train))
print(predict((9, 9), train))

test = [((1, 2), "A"), ((9, 7), "B")]
correct = sum(1 for point, actual in test if predict(point, train) == actual)
print(f"Accuracy: {correct}/{len(test)}")

Output:

A
B
Accuracy: 2/2

Every component is a course friend: Euclidean distance (Lesson 2's formula), sorted with a key lambda (Lesson 15), dictionary voting (Lesson 5), max by .get. And the professional discipline is present: the model is *evaluated* on separate test points it never trained on, and the 2/2 accuracy is reported as a measurement, not a vibe. Real machine learning scales this thinking — more data, more features, better algorithms (the portal's ML Foundations course goes deeper) — but a model you cannot evaluate honestly is worthless at any scale.

Common mistakes

• Reporting the mean for skewed data. Executive pay, city house prices, website load times — wherever extremes live, lead with the median and say why.
• Charts that lie by construction. Starting a bar chart's axis above zero, or scaling lengths by value but widths too (doubling area effect), distorts comparison. Our scaling keeps zero at zero and proportions exact.
• Testing a model on its training data. Accuracy there measures memorisation, not learning. Always hold out test examples the model has never seen.
• Ignoring scale in distance methods. k-NN compares features numerically: if one feature ranges to 1000 and another to 1, the first dominates every distance. Real projects normalise features first — a limitation worth naming before it bites.

Real-World Challenge

The task. A science class measured plant heights (cm) after two weeks for seedlings grown in sunlight and in shade. Build the analysis: group the results, compute each group's mean, print a report, and draw a scaled text chart comparing the groups.

Worked solution.

def group_means(records):
    totals, counts = {}, {}
    for group, value in records:
        totals[group] = totals.get(group, 0) + value
        counts[group] = counts.get(group, 0) + 1
    return {g: totals[g] / counts[g] for g in totals}

def bar_chart(data, width=40):
    biggest = max(data.values())
    for label, value in data.items():
        bars = "#" * round(value / biggest * width)
        print(f"{label}: {bars} {value}")

results = [
    ("Sunlight", 12), ("Sunlight", 14), ("Sunlight", 13),
    ("Shade", 7), ("Shade", 9), ("Shade", 8),
]
means = group_means(results)
for group, avg in means.items():
    print(f"{group}: {avg:.1f} cm")
bar_chart(means, width=20)

Output:

Sunlight: 13.0 cm
Shade: 8.0 cm
Sunlight: #################### 13.0
Shade: ############ 8.0

How it works. The records arrive as (group, value) pairs — the shape real experimental logs take. group_means runs two accumulators in parallel (totals and counts per group, both by the counting pattern) and finishes with a dictionary comprehension dividing each total by its count: Sunlight (12 + 14 + 13) ÷ 3 = 13.0 cm, Shade (7 + 9 + 8) ÷ 3 = 8.0 cm. The chart reuses bar_chart at a smaller width, and the proportions survive the resize — 8/13 of 20 rounds to 12 characters. A result, a number and a picture, all mutually consistent: that consistency is what makes a data report trustworthy, and it falls out of computing everything from the same source values rather than typing numbers into prose by hand.

Extension. Three plants per group is a tiny sample. Discuss (in comments) what could mislead: one sickly seedling, uneven watering. Then extend the code with per-group standard deviations using pstdev from this lesson, and print them beside the means — "13.0 ± 1.0" is a claim with its uncertainty attached, the grown-up form of every measurement in this course.

Going further — percentages, samples and honest denominators

Counts persuade nobody until they share a denominator. Survey results, election shares and test pass rates are all reported as percentages — count divided by total, times one hundred — and the formatting discipline from this course applies unchanged:

survey = {"Yes": 47, "No": 38, "Unsure": 15}
total_votes = sum(survey.values())
for answer, count in survey.items():
    print(answer, f"{count / total_votes * 100:.1f}%")

Output:

Yes 47.0%
No 38.0%
Unsure 15.0%

Behind the trivial arithmetic sit the two questions that decide whether a percentage means anything. First: *what is the total?* — 47 of 100 respondents is a very different claim from 47 of 52 (which would be 90.4%); always state or check the denominator, because percentage-without-base is the second-oldest trick in data presentation. Second: *who was asked?* A survey of a class, a school and a country support different conclusions even at identical percentages; data scientists call the asked group the sample and the group you conclude about the population, and the gap between them is where misleading statistics live. The technical skills of this lesson — grouping, averaging, charting — are the easy half of data science. The other half is this scepticism, applied to your own charts first.

Practice corner — quartiles: the median's family

One median splits data in half; quartiles split it in quarters, and the library computes them (Lesson 12's verified statistics module again):

import statistics

data = [70, 72, 78, 80, 85, 85, 85, 88, 92, 95]
print(statistics.quantiles(data, n=4))

Output: [76.5, 85.0, 89.0] — the three cut points: a quarter of scores fall below 76.5, half below 85.0 (the median, appearing as expected), three quarters below 89.0. The gap between the outer quartiles (89.0 − 76.5 = 12.5, the interquartile range) is a spread measure with the median's outlier resistance, which is why exam boards and salary reports quote it. Together — median, quartiles, standard deviation — you now own the standard summary kit: centre, spread, and shape, each checkable by hand and each one import away.