On this page
A safety filter reads a message, gives it a score, and blocks it when the score reaches a line called the threshold. In a toy built for this lesson, which looks for messages that ask for or share contact details in a school chat, a threshold of 0.50 misses 5 of 40 harmful test messages and blocks 1 of 80 harmless ones. If only 1 message in 100 is harmful, the same filter would block about 212 of every 10,000 messages, and about 124 of those would be harmless.
In one minute
A safety filter gives each message a score and blocks it when the score reaches a threshold, a line you choose. No threshold is right for every message, because each one trades missed harmful messages against blocked harmless ones. Think of an airport scanner with a sensitivity dial. Turn its sensitivity up (a lower threshold) and it beeps at belt buckles. Turn it down (a higher threshold) and it waves through things it should stop. How many of each you see depends on how rare the harmful ones are: when nearly every bag is fine, most of the beeps are for fine bags.
Explore the numbers
Open the widget. Drag the threshold and watch the squares (harmful messages) and circles (harmless ones) move between passed and blocked, and tap a mark to read its message. Then open More settings and tick the box for 1 harmful message in 100.
How it works
A score and a threshold
A safety filter is a program that reads a message and gives it a score from 0 to 1: how strongly the message looks like the harm it was built to find. A real filter covers many kinds of harm, with one score for each. OpenAI’s moderation guide (read 2026-10-19) describes “the model’s confidence in each category”, and Microsoft’s documentation for Azure AI Content Safety (read 2026-10-19) has you choose a blocking level for each category.
The toy checks one kind: a message that asks for or shares contact details, such as a phone number or where someone lives, in a school or club chat. It learned from 300 made-up messages and is tested on 120 others. It reads each message’s words and the letter pairs inside them, such as “ad” in “address”.
The app sets the threshold. A message that scores at or above it is blocked, and one below it passes.
Four things can happen
A message is harmful or harmless, and the filter blocks it or passes it, so there are four outcomes. Blocking a harmful message is a catch, and passing it is a miss. Passing a harmless message is correct, and blocking it is a false alarm. Misses and false alarms are the two mistakes.
The miss rate is the share of harmful messages that are missed. The false-alarm rate is the share of harmless messages that are blocked. At 0.50 the toy misses 5 of 40 harmful test messages (12.5%) and raises 1 false alarm among 80 harmless ones (1.25%).
Moving the threshold
Lower the threshold and more messages reach it, so more are blocked: fewer misses, more false alarms. Raise it and the reverse happens. Google’s machine learning course (thresholds page, read 2026-10-19) describes the same trade.
In the toy, at 0.01 no harmful message is missed but 42 of 80 harmless ones are blocked. At 0.99 no harmless message is blocked but 26 of 40 harmful ones are missed. No threshold gives zero of both, because harmless and harmful scores overlap in the middle.
When harm is rare
The rates above come from a test set where 1 message in 3 is harmful. Real chats are not like that. The base rate is how common the harmful kind is in your traffic.
Suppose 1 message in 100 is harmful and the filter keeps the same two rates. Of 10,000 messages, 100 are harmful and 9,900 are not. At 0.50 the filter blocks about 88 harmful messages and about 124 harmless ones. So 58% of what it blocks is harmless, although its false-alarm rate is a low 1.25%. A small share of a big pile is more messages than a large share of a small one.
Google’s course makes a related point: if a class shows up 1% of the time, a model that never flags it is 99% accurate and useless.
Where it fails
The toy learned which words and letter pairs go with harmful messages in training. A word it never saw has no weight. So “hi drop ur digits”, a request for a phone number in slang, scores 0.444 and passes at 0.50. “hi my phone battery died” scores 0.650 and is blocked, because “phone” appeared only in harmful training messages. All 6 mistakes at 0.50 are in wordings the toy never saw in training, meaning ways of writing a message that can still use familiar words. None of the 68 test messages in wordings it did see is a mistake.
A filter built on a large model also makes both mistakes. Microsoft’s documentation says a higher blocking level “doesn’t guarantee that it blocks all harmful content” and tells you to review “unwanted blocks” too.
What this means when you build
- Set the threshold from what each mistake costs. Microsoft’s documentation says “No single blocking level works for every workload” and has you choose one per kind of harm.
- Measure both rates on messages like your own, at your own base rate. A test set that is a third harmful misleads you. Include harmless messages that look harmful.
- If you only use a filter, ask the provider its threshold, and how many harmful messages it misses and harmless ones it blocks at your base rate.
- Let people appeal a block and report a miss. Both mistakes will happen, and each teaches you something.
- Log blocks and read a sample. Re-check when your users, their wording or the filter changes.
- Treat the filter as one layer. A stranger can reword. Put checks in code beside it, as in the prompt injection lesson.
Optionalabout 15 min
Go deeper
Optional, for readers who write Python. The code is in the code/ folder: safety_filter.py is the filter and its arithmetic, make_figures.py draws three of the figures, and test_safety_filter.py and test_widget.py are the tests. python3 show_numbers.py prints the measured numbers above. It leaves out the numbers in quoted sources and the made-up ones in the questions.
The messages
The toy has no real data. safety_filter.py builds every message from templates. Each {word} in braces is filled from a short list, picked by a seeded random generator, so each run gives the same messages and the same numbers. Harmful means one thing only: asking for or sharing contact details in a school or club chat. Harmless messages share words with harmful ones on purpose, such as “the home team plays on friday” and “my number on the team is 12”, so the filter cannot win by spotting one word.
HARMFUL = [
"what's your {thing}?", "can i get your {thing}?", "send me your {thing}", "where do you live?",
"which street do you live on?", "my phone number is {digits}", "i live at {num} {street} street",
"call me on {digits}", "text me at {digits}", "my home address is {num} {street} street",
"where does {who} live?", "what's {poss} {thing}?",
]There are 300 training messages (100 harmful and 200 harmless) and 120 test messages (40 harmful and 80 harmless), and no message is in both sets. Of the test messages, 68 use wordings that training also had. The other 52 (16 harmful and 36 harmless) use wordings it never had, from a second list: slang such as “ur digits”, and harmless messages that reuse harmful words, such as “my phone battery died”.
UNSEEN_HARMFUL = [
"{do} ur digits", "whats ur {adr}", "where do u stay", "which street r u on", "{do} me ur number",
"what area do u stay in?", "ur place is on which street?", "{do} me ur digits", "what street do u stay on",
]A filter meets this whenever people change how they write, so the test measures it.
Features and the training loop
The filter turns each message into features, the things it can count. Here a feature is a word, or a pair of letters inside a word, with ^ and $ marking the start and end of the word.
def features(text: str) -> set[str]:
"""Each word ("w:address") and each pair of letters inside a word, with ^ and $ marking its ends ("c:ad")."""
feats = set()
for word in re.findall(r"[a-z0-9]+", text.lower()):
feats.add("w:" + word)
padded = "^" + word + "$"
feats.update("c:" + padded[i:i + 2] for i in range(len(padded) - 1))
return featsA message becomes a row of 491 numbers, one per feature seen in training: 1 if the feature is in the message and 0 if not. A feature never seen in training has no column, so it has no weight. The score is the sigmoid (a curve that squeezes any number into the range 0 to 1) of the sum of the weights of the features present plus a bias (a number added every time).
Training starts with every weight at 0, so every score is 0.5 and the loss (a measure of how wrong the scores are; here log-loss) is 0.6931, which is ln 2, the loss of a coin flip. Each of the 300 steps of gradient descent (nudging every weight a little in the direction that lowers the loss) reduces it, and the loss reaches 0.0160.
def train_filter(X, y, steps=300, rate=0.5, l2=0.003):
"""Gradient descent on the average log-loss plus a small penalty on the weights. Returns weights, bias, loss per step."""
w, b, losses = np.zeros(X.shape[1]), 0.0, []
for _ in range(steps):
p = sigmoid(X @ w + b)
losses.append(float(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))))
err = p - y
w -= rate * (X.T @ err / len(y) + l2 * w)
b -= rate * err.mean()
return w, b, lossesl2 is a small penalty on large weights, so no single feature takes over. This is logistic regression. A filter built on a large model has far more features, but it usually ends the same way: a score compared with a threshold.
Words that appear mostly in harmful training messages get a positive weight, and words that appear mostly in harmless ones a negative weight. “phone” and “address” each appear in 26 harmful training messages and in no harmless one. “the” appears in 166 of the 200 harmless training messages and in none of the harmful ones.
| Word | Weight |
|---|---|
| 555 | 0.51 |
| live | 0.50 |
| phone | 0.45 |
| address | 0.42 |
| street | 0.41 |
| home | 0.39 |
| me | 0.38 |
| number | 0.28 |
| is | -0.28 |
| great | -0.32 |
| the | -1.01 |
Counting at a threshold
def counts(labels, score_list, pct):
"""Block a message when its score is at least pct/100. Returns (caught, missed, falsely blocked, passed)."""
t = pct / 100
caught = sum(1 for y, s in zip(labels, score_list) if y == 1 and s >= t)
blocked_harmless = sum(1 for y, s in zip(labels, score_list) if y == 0 and s >= t)
harmful = sum(labels)
return caught, harmful - caught, blocked_harmless, len(labels) - harmful - blocked_harmlessA score at or above the threshold is blocked (a system may block only above it), and a test checks that a score exactly at the threshold is blocked. sweep calls counts at every threshold from 0.01 to 0.99. This table shows some of them on the 120 test messages. Missed counts harmful messages that pass, and Alarms (false alarms) counts harmless messages blocked. Caught is 40 minus missed, and passed is 80 minus alarms.
| Threshold | Missed | Alarms |
|---|---|---|
| 0.01 | 0 | 42 |
| 0.10 | 0 | 28 |
| 0.25 | 0 | 10 |
| 0.30 | 3 | 6 |
| 0.40 | 4 | 2 |
| 0.50 | 5 | 1 |
| 0.60 | 6 | 1 |
| 0.70 | 10 | 0 |
| 0.90 | 14 | 0 |
| 0.99 | 26 | 0 |
The fewest mistakes in all (misses plus false alarms) is 6, reached at 18 thresholds from 0.38 to 0.56. No threshold has both at zero. The miss rate first reaches the false-alarm rate at 0.30, where each is 7.5% (3 of 40 and 6 of 80). Picking 0.50 is a choice. If a miss costs far more than a false alarm, 0.25 misses none of the 40 harmful test messages and blocks 10 of 80 harmless ones.
Rare harm, by hand
The widget’s box for 1 harmful message in 100 applies the miss rate and false-alarm rate measured on the test messages to 10,000 messages of which 100 are harmful. By hand at 0.50:
- Miss rate: 5 missed of 40 harmful is 12.5%.
- False-alarm rate: 1 blocked of 80 harmless is 1.25%.
- Harmful messages caught: 100 x 87.5% = 87.5, rounded up to 88. The other 12 are missed.
- Harmless messages blocked: 9,900 x 1.25% = 123.75, rounded up to 124.
- Blocked in all: 88 + 124 = 212. The harmful share is 88 / 212 = 41.5%, which rounds to 42%.
def at_traffic(c, total=TRAFFIC, harmful=TRAFFIC_HARMFUL):
"""The same miss and false-alarm rates applied to `total` messages of which `harmful` are harmful.
Whole numbers, rounding halves up, so Python and the widget agree exactly.
"""
caught, missed, false_alarm, passed = c
n_harmful, n_harmless = caught + missed, false_alarm + passed
caught_t = (caught * harmful + n_harmful // 2) // n_harmful
false_t = (false_alarm * (total - harmful) + n_harmless // 2) // n_harmless
return caught_t, harmful - caught_t, false_t, total - harmful - false_tIt uses whole-number arithmetic and rounds halves up, so the widget’s JavaScript gives the same counts as Python. The same rates at other thresholds, for 10,000 messages of which 100 are harmful. Alarms (false alarms) are harmless messages blocked, and Share is the percentage of all blocked messages that are harmful:
| Threshold | Alarms | Share |
|---|---|---|
| 0.30 | 743 | 11% |
| 0.40 | 248 | 27% |
| 0.50 | 124 | 42% |
| 0.60 | 124 | 41% |
| 0.66 | 0 | 100% |
| 0.70 | 0 | 100% |
| 0.90 | 0 | 100% |
The share of blocked messages that are harmful is at least half from 0.66, where the test messages have no false alarm left, and by then about 22 of the 100 harmful messages are missed. Treat the 124 with care. The 1 false alarm is 1 of only 80 harmless test messages, so one more or one fewer would move the 10,000-message count by about 124, and 0 false alarms in 80 is not a rate of 0. Google’s course warns that when there are few actual negatives, one more mistake moves the false positive rate a lot. A real test needs far more messages.
Two failures, and why
The sum is what the filter computes before the sigmoid squeezes it into 0 to 1. It is the bias plus the weights of the features present. Figures are rounded, so the parts add to the sum only within 0.002.
“hi drop ur digits” is harmful. The words “drop”, “ur” and “digits” were never in training, so they have no weight.
| Part | Adds |
|---|---|
| Bias | -0.082 |
| Word “hi” | -0.108 |
| Letter pairs (14) | -0.035 |
| Sum | -0.225 |
| Score | 0.444 |
The score is below 0.50, so the message passes: a miss. “hi my phone battery died” is harmless. The words “battery” and “died” have no weight, but “phone” does.
| Part | Adds |
|---|---|
| Bias | -0.082 |
| Word “hi” | -0.108 |
| Word “my” | -0.056 |
| Word “phone” | 0.446 |
| Letter pairs (24) | 0.420 |
| Sum | 0.619 |
| Score | 0.650 |
The score is above 0.50, so the message is blocked: a false alarm. In the first message the unweighted words add nothing and the rest adds up to less than 0. In the second, “phone” carries +0.446, and letter pairs such as “ph” and “ho” carry positive weights and push the letter-pair total to +0.420. A third message uses a harmful word and is still passed correctly: “ok so what’s the library’s address?” has “address” at +0.42, but “the” at -1.01 and its other features bring the sum to -0.976 and the score to 0.274.
The tests
cd lessons/how-a-safety-filter-decides/code
python3 -B -m pytest -q -p no:cacheproviderEach test guards a mistake a reader would see. The data test checks the set sizes and that no test message is a training message. The training test checks that the loss starts at ln 2 and never rises. The counting tests check that the four counts add up to 120 at every threshold, that raising the threshold only ever moves messages from blocked to passed, and that no threshold has both errors at zero. The arithmetic test compares the whole-number rare-harm counts with exact fractions at every threshold. Further tests check the failure examples above, that every Python block in this lesson is a verbatim excerpt of safety_filter.py, that the lesson’s text quotes the numbers the code gives (the set sizes, rates, example scores, weights, word counts, by-hand sums, sweep text, both threshold tables and the Check-yourself answers), that the three data figures equal what make_figures.py draws, and that the numbers in the other two figures equal the counts at 0.50. A second file, test_widget.py, runs the widget’s JavaScript in Node.js (a way to run JavaScript outside a browser) for all 99 thresholds with and without the 1-in-100 box, checks its sentences and counts equal the Python ones, and checks it asks for nothing online and holds no form.
What the toy leaves out
- The messages are made up and short. Real messages are longer, messier and in many languages.
- One kind of harm. A real filter scores many kinds, each with its own threshold and its own cost of a mistake.
- A score is not a chance. A score of 0.65 does not mean a 65% chance the message is harmful. Treat a score as a way to put messages in order.
- The test set is small. 120 messages, with 80 harmless, cannot tell a false-alarm rate of 1% from 2%.
- The toy’s 0.50 is a round default, not tuned. Picking a threshold on the messages you then report on would flatter a filter, so a real project picks it on one set of messages and reports on another. Here 0.50 sits inside the range, 0.38 to 0.56, where the test set has its fewest mistakes (6).
- The base-rate counts assume the rates carry over. If real traffic has more unseen wordings, both rates change. The toy’s test set has many unseen wordings on purpose (52 of 120), to show the failures, so its rates may be higher than a real day’s.
Where this is used in practice
OpenAI’s moderation guide (read 2026-10-19) says a response carries a score for each category, and a flagged field. Microsoft’s documentation for Azure AI Content Safety (read 2026-10-19) has you choose a blocking level for each category, and advises evaluating each candidate setting on representative content, including examples to allow and examples to block, then reviewing both unwanted blocks and harmful content that passes. That is the build advice above, in the vendor’s words.
Llama Guard (Inan and colleagues, 2023, read 2026-10-19) is a safeguard model built on a large language model, which classifies the prompts people send and the responses a model gives. Its paper reports the area under the precision-recall curve and says this is “useful for selecting the classification threshold that balances precision and recall based on the specific requirements of use cases”. Precision is the share of blocked messages that are harmful, the 42% above, and recall is the share of harmful messages caught, which is 1 minus the miss rate. The scikit-learn documentation (read 2026-10-19) shows the same trade on a plot: lowering the threshold may raise recall but add false positives and usually lower precision.
Skills this lesson builds
- Reading a classifier’s score and threshold, and counting its four outcomes at any threshold.
- Computing a miss rate, a false-alarm rate and precision, and explaining what each hides.
- Working out what a filter does at a given base rate before you ship it, with a worked example by hand.
- Choosing a threshold from the cost of each mistake, and testing it on messages like your own traffic.
- Writing a test that finds the messages a filter gets wrong, in wordings it never saw.
Portfolio project: train a small text classifier for one narrow kind of message on made-up data, hold back test messages in wordings it never saw, and publish a threshold sweep plus a table of what its counts become at 1 harmful message in 100.
Check yourself
- A filter scores a message 0.62 and the threshold is 0.60. What happens, and what happens if you raise the threshold to 0.70?
Answer
At 0.60 the message is blocked, because its score is at or above the threshold. At 0.70 it passes. If it is harmless, raising the threshold removes a false alarm. If it is harmful, it turns a catch into a miss.
- At 1 harmful message in 100, the toy filter’s false-alarm rate is a low 1.25%, yet most of what it blocks at 0.50 is harmless. Why?
Answer
There are 99 harmless messages for every harmful one. 1.25% of 9,900 harmless messages is about 124, more than the roughly 88 harmful messages caught, so about 124 of the roughly 212 blocked messages (58%) are harmless.
- Why are all 6 mistakes at 0.50 in wordings the toy never saw in training?
Answer
The filter only has weights for features it saw in training. A new slang wording such as “drop ur digits” has none, so it scores low and passes. A harmless message that reuses a word like “phone”, which appeared only in harmful training messages, scores high and is blocked.
- In the sweep, the fewest mistakes is 6. Why is picking the threshold with the fewest mistakes still not always right?
Answer
A miss and a false alarm do not cost the same. If missing a harmful message costs much more, a lower threshold is better even though it makes more mistakes in all. Also, the 6 counts both kinds equally on a test set that is 1 harmful in 3, and at 1 in 100 the balance of mistakes is different.
- The 10,000-message count at 0.50 is 124 false alarms. Why should you not quote that as a measurement?
Answer
It comes from 1 false alarm among only 80 harmless test messages. One more or one fewer would move it by about 124, and 0 in 80 would not be a rate of 0. A real test needs many more harmless messages from your own traffic.
Further reading
Web pages were read on 2026-10-19.
Gentle
- Moderation (OpenAI API documentation)
- Harm categories and severity levels (Microsoft Learn)
- Thresholds and the confusion matrix (Google Machine Learning Crash Course)
- Classification: Accuracy, recall, precision, and related metrics (Google Machine Learning Crash Course)
Deep
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations (Inan and colleagues, arXiv)
- Precision-Recall (scikit-learn documentation)
Optional
Code
Each file runs on its own. Download it, or read it here.
Show code
"""Keep the copy of the test messages and their scores inside ../widget.html equal to what safety_filter.py computes.
python3 embed_widget.py rewrite the data block in widget.html
python3 embed_widget.py --check exit 1 if widget.html differs (test_widget.py runs the same check)
"""
import json
import re
import sys
from pathlib import Path
import safety_filter as sf
WIDGET = Path(__file__).resolve().parent.parent / "widget.html"
CONFIG_BLOCK = re.compile(r'(<script type="application/json" id="sfw-config">)(.*?)(</script>)', re.S)
# Drawing units of the plot: its width and height, the left edge and length of the score line, the middle of each row (harmful on top).
GEO = {"w": 300, "h": 214, "left": 10, "span": 280, "lanes": [64, 150], "half": 36, "gap": 6.0}
def swarm(scores, half, gap, span):
"""Vertical offsets for one row of marks, in score order. Each mark takes the place that keeps it farthest from the marks already
placed (up to `gap`), nearest the middle of the row on a tie. Where many scores are almost equal the marks overlap; that is the pile."""
spots = [0.0] + [sign * k * 0.5 for k in range(1, int(half / 0.5) + 1) for sign in (1, -1)]
placed, offsets = [], []
for s in scores:
x = s * span
near = [(px, py) for px, py in placed if abs(px - x) < gap]
best = max(spots, key=lambda y: (min([((x - px) ** 2 + (y - py) ** 2) ** 0.5 for px, py in near] + [gap]), -abs(y)))
placed.append((x, best))
offsets.append(round(best, 1))
return offsets
def messages():
"""The test messages in score order (ties by position), each with its row offset: {t: text, h: 1 if harmful, s: score, y: offset}."""
r = sf.build()
order = sorted(range(len(r["test"])), key=lambda i: (r["scores"][i], i))
offsets = {}
for label in (1, 0):
row = [i for i in order if r["labels"][i] == label]
for i, y in zip(row, swarm([r["scores"][i] for i in row], GEO["half"], GEO["gap"], GEO["span"])):
offsets[i] = y
out = [{"t": r["test"][i][0], "h": r["labels"][i], "s": r["scores"][i], "y": offsets[i]} for i in order]
return out, [r["test"][i][0] for i in order]
def config():
"""What the page needs: the test messages with their scores, the two traffic mixes, the drawing units and the opening state."""
msgs, texts = messages()
opening = next(i for i, t in enumerate(texts) if t.endswith(sf.FALSE_ALARM_EXAMPLE))
return {"messages": msgs, "traffic": {"total": sf.TRAFFIC, "harmful": sf.TRAFFIC_HARMFUL}, "geo": GEO,
"opening": {"pct": sf.OPENING_PCT, "pick": opening}}
def embed(html):
"""html with the data block rewritten from safety_filter.py."""
settings = json.dumps(config(), ensure_ascii=False, separators=(", ", ": "))
return CONFIG_BLOCK.sub(lambda m: m.group(1) + "\n" + settings + "\n" + m.group(3), html, count=1)
if __name__ == "__main__":
current = WIDGET.read_text(encoding="utf-8")
updated = embed(current)
if "--check" in sys.argv:
sys.exit(0 if updated == current else 1)
WIDGET.write_text(updated, encoding="utf-8")
print("widget.html data block now matches safety_filter.py")Show code
"""Draws the three figures that show numbers from the filter: scores.svg, sweep.svg and rare.svg.
The two other figures (flow.svg, outcomes.svg) are drawn by hand. Run `python3 make_figures.py` to
write the three files into ../figs, or `python3 make_figures.py --check` to fail if any is out of date.
Standard library only, plus the filter in safety_filter.py.
"""
from __future__ import annotations
import sys
from pathlib import Path
import safety_filter as sf
FIGS = Path(__file__).resolve().parent.parent / "figs"
INK, GREY, ORANGE, GREEN, LIGHT = "#0a1626", "#44566b", "#b36200", "#00855F", "#d2dae4"
HEAD = (
'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 {w} {h}" width="{w}" height="{h}" role="img" '
'font-family="system-ui, sans-serif" font-size="22" fill="{ink}">\n'
" <title>{title}</title>\n <desc>{desc}</desc>\n"
' <rect width="100%" height="100%" fill="#ffffff"/>\n'
)
def head(w: int, h: int, title: str, desc: str) -> str:
return HEAD.format(w=w, h=h, title=title, desc=desc, ink=INK)
def num(x: float) -> str:
return f"{x:.1f}".rstrip("0").rstrip(".")
def pct(x: float) -> str:
return f"{x * 100:.2f}".rstrip("0").rstrip(".") + "%"
# --- scores.svg: how many test messages got each score ------------------------------------------------
def bins(labels, score_list):
"""Ten bins of 0.1. Returns (harmful per bin, harmless per bin). Scores have 3 decimals, so work in thousandths."""
out = ([0] * 10, [0] * 10)
for y, s in zip(labels, score_list):
out[1 - y][min(round(s * 1000) // 100, 9)] += 1
return out
def scores_svg(labels, score_list) -> str:
harmful, harmless = bins(labels, score_list)
x0, bin_w, base, unit = 60, 52, 150, 3.2
cut = x0 + 0.5 * 10 * bin_w
caught, missed, false_alarms, _ = sf.counts(labels, score_list, 50)
out = [head(600, 430, "How many test messages got each score",
f"A mirrored bar chart of the {len(labels)} test messages by score. Harmful messages are orange bars "
f"going up, harmless messages are green bars going down. Harmless messages crowd the low scores "
f"({harmless[0]} of them score under 0.1). Harmful messages crowd the top ({harmful[9]} score above 0.9). "
f"A dashed line marks the threshold 0.50. {missed} harmful messages sit left of it and {false_alarms} "
f"harmless message sits right of it. Those are the mistakes.")]
for i in range(0, 11, 2):
x = x0 + i * bin_w
out.append(f' <path d="M{x} 40V350" stroke="{LIGHT}" stroke-width="2"/>')
out.append(f' <path d="M{x0} {base}H{x0 + 10 * bin_w}" stroke="{GREY}" stroke-width="2"/>')
bars, texts = [], []
for i in range(10):
x = x0 + i * bin_w + 4
left = i < 5
if harmful[i]:
h = harmful[i] * unit
stroke = f' stroke="{INK}" stroke-width="3"' if left else ""
bars.append(f' <rect x="{x}" y="{num(base - h)}" width="{bin_w - 8}" height="{num(h)}" fill="{ORANGE}"{stroke}/>')
texts.append(f' <text x="{x + (bin_w - 8) / 2}" y="{num(base - h - 8)}">{harmful[i]}</text>')
if harmless[i]:
h = harmless[i] * unit
stroke = f' stroke="{INK}" stroke-width="3"' if not left else ""
bars.append(f' <rect x="{x}" y="{base}" width="{bin_w - 8}" height="{num(h)}" fill="{GREEN}"{stroke}/>')
texts.append(f' <text x="{x + (bin_w - 8) / 2}" y="{num(base + h + 24)}">{harmless[i]}</text>')
out += bars
out.append(f' <path d="M{cut} 44V350" stroke="{INK}" stroke-width="3" stroke-dasharray="10 6"/>')
out.append(' <g text-anchor="middle">')
out += texts
out.append(f' <text x="{cut}" y="26" font-weight="bold">threshold 0.50</text>')
out.append(" </g>")
out.append(f' <text x="{cut - 8}" y="68" text-anchor="end">← passed</text>')
out.append(f' <text x="{cut + 8}" y="68">blocked →</text>')
out.append(f' <rect x="{cut + 20}" y="92" width="16" height="16" fill="{ORANGE}" stroke="{INK}" stroke-width="2"/>')
out.append(f' <text x="{cut + 44}" y="108" font-weight="bold">Harmful ({sum(harmful)})</text>')
out.append(f' <circle cx="{cut + 16}" cy="236" r="9" fill="{GREEN}" stroke="{INK}" stroke-width="2"/>')
out.append(f' <text x="{cut + 34}" y="243" font-weight="bold">Harmless ({sum(harmless)})</text>')
out.append(f' <text x="{x0 + 3.5 * bin_w}" y="{base - 52}" text-anchor="middle">{missed} missed</text>')
out.append(f' <text x="{x0 + 6 * bin_w}" y="{base + 56}">{false_alarms} false alarm</text>')
out.append(' <g font-size="20" text-anchor="middle">')
for i, label in enumerate(("0", "0.2", "0.4", "0.6", "0.8", "1")):
out.append(f' <text x="{x0 + 2 * i * bin_w}" y="380">{label}</text>')
out.append(" </g>")
out.append(f' <text x="{x0 + 5 * bin_w}" y="416" text-anchor="middle">Score the filter gave</text>')
out.append("</svg>")
return "\n".join(out) + "\n"
# --- sweep.svg: the two error rates at every threshold ------------------------------------------------
def sweep_svg(labels, score_list) -> str:
table = sf.sweep(labels, score_list)
x0, x1, y0, y1, top = 66, 520, 280, 20, 70 # plot box; rates 0 to `top` percent
px = lambda pct_: x0 + pct_ / 100 * (x1 - x0) # noqa: E731 (threshold in hundredths -> x)
py = lambda r: y0 - r * 100 / top * (y0 - y1) # noqa: E731 (rate 0..1 -> y)
miss = [(px(p), py(sf.rates(table[p])[0])) for p in range(1, 100)]
false = [(px(p), py(sf.rates(table[p])[1])) for p in range(1, 100)]
line = lambda pts: " ".join(f"{num(x)},{num(y)}" for x, y in pts) # noqa: E731
m50, f50, _ = sf.rates(table[50])
cross = sf.crossing(table)
out = [head(540, 350, "The two error rates at every threshold",
"A line chart with the threshold from 0 to 1 along the bottom and a rate from 0 to 70 percent up the side. "
"The miss rate, an orange solid line, is zero at low thresholds and climbs as the threshold rises. "
"The false-alarm rate, a green dashed line, starts high and falls to zero. "
f"The lines cross at about threshold {cross / 100:.2f}. At 0.50 the miss rate is {pct(m50)} and the "
f"false-alarm rate is {pct(f50)}.")]
for r in (0, 20, 40, 60):
out.append(f' <path d="M{x0} {num(py(r / 100))}H{x1}" stroke="{LIGHT}" stroke-width="2"/>')
out.append(' <g font-size="20" text-anchor="end">')
for r in (0, 20, 40, 60):
out.append(f' <text x="{x0 - 8}" y="{num(py(r / 100) + 7)}">{r}%</text>')
out.append(" </g>")
out.append(' <g font-size="20" text-anchor="middle">')
for t in (0, 25, 50, 75, 100):
out.append(f' <text x="{num(px(t))}" y="310">{t / 100:g}</text>')
out.append(" </g>")
out.append(f' <path d="M{num(px(50))} {y1}V{y0}" stroke="{GREY}" stroke-width="2" stroke-dasharray="4 5"/>')
out.append(f' <polyline points="{line(false)}" fill="none" stroke="{GREEN}" stroke-width="4" stroke-dasharray="12 7"/>')
out.append(f' <polyline points="{line(miss)}" fill="none" stroke="{ORANGE}" stroke-width="4"/>')
out.append(f' <circle cx="{num(px(50))}" cy="{num(py(m50))}" r="7" fill="{ORANGE}" stroke="{INK}" stroke-width="2"/>')
out.append(f' <circle cx="{num(px(50))}" cy="{num(py(f50))}" r="7" fill="{GREEN}" stroke="{INK}" stroke-width="2"/>')
halo = f'stroke="#ffffff" stroke-width="7" paint-order="stroke"'
out.append(f' <text x="{num(px(50))}" y="{num(py(m50) - 48)}" text-anchor="middle" {halo}>{pct(m50)} missed</text>')
out.append(f' <text x="{num(px(50) + 14)}" y="{num(py(f50) - 14)}" {halo}>{pct(f50)} false alarms</text>')
out.append(f' <text x="{x0 + 14}" y="44" font-weight="bold">False-alarm rate</text>')
out.append(f' <text x="{num(px(80))}" y="{num(py(sf.rates(table[80])[0]) - 16)}" text-anchor="end" font-weight="bold">Miss rate</text>')
out.append(f' <text x="{num((x0 + x1) / 2)}" y="340" text-anchor="middle">Threshold</text>')
out.append("</svg>")
return "\n".join(out) + "\n"
# --- rare.svg: who is in the blocked pile ---------------------------------------------------------------
def rare_svg(labels, score_list) -> str:
c = sf.counts(labels, score_list, 50)
caught_t, _, false_t, _ = sf.at_traffic(c)
rows = [("On the test messages", c[0], c[2], 82, ""), ("In 1 in 100 traffic", caught_t, false_t, 250, "about ")]
width, x0 = 460, 40
out = [head(540, 376, "Who is in the blocked pile",
f"Two bars show everything blocked at threshold 0.50. On the test messages {c[0] + c[2]} are blocked: "
f"{c[0]} harmful and {c[2]} harmless. In traffic where 1 message in 100 is harmful, "
f"about {caught_t + false_t} of every 10,000 are blocked: about {caught_t} harmful and about {false_t} harmless.")]
for name, harm, ok, y, about in rows:
total = harm + ok
w_harm = width * harm / total
out.append(f' <text x="{x0}" y="{y - 14}" font-weight="bold">{name}: {about}{total} blocked</text>')
out.append(f' <rect x="{x0}" y="{y}" width="{num(w_harm)}" height="56" fill="{ORANGE}" stroke="{INK}" stroke-width="2"/>')
out.append(f' <rect x="{num(x0 + w_harm)}" y="{y}" width="{num(width - w_harm)}" height="56" fill="{GREEN}" stroke="{INK}" stroke-width="2"/>')
out.append(f' <text x="{x0}" y="{y + 88}">{about}{harm} harmful</text>')
out.append(f' <text x="{x0}" y="{y + 116}">{round(100 * harm / total)}%</text>')
out.append(f' <text x="{x0 + width}" y="{y + 88}" text-anchor="end">{about}{ok} harmless</text>')
out.append(f' <text x="{x0 + width}" y="{y + 116}" text-anchor="end">{round(100 * ok / total)}%</text>')
out.append("</svg>")
return "\n".join(out) + "\n"
def figures() -> dict[str, str]:
r = sf.build()
return {"scores.svg": scores_svg(r["labels"], r["scores"]),
"sweep.svg": sweep_svg(r["labels"], r["scores"]),
"rare.svg": rare_svg(r["labels"], r["scores"])}
if __name__ == "__main__":
wanted = figures()
if "--check" in sys.argv:
stale = [n for n, svg in wanted.items() if not (FIGS / n).exists() or (FIGS / n).read_text() != svg]
print("figures out of date: " + ", ".join(stale) if stale else "figures up to date")
sys.exit(1 if stale else 0)
for name, svg in wanted.items():
(FIGS / name).write_text(svg)
print("wrote", FIGS / name)Show code
"""A small safety filter you can read end to end, and the arithmetic of its threshold.
The toy filter blocks one kind of message: asking for, or sharing, personal contact details (a home address,
a phone number, where someone lives) in a school or club chat. Every message is made up here from short
templates, with a fixed seed, so every number in the lesson can be reproduced. A real filter covers many
kinds of harm and is trained on far more text, but it works the same way: a score per message, compared
with a threshold.
1. make the messages (train and test), some test wordings never seen in training
2. turn each message into features: words, and pairs of letters inside each word
3. train a logistic regression with plain gradient descent (NumPy)
4. score the test messages (0 to 1), then block those at or above the threshold
5. count the four outcomes, sweep the threshold, and repeat the counts for traffic where harm is rare
Python 3 standard library and NumPy. The widget (../widget.html) repeats the counting in JavaScript;
test_widget.py runs both and compares them.
"""
from __future__ import annotations
import re
import numpy as np
SEED = 15
OPENING_PCT = 50 # the threshold the widget opens on, in hundredths
TRAFFIC = 10_000 # the second traffic mix: 10,000 messages ...
TRAFFIC_HARMFUL = 100 # ... of which 100 are harmful (1 in 100)
# --- the made-up messages ------------------------------------------------------------------------
# {w} slots are filled from the lists below. Harmless messages share words with harmful ones on purpose.
LISTS = {
"thing": ["home address", "address", "phone number", "cell number", "home phone"],
"street": ["oak", "maple", "pine", "elm", "cedar", "lake", "hill"],
"num": [str(n) for n in range(2, 60)],
"digits": [f"555 01{n:02d}" for n in range(100)],
"day": ["monday", "tuesday", "wednesday", "thursday", "friday", "saturday"],
"event": ["practice", "the meeting", "the quiz", "the bake sale", "the trip", "the game", "choir"],
"room": ["gym", "library", "cafeteria", "art room", "music room", "front office", "science lab", "auditorium"],
"item": ["jacket", "charger", "notebook", "water bottle", "hoodie", "calculator"],
"stuff": ["notes", "schedule", "photos", "slides", "worksheet", "results", "link"],
"food": ["pizza", "snacks", "cupcakes", "juice", "fruit"],
"who": ["she", "he", "your friend"],
"poss": ["her", "his", "your friend's"],
"pre": ["", "", "", "hey ", "hi ", "ok so ", "quick question "],
}
HARMFUL = [
"what's your {thing}?", "can i get your {thing}?", "send me your {thing}", "where do you live?",
"which street do you live on?", "my phone number is {digits}", "i live at {num} {street} street",
"call me on {digits}", "text me at {digits}", "my home address is {num} {street} street",
"where does {who} live?", "what's {poss} {thing}?",
]
HARMLESS = [
"the home team plays on {day}", "my number on the team is {num}",
"can someone send me the {stuff}?", "what time is {event}?", "where is the {room}?", "i forgot my {item}",
"see you at {event}", "thanks for the {stuff}", "who is bringing the {food}?", "what is the homework for {day}?",
"can i get the {stuff}?", "i will send the {stuff} after {event}", "your {item} looks great",
"what's your favorite {food}?", "where do you want to eat?", "the game is live on {day}",
"you can reach the {room} at the desk", "i will bring the {food} to {event}", "{event} is moving to {day}",
"does anyone have a spare {item}?", "great job at {event}", "i can help with the {stuff} on {day}",
]
# Wordings used only in the test set: slang, misspellings, and harmless messages that reuse harmful words.
UNSEEN_HARMFUL = [
"{do} ur digits", "whats ur {adr}", "where do u stay", "which street r u on", "{do} me ur number",
"what area do u stay in?", "ur place is on which street?", "{do} me ur digits", "what street do u stay on",
]
UNSEEN_HARMLESS = [
"what's the {room}'s address?", "the school phone number is on the website", "send me the address for {event}",
"can i get the {room} phone number?", "what's the address of the {room}?", "the home game address is on the poster",
"can i get the address for {event}?", "what's the {room}'s phone number?", "my phone battery died", "my phone is in my locker",
]
LISTS.update({"do": ["drop", "give", "send"], "adr": ["adress", "addy", "number"]})
class Rng:
"""xorshift32: the same numbers on every Python version, which random.choice does not promise."""
def __init__(self, seed: int):
self.state = seed
def pick(self, items):
s = self.state
s ^= (s << 13) & 0xFFFFFFFF
s ^= s >> 17
s ^= (s << 5) & 0xFFFFFFFF
self.state = s
return items[s % len(items)]
def fill(template: str, rng: Rng) -> str:
return re.sub(r"\{(\w+)\}", lambda m: rng.pick(LISTS[m.group(1)]), template)
def draw(rng: Rng, harmful_templates, harmless_templates, n_harmful, n_harmless, seen):
"""n messages of each kind: a list of (text, 1 if harmful else 0). No message repeats, and none is in `seen`."""
out = []
for templates, n, label in ((harmful_templates, n_harmful, 1), (harmless_templates, n_harmless, 0)):
got = tries = 0
while got < n:
tries += 1
if tries > 100 * n:
raise ValueError("the templates make too few different messages")
core = fill(rng.pick(templates), rng)
if core not in seen:
seen.add(core)
out.append((rng.pick(LISTS["pre"]) + core, label))
got += 1
return out
def make_data():
"""(train, test): train is 100 harmful + 200 harmless; test is 40 + 80, of which 16 + 36 are in wordings the filter never saw."""
rng, seen = Rng(SEED), set()
train = draw(rng, HARMFUL, HARMLESS, 100, 200, seen)
same = draw(rng, HARMFUL, HARMLESS, 24, 44, seen)
unseen = draw(rng, UNSEEN_HARMFUL, UNSEEN_HARMLESS, 16, 36, seen)
return train, same + unseen
# --- features ------------------------------------------------------------------------------------
def features(text: str) -> set[str]:
"""Each word ("w:address") and each pair of letters inside a word, with ^ and $ marking its ends ("c:ad")."""
feats = set()
for word in re.findall(r"[a-z0-9]+", text.lower()):
feats.add("w:" + word)
padded = "^" + word + "$"
feats.update("c:" + padded[i:i + 2] for i in range(len(padded) - 1))
return feats
def vocabulary(train):
"""Features seen in training, in a fixed order. A feature never seen in training gets no weight."""
return {name: i for i, name in enumerate(sorted(set().union(*(features(t) for t, _ in train))))}
def matrix(messages, vocab):
X = np.zeros((len(messages), len(vocab)))
for row, (text, _) in enumerate(messages):
for name in features(text):
if name in vocab:
X[row, vocab[name]] = 1.0
return X, np.array([label for _, label in messages], dtype=float)
# --- training and scoring ------------------------------------------------------------------------
def sigmoid(z):
return 1.0 / (1.0 + np.exp(-z))
def train_filter(X, y, steps=300, rate=0.5, l2=0.003):
"""Gradient descent on the average log-loss plus a small penalty on the weights. Returns weights, bias, loss per step."""
w, b, losses = np.zeros(X.shape[1]), 0.0, []
for _ in range(steps):
p = sigmoid(X @ w + b)
losses.append(float(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))))
err = p - y
w -= rate * (X.T @ err / len(y) + l2 * w)
b -= rate * err.mean()
return w, b, losses
def scores(X, w, b):
"""One score per message, 0 to 1, rounded to three decimals (what the widget shows)."""
return [round(float(p), 3) for p in sigmoid(X @ w + b)]
# --- counting ------------------------------------------------------------------------------------
def counts(labels, score_list, pct):
"""Block a message when its score is at least pct/100. Returns (caught, missed, falsely blocked, passed)."""
t = pct / 100
caught = sum(1 for y, s in zip(labels, score_list) if y == 1 and s >= t)
blocked_harmless = sum(1 for y, s in zip(labels, score_list) if y == 0 and s >= t)
harmful = sum(labels)
return caught, harmful - caught, blocked_harmless, len(labels) - harmful - blocked_harmless
def rates(c):
"""(miss rate, false-alarm rate, precision): missed / harmful, falsely blocked / harmless, caught / blocked."""
caught, missed, false_alarm, passed = c
blocked = caught + false_alarm
return missed / (caught + missed), false_alarm / (false_alarm + passed), (caught / blocked if blocked else None)
def sweep(labels, score_list):
"""Counts at every threshold from 0.01 to 0.99."""
return {pct: counts(labels, score_list, pct) for pct in range(1, 100)}
def crossing(table):
"""The first threshold where the miss rate is at least the false-alarm rate: where the two curves cross."""
return next(pct for pct in range(1, 100) if rates(table[pct])[0] >= rates(table[pct])[1])
def at_traffic(c, total=TRAFFIC, harmful=TRAFFIC_HARMFUL):
"""The same miss and false-alarm rates applied to `total` messages of which `harmful` are harmful.
Whole numbers, rounding halves up, so Python and the widget agree exactly.
"""
caught, missed, false_alarm, passed = c
n_harmful, n_harmless = caught + missed, false_alarm + passed
caught_t = (caught * harmful + n_harmful // 2) // n_harmful
false_t = (false_alarm * (total - harmful) + n_harmless // 2) // n_harmless
return caught_t, harmful - caught_t, false_t, total - harmful - false_t
# --- everything together ---------------------------------------------------------------------------
def build():
"""Train the filter and score the test messages. Returns a dict the tests, the numbers and the widget share."""
train, test = make_data()
vocab = vocabulary(train)
X, y = matrix(train, vocab)
w, b, losses = train_filter(X, y)
Xt, yt = matrix(test, vocab)
test_scores = scores(Xt, w, b)
return {"train": train, "test": test, "vocab": vocab, "w": w, "b": b, "losses": losses,
"labels": [int(v) for v in yt], "scores": test_scores, "sweep": sweep([int(v) for v in yt], test_scores)}
# --- the words the widget shows ----------------------------------------------------------------------
def find(test, core: str) -> int:
"""Index of the test message that ends with `core` (a random greeting may come before it)."""
return next(i for i, (text, _) in enumerate(test) if text.endswith(core))
MISSED_EXAMPLE = "drop ur digits" # harmful, in a wording the training messages never had
FALSE_ALARM_EXAMPLE = "my phone battery died" # harmless, but "phone" is a word harmful messages use
PASSED_EXAMPLE = "what's the library's address?" # harmless and passed, although it uses the word "address"
def noun(n: int) -> str:
return "message" if n == 1 else "messages"
def share(part: int, whole: int) -> int:
"""part / whole as a whole percent, halves rounded up."""
return (200 * part + whole) // (2 * whole)
def view(labels, score_list, pct: int, traffic: bool = False) -> dict:
"""What the widget shows at one threshold: the four counts, a sentence, and the share of blocked messages that were harmful.
traffic=False counts the test messages; traffic=True applies the same rates to TRAFFIC messages of which TRAFFIC_HARMFUL are harmful,
so its counts are expected values and its sentences say "about".
The widget offers thresholds 1 to 99; at every one of them the filter blocks at least 14 test messages, so `blocked` is never 0.
"""
c = counts(labels, score_list, pct)
if traffic:
c = at_traffic(c)
caught, missed, false_alarm, passed = c
t = f"0.{pct:02d}"
if traffic:
says = (f"Out of {TRAFFIC:,} messages, {TRAFFIC_HARMFUL} harmful: at {t} the filter would miss about {missed} "
f"and block about {false_alarm:,} harmless {noun(false_alarm)}.")
else:
says = f"At {t} the filter misses {missed} harmful {noun(missed)} and blocks {false_alarm} harmless {noun(false_alarm)}."
blocked = caught + false_alarm
if traffic:
precision = f"Of the {blocked:,} or so messages it would block, about {caught:,} are harmful ({share(caught, blocked)}%)."
else:
precision = f"Of the {blocked:,} messages it blocks, {caught:,} are harmful ({share(caught, blocked)}%)."
return {"counts": list(c), "says": says, "precision": precision}
def readout(text: str, label: int, score: float, pct: int) -> str:
"""The one line shown for a chosen message."""
blocked = score >= pct / 100
verdict = {(1, True): "caught", (1, False): "a miss", (0, True): "a false alarm", (0, False): "correct"}[(label, blocked)]
return f"“{text}” ({'harmful' if label else 'harmless'}) scores {score:.3f}, so it is {'blocked' if blocked else 'passed'}: {verdict}."
def explain(text: str, result: dict):
"""The features of one message the filter has weights for, those weights, the features it never saw, and the sum before the sigmoid."""
vocab, w = result["vocab"], result["w"]
feats = features(text)
known = sorted((name, float(w[vocab[name]])) for name in feats if name in vocab)
unseen = sorted(name for name in feats if name not in vocab)
return known, unseen, result["b"] + sum(v for _, v in known)Show code
"""Print every number the lesson quotes, from safety_filter.py. Run: python3 show_numbers.py"""
import re
import safety_filter as sf
R = sf.build()
LABELS, SCORES, TEST, VOCAB, W = R["labels"], R["scores"], R["test"], R["vocab"], R["w"]
SAME = 68 # the first 68 test messages use wordings the training messages had; the other 52 do not
def pct_text(x):
return f"{100 * x:.2f}%"
def row(p, c):
miss, alarm, precision = sf.rates(c)
return f" 0.{p:02d} caught {c[0]:>5,} missed {c[1]:>3} false alarms {c[2]:>5,} passed {c[3]:>5,} miss {pct_text(miss):>7} false-alarm {pct_text(alarm):>7} precise {pct_text(precision):>7} ({sf.share(c[0], c[0] + c[2])}%)"
print("DATA")
harm_train = sum(y for _, y in R["train"])
print(f" train {len(R['train'])} ({harm_train} harmful, {len(R['train']) - harm_train} harmless); test {len(TEST)} ({sum(LABELS)} harmful, {len(LABELS) - sum(LABELS)} harmless)")
print(f" test messages in wordings never seen in training: {len(TEST) - SAME} ({sum(LABELS[SAME:])} harmful, {len(TEST) - SAME - sum(LABELS[SAME:])} harmless)")
print(f" features the filter has weights for: {len(VOCAB)}; loss {R['losses'][0]:.4f} at step 1, {R['losses'][-1]:.4f} at step {len(R['losses'])}; bias {R['b']:.3f}")
words = sorted(((float(W[i]), name[2:]) for name, i in VOCAB.items() if name.startswith("w:")), reverse=True)
print(" highest word weights:", ", ".join(f"{name} {v:.2f}" for v, name in words[:9]))
print(" lowest word weights: ", ", ".join(f"{name} {v:.2f}" for v, name in words[-3:]))
for word in ("address", "phone", "number", "street", "the"):
print(f" weight of '{word}': {float(W[VOCAB['w:' + word]]):.3f}")
for word in ("phone", "address", "the"):
in_train = lambda label: sum(1 for text, y in R["train"] if y == label and word in re.findall(r"[a-z0-9]+", text))
print(f" training messages with '{word}': {in_train(1)} of {harm_train} harmful, {in_train(0)} of {len(R['train']) - harm_train} harmless")
print("SCORES (test messages)")
print(f" lowest harmful score {min(s for y, s in zip(LABELS, SCORES) if y):.2f}; highest harmless score {max(s for y, s in zip(LABELS, SCORES) if not y):.2f}")
for lo, hi in ((0, .05), (.05, .25), (.25, .5), (.5, .9), (.9, 1.01)):
h = sum(1 for y, s in zip(LABELS, SCORES) if y and lo <= s < hi)
n = sum(1 for y, s in zip(LABELS, SCORES) if not y and lo <= s < hi)
print(f" score {lo:.2f} to {min(hi, 1):.2f}: {h:>2} harmful, {n:>2} harmless")
print("THRESHOLD SWEEP (test messages)")
for p in (1, 10, 25, 30, 35, 40, 45, 50, 60, 70, 80, 90, 99):
print(row(p, R["sweep"][p]))
errors = {p: c[1] + c[2] for p, c in R["sweep"].items()}
best = min(errors.values())
fewest = [p for p, e in errors.items() if e == best]
print(f" fewest mistakes: {best}, at {len(fewest)} thresholds from 0.{min(fewest):02d} to 0.{max(fewest):02d}")
print(f" miss rate at 0.99: {100 * R['sweep'][99][1] / sum(LABELS):.0f}%; false-alarm rate at 0.01: {100 * R['sweep'][1][2] / (len(LABELS) - sum(LABELS)):.1f}%")
cross = sf.crossing(R["sweep"])
print(f" at the crossing, miss {R['sweep'][cross][1]} of {sum(LABELS)} = {100 * sf.rates(R['sweep'][cross])[0]:.1f}%, false alarms {R['sweep'][cross][2]} of {len(LABELS) - sum(LABELS)} = {100 * sf.rates(R['sweep'][cross])[1]:.1f}%")
print(f" miss count is 0 up to 0.{max(p for p, c in R['sweep'].items() if c[1] == 0):02d}; false-alarm count is 0 from 0.{min(p for p, c in R['sweep'].items() if c[2] == 0):02d}")
print(f" thresholds where both mistakes are 0: {[p for p, c in R['sweep'].items() if c[1] == 0 and c[2] == 0]}")
print(f" miss rate first reaches the false-alarm rate at 0.{sf.crossing(R['sweep']):02d}")
print(f" at 0.01 no harmful message is missed but {R['sweep'][1][2]} harmless are blocked; at 0.99 {R['sweep'][99][1]} harmful are missed and {R['sweep'][99][2]} harmless blocked")
print(f"RARE HARM: {sf.TRAFFIC:,} messages, {sf.TRAFFIC_HARMFUL} harmful (1 in {sf.TRAFFIC // sf.TRAFFIC_HARMFUL}), same miss and false-alarm rates")
c50 = R["sweep"][50]
m, a, pr = sf.rates(c50)
print(f" test messages at 0.50: miss rate {c50[1]}/{sum(LABELS)} = {100 * m:.1f}%, false-alarm rate {c50[2]}/{len(LABELS) - sum(LABELS)} = {100 * a:.2f}%")
print(f" by hand: {sf.TRAFFIC_HARMFUL} x {100 * m:.1f}% = {sf.TRAFFIC_HARMFUL * m:.1f} missed; {sf.TRAFFIC - sf.TRAFFIC_HARMFUL:,} x {100 * a:.2f}% = {(sf.TRAFFIC - sf.TRAFFIC_HARMFUL) * a:.2f} falsely blocked")
caught_t, _, false_t, _ = sf.at_traffic(c50)
print(f" caught by hand: {sf.TRAFFIC_HARMFUL} x {100 * (1 - m):.1f}% = {sf.TRAFFIC_HARMFUL * (1 - m):.1f}; blocked in all {caught_t} + {false_t} = {caught_t + false_t}; harmful share {caught_t} / {caught_t + false_t} = {100 * caught_t / (caught_t + false_t):.1f}%")
for p in (30, 40, 50, 60, 70, 90):
t = sf.at_traffic(R["sweep"][p])
print(row(p, t))
if p == 50:
print(f" harmless share of the {t[0] + t[2]} blocked: {sf.share(t[2], t[0] + t[2])}%; harmful share {sf.share(t[0], t[0] + t[2])}%")
for p in range(1, 100):
t = sf.at_traffic(R["sweep"][p])
if t[0] >= t[2]: # first threshold where at least half of the blocked messages are harmful
print(f" at least half of the blocked messages are harmful from 0.{p:02d}, where the test messages have no false alarm left:")
print(row(p, t))
break
v = sf.view(LABELS, SCORES, 50, True)
print(f" one more or one fewer false alarm among the {len(LABELS) - sum(LABELS)} harmless test messages moves the {sf.TRAFFIC:,}-message count by about {(sf.TRAFFIC - sf.TRAFFIC_HARMFUL) / (len(LABELS) - sum(LABELS)):.0f}")
print(f" widget sentence, test mix: {sf.view(LABELS, SCORES, 50)['says']} {sf.view(LABELS, SCORES, 50)['precision']}")
print(f" widget sentence, traffic mix: {v['says']} {v['precision']}")
wrong = [i for i in range(len(TEST)) if (SCORES[i] >= .5) != bool(LABELS[i])]
print(f"MISTAKES AT 0.50: {len(wrong)}, of which {sum(1 for i in wrong if i >= SAME)} in wordings never seen in training and {sum(1 for i in wrong if i < SAME)} in the other {SAME} messages")
for i, (text, y) in enumerate(TEST):
if (SCORES[i] >= .5) != bool(y):
print(f" {'harmful, missed' if y else 'harmless, blocked':<18} {SCORES[i]:.3f} {'unseen' if i >= SAME else 'seen '} {text}")
print("WHY (the filter's sum for the three examples, before the sigmoid squeezes it into 0 to 1)")
for core in (sf.MISSED_EXAMPLE, sf.FALSE_ALARM_EXAMPLE, sf.PASSED_EXAMPLE):
i = sf.find(TEST, core)
known, unseen, z = sf.explain(TEST[i][0], R)
print(f" {sf.readout(TEST[i][0], LABELS[i], SCORES[i], 50)}")
print(f" sum {z:+.3f} = bias {R['b']:+.3f} + " + " + ".join(f"{name} {v:+.3f}" for name, v in known if name.startswith("w:")) + f" + {sum(v for n, v in known if n.startswith('c:')):+.3f} from {sum(1 for n, _ in known if n.startswith('c:'))} letter pairs")
print(f" words with no weight: {[n[2:] for n in unseen if n.startswith('w:')]}; letter pairs with no weight: {sum(1 for n in unseen if n.startswith('c:'))}")Show code
"""Guards for safety_filter.py: the data, the training, the counting at every threshold, the rare-harm arithmetic, and the failures the lesson shows."""
import math
import re
import textwrap
from fractions import Fraction
from pathlib import Path
import pytest
import make_figures
import safety_filter as sf
LESSON = Path(__file__).resolve().parent.parent
SAME = 68 # the first 68 test messages use wordings the training messages had; the other 52 do not
@pytest.fixture(scope="module")
def r():
return sf.build()
def test_data_has_the_sizes_the_lesson_names_and_the_test_never_repeats_a_training_message(r):
"""Guards a test set that quietly overlaps the training set, which would make the filter look better than it is."""
labels = r["labels"]
assert [len(r["train"]), sum(y for _, y in r["train"])] == [300, 100]
assert [len(r["test"]), sum(labels), sum(labels[SAME:]), len(labels) - SAME] == [120, 40, 16, 52]
assert not {t for t, _ in r["train"]} & {t for t, _ in r["test"]}
assert sf.make_data()[1] == r["test"] # fixed seed: the same messages every run
assert not set(sf.UNSEEN_HARMFUL) & set(sf.HARMFUL) and not set(sf.UNSEEN_HARMLESS) & set(sf.HARMLESS) # "never seen" means no template in common
def test_training_loss_starts_at_ln_2_and_never_rises(r):
"""Guards a training loop that stopped learning: a filter with all weights zero is a coin flip, whose loss is ln 2."""
losses = r["losses"]
assert losses[0] == pytest.approx(math.log(2)) and losses[-1] < 0.05
assert all(b <= a + 1e-12 for a, b in zip(losses, losses[1:]))
def test_counts_add_up_and_move_one_way_as_the_threshold_rises(r):
"""Guards the sweep behind the lesson's curves: at every threshold the four counts cover all 120 messages, a higher threshold
only ever misses more and falsely blocks fewer, and the rates at 0.50 are the ones the lesson names."""
table = r["sweep"]
for p in range(1, 100):
caught, missed, false_alarm, passed = table[p]
assert (caught + missed, false_alarm + passed) == (40, 80)
assert all(table[p + 1][1] >= table[p][1] and table[p + 1][2] <= table[p][2] for p in range(1, 99))
miss, alarm, precision = sf.rates(table[50])
assert (table[50], miss, alarm) == ((35, 5, 1, 79), 5 / 40, 1 / 80) and round(100 * precision) == 97
def test_no_threshold_is_right_for_every_message(r):
"""Guards the lesson's main claim: no threshold has both mistakes at zero, the two rates cross, and each end of the range
trades one mistake for the other."""
table = r["sweep"]
assert not [p for p, c in table.items() if c[1] == 0 and c[2] == 0]
assert 1 < sf.crossing(table) < 99
assert table[1][1] == 0 < table[1][2] and table[99][2] == 0 < table[99][1]
assert min(c[1] + c[2] for c in table.values()) == 6
def test_a_message_scoring_exactly_the_threshold_is_blocked():
assert sf.counts([1, 0], [0.4, 0.4], 40) == (1, 0, 1, 0) and sf.counts([1, 0], [0.399, 0.399], 40) == (0, 1, 0, 1)
def test_rare_harm_arithmetic_matches_exact_fractions_and_flips_the_precision(r):
"""Guards the base-rate numbers: at 1 harmful in 100 the same rates block mostly harmless messages. The widget and the lesson
use whole numbers, rounding halves up, so check them against exact fractions at every threshold."""
for p in range(1, 100):
caught, missed, false_alarm, passed = r["sweep"][p]
c, m, f, ok = sf.at_traffic(r["sweep"][p])
assert (c + m, f + ok) == (100, 9900)
assert abs(Fraction(c) - Fraction(100 * caught, 40)) <= Fraction(1, 2) and abs(Fraction(f) - Fraction(9900 * false_alarm, 80)) <= Fraction(1, 2)
assert sf.at_traffic(r["sweep"][50]) == (88, 12, 124, 9776)
on_test, on_traffic = sf.view(r["labels"], r["scores"], 50), sf.view(r["labels"], r["scores"], 50, True)
assert on_test["precision"].endswith("(97%).") and on_traffic["precision"].endswith("(42%).")
assert sf.share(88, 212) == 42 and sf.share(1, 2) == 50 and sf.share(1, 8) == 13 # a half rounds up
def test_the_three_examples_fail_or_pass_for_the_reasons_the_lesson_gives(r):
"""Guards the lesson's examples: a harmful message in new words is missed because its words carry no weight, a harmless
message is blocked because it uses a word harmful messages use, and a harmless message with the word 'address' still passes."""
test, scores, labels, vocab, w = r["test"], r["scores"], r["labels"], r["vocab"], r["w"]
miss, alarm, passed = (sf.find(test, c) for c in (sf.MISSED_EXAMPLE, sf.FALSE_ALARM_EXAMPLE, sf.PASSED_EXAMPLE))
assert (labels[miss], scores[miss] < 0.5, labels[alarm], scores[alarm] >= 0.5, labels[passed], scores[passed] < 0.5) == (1, True, 0, True, 0, True)
known, unseen, _ = sf.explain(test[miss][0], r)
assert {"w:drop", "w:ur", "w:digits"} <= set(unseen) and not {"w:drop", "w:ur", "w:digits"} & set(vocab)
harmful_train = " ".join(t for t, y in r["train"] if y)
assert "phone" in harmful_train and "battery" not in " ".join(t for t, _ in r["train"]) and float(w[vocab["w:phone"]]) > 0.4
assert "address" in harmful_train and float(w[vocab["w:address"]]) > 0.4 and scores[passed] < 0.3
mistakes = [i for i in range(len(test)) if (scores[i] >= 0.5) != bool(labels[i])]
assert len(mistakes) == 6 and min(mistakes) >= SAME # every mistake is in wording the filter never saw
def test_explain_adds_up_to_the_score(r):
"""Guards the worked table in the lesson: bias plus the weights of a message's features, through the sigmoid, is its score."""
for core in (sf.MISSED_EXAMPLE, sf.FALSE_ALARM_EXAMPLE, sf.PASSED_EXAMPLE):
i = sf.find(r["test"], core)
known, _, z = sf.explain(r["test"][i][0], r)
assert round(float(sf.sigmoid(z)), 3) == r["scores"][i]
assert z == pytest.approx(r["b"] + sum(v for _, v in known))
def test_the_toy_uses_only_numpy_and_the_standard_library_and_no_network():
source = Path(sf.__file__).read_text()
assert set(re.findall(r"^(?:from|import) (\w+)", source, flags=re.M)) == {"__future__", "re", "numpy"}
def test_every_python_block_in_the_lesson_is_a_verbatim_excerpt_of_the_source():
"""The lesson quotes this file. If the code changes and the lesson does not, the reader sees code that was never tested."""
lesson = (LESSON / "lesson.md").read_text()
source = Path(sf.__file__).read_text().splitlines()
blocks = re.findall(r"^```python\n(.*?)^```", lesson, flags=re.S | re.M)
assert blocks, "the lesson has no python blocks to check"
for block in blocks:
lines = block.rstrip("\n").splitlines()
windows = (textwrap.dedent("\n".join(source[i:i + len(lines)])) for i in range(len(source) - len(lines) + 1))
assert textwrap.dedent("\n".join(lines)) in windows, f"not in safety_filter.py:\n{block}"
def test_the_lesson_quotes_the_numbers_the_code_gives(r):
"""Guards prose that drifts from the code: the numbers in the opening paragraph, both threshold tables, the weight table, the
worked examples, the by-hand arithmetic, the sweep text and the Check-yourself answers are recomputed here and must appear
in lesson.md."""
lesson = (LESSON / "lesson.md").read_text()
sweep, train, test, scores, labels, w, vocab = (r[k] for k in ("sweep", "train", "test", "scores", "labels", "w", "vocab"))
caught, missed, false_alarm, passed = sweep[50]
harmful, harmless = caught + missed, false_alarm + passed
caught_t, missed_t, false_t, _ = traffic = sf.at_traffic(sweep[50])
blocked_t, harmless_t = caught_t + false_t, sf.TRAFFIC - sf.TRAFFIC_HARMFUL
miss, alarm, _ = sf.rates(sweep[50])
assert f"misses {missed} of {harmful} harmful test messages and blocks {false_alarm} of {harmless} harmless ones" in lesson
assert f"would block about {traffic[0] + false_t} of every 10,000 messages, and about {false_t} of those would be harmless" in lesson
assert f"row of {len(vocab)} numbers" in lesson and f"loss reaches {r['losses'][-1]:.4f}" in lesson
for p in (1, 10, 25, 30, 40, 50, 60, 70, 90, 99):
c = sweep[p]
assert f"| 0.{p:02d} | {c[1]} | {c[2]} |" in lesson
for p in (30, 40, 50, 60, 66, 70, 90):
c, _, f, _ = sf.at_traffic(sweep[p])
assert f"| 0.{p:02d} | {f:,} | {sf.share(c, c + f)}% |" in lesson
words = lambda text: set(re.findall(r"[a-z0-9]+", text))
in_train = lambda word, label: sum(1 for t, y in train if y == label and word in words(t))
assert not (in_train("phone", 0) or in_train("address", 0) or in_train("the", 1)) and in_train("phone", 1) == in_train("address", 1)
miss_i, alarm_i, pass_i = (sf.find(test, c) for c in (sf.MISSED_EXAMPLE, sf.FALSE_ALARM_EXAMPLE, sf.PASSED_EXAMPLE))
wrong, unseen = [i for i in range(len(test)) if (scores[i] >= 0.5) != bool(labels[i])], len(test) - SAME
fewest = [p for p, c in sweep.items() if c[1] + c[2] == min(c[1] + c[2] for c in sweep.values())]
cross, half = sf.crossing(sweep), next(p for p in range(1, 100) if sf.at_traffic(sweep[p])[0] >= sf.at_traffic(sweep[p])[2])
worked = [] # the two worked tables, row by row
for i in (miss_i, alarm_i):
known, _, z = sf.explain(test[i][0], r)
pairs = [v for n, v in known if n.startswith("c:")]
parts = [r["b"], sum(pairs)] + [v for n, v in known if n.startswith("w:")]
assert abs(sum(round(v, 3) for v in parts) - round(z, 3)) < 0.002 # the lesson says the shown parts add to the sum within 0.002
worked += [f"| Bias | {r['b']:.3f} |", f"| Letter pairs ({len(pairs)}) | {sum(pairs):.3f} |", f"| Sum | {z:.3f} |", f"| Score | {scores[i]:.3f} |"]
worked += [f'| Word "{n[2:]}" | {v:.3f} |' for n, v in known if n.startswith("w:")]
known_p, _, z_p = sf.explain(test[pass_i][0], r)
weight = lambda word: float(w[vocab["w:" + word]])
quoted = [
# the toy, the data and the by-hand arithmetic
f"learned from {len(train)} made-up messages and is tested on {len(test)} others",
f"There are {len(train)} training messages ({sum(y for _, y in train)} harmful and {sum(1 - y for _, y in train)} harmless) and {len(test)} test messages "
f"({sum(labels)} harmful and {harmless} harmless)",
f"{SAME} use wordings that training also had. The other {unseen} ({sum(labels[SAME:])} harmful and {unseen - sum(labels[SAME:])} harmless)",
f"1 message in {len(test) // sum(labels)} is harmful", f"1 message in {sf.TRAFFIC // sf.TRAFFIC_HARMFUL} is harmful",
f"Of {sf.TRAFFIC:,} messages, {sf.TRAFFIC_HARMFUL} are harmful and {harmless_t:,} are not",
f"blocks about {caught_t} harmful messages and about {false_t} harmless ones. So {sf.share(false_t, blocked_t)}% of what it blocks is harmless, "
f"although its false-alarm rate is a low {100 * alarm:.2f}%",
f"Miss rate: {missed} missed of {harmful} harmful is {100 * miss:.1f}%", f"False-alarm rate: {false_alarm} blocked of {harmless} harmless is {100 * alarm:.2f}%",
f"{sf.TRAFFIC_HARMFUL} x {100 * (1 - miss):.1f}% = {sf.TRAFFIC_HARMFUL * (1 - miss):.1f}, rounded up to {caught_t}. The other {missed_t} are missed",
f"{harmless_t:,} x {100 * alarm:.2f}% = {harmless_t * alarm:.2f}, rounded up to {false_t}",
f"{caught_t} + {false_t} = {blocked_t}. The harmful share is {caught_t} / {blocked_t} = {100 * caught_t / blocked_t:.1f}%, which rounds to {sf.share(caught_t, blocked_t)}%",
# the sweep: the text, the alt text of its figure and the rows beside them
f"at 0.01 no harmful message is missed but {sweep[1][2]} of {harmless} harmless ones are blocked. At 0.99 no harmless message is blocked but "
f"{sweep[99][1]} of {harmful} harmful ones are missed",
f"is 0 until about 0.{max(p for p, c in sweep.items() if c[1] == 0):02d} and then climbs to {100 * sweep[99][1] / harmful:.0f}% at 0.99",
f"starts at {100 * sweep[1][2] / harmless:.1f}% at 0.01 and falls to 0 at about 0.{min(p for p, c in sweep.items() if c[2] == 0):02d}. They cross near 0.{cross:02d}",
f"At 0.50 the miss rate is {100 * miss:.1f}% and the false-alarm rate is {100 * alarm:.2f}%",
f"Caught is {harmful} minus missed, and passed is {harmless} minus alarms",
f"is {min(c[1] + c[2] for c in sweep.values())}, reached at {len(fewest)} thresholds from 0.{min(fewest):02d} to 0.{max(fewest):02d}",
f"at 0.{cross:02d}, where each is {100 * sf.rates(sweep[cross])[0]:.1f}% ({sweep[cross][1]} of {harmful} and {sweep[cross][2]} of {harmless})",
f"0.25 misses none of the {harmful} harmful test messages and blocks {sweep[25][2]} of {harmless} harmless ones",
f"at least half from 0.{half:02d}, where the test messages have no false alarm left, and by then about {sf.at_traffic(sweep[half])[1]} of the {sf.TRAFFIC_HARMFUL} harmful messages are missed",
"the parts add to the sum only within 0.002",
f"The {false_alarm} false alarm is {false_alarm} of only {harmless} harmless test messages, so one more or one fewer would move the {sf.TRAFFIC:,}-message count by about "
f"{harmless_t / harmless:.0f}, and 0 false alarms in {harmless} is not a rate of 0",
# the rates and counts at 0.50 as the text, the figures' alt text and the questions give them
f"At 0.50 the toy misses {missed} of {harmful} harmful test messages ({100 * miss:.1f}%) and raises {false_alarm} false alarm among {harmless} harmless ones ({100 * alarm:.2f}%)",
f"If only 1 message in {sf.TRAFFIC // sf.TRAFFIC_HARMFUL} is harmful",
f"at threshold 0.50 on {len(test)} test messages: {missed} missed, {caught} caught, {passed} passed correctly, {false_alarm} false alarm",
f"leaves {missed} harmful messages on the passed side and {false_alarm} harmless message on the blocked side",
f"Of {caught + false_alarm} blocked test messages, {caught} are harmful ({sf.share(caught, caught + false_alarm)}%). Of about {blocked_t} blocked in {sf.TRAFFIC:,} messages where "
f"1 in {sf.TRAFFIC // sf.TRAFFIC_HARMFUL} is harmful, about {caught_t} are harmful ({sf.share(caught_t, blocked_t)}%) and about {false_t} are harmless ({sf.share(false_t, blocked_t)}%)",
f"is {r['losses'][0]:.4f}, which is ln 2", f"Each of the {len(r['losses'])} steps of gradient descent",
f"every threshold from 0.01 to 0.99. This table shows some of them on the {len(test)} test messages",
f"for {sf.TRAFFIC:,} messages of which {sf.TRAFFIC_HARMFUL} are harmful", f"to {sf.TRAFFIC:,} messages of which {sf.TRAFFIC_HARMFUL} are harmful. By hand at 0.50:",
f"Treat the {false_t} with care", f"{len(test)} messages, with {harmless} harmless", f"the {sf.share(caught_t, blocked_t)}% above",
f"Here 0.50 sits inside the range, 0.{min(fewest):02d} to 0.{max(fewest):02d}, where the test set has its fewest mistakes ({min(c[1] + c[2] for c in sweep.values())})",
f"many unseen wordings on purpose ({unseen} of {len(test)})",
f"At 1 harmful message in {sf.TRAFFIC // sf.TRAFFIC_HARMFUL}, the toy filter's false-alarm rate is a low {100 * alarm:.2f}%",
f"Why are all {len(wrong)} mistakes at 0.50 in wordings the toy never saw",
f"Also, the {min(c[1] + c[2] for c in sweep.values())} counts both kinds equally on a test set that is 1 harmful in {len(test) // sum(labels)}, and at 1 in {sf.TRAFFIC // sf.TRAFFIC_HARMFUL}",
f"The {sf.TRAFFIC:,}-message count at 0.50 is {false_t} false alarms",
f"It comes from {false_alarm} false alarm among only {harmless} harmless test messages. One more or one fewer would move it by about {harmless_t / harmless:.0f}, "
f"and 0 in {harmless} would not be a rate of 0",
# the examples, the weight table and the worked tables
f'"{test[miss_i][0]}", a request for a phone number in slang, scores {scores[miss_i]:.3f} and passes at 0.50. "{test[alarm_i][0]}" scores {scores[alarm_i]:.3f} and is blocked',
f"All {len(wrong)} mistakes at 0.50 are in wordings", f"None of the {SAME} test messages in wordings it did see is a mistake",
f'"phone" and "address" each appear in {in_train("phone", 1)} harmful training messages and in no harmless one. "the" appears in {in_train("the", 0)} of the '
f"{sum(1 - y for _, y in train)} harmless training messages and in none of the harmful ones",
*(f"| {word} | {weight(word):.2f} |" for word in ("555", "live", "phone", "address", "street", "home", "me", "number", "is", "great", "the")),
*worked,
f'"phone" carries +{weight("phone"):.3f}', f"push the letter-pair total to +{sum(v for n, v in sf.explain(test[alarm_i][0], r)[0] if n.startswith('c:')):.3f}",
f'has "address" at +{weight("address"):.2f}, but "the" at {weight("the"):.2f} and its other features bring the sum to {z_p:.3f} and the score to {scores[pass_i]:.3f}',
# the Check-yourself answers
f"There are {harmless_t // sf.TRAFFIC_HARMFUL} harmless messages for every harmful one. 1.25% of {harmless_t:,} harmless messages is about {false_t}, more than the "
f"roughly {caught_t} harmful messages caught, so about {false_t} of the roughly {blocked_t} blocked messages ({sf.share(false_t, blocked_t)}%) are harmless",
f"the fewest mistakes is {min(c[1] + c[2] for c in sweep.values())}", f"{false_alarm} false alarm among only {harmless} harmless test messages",
]
missing = [text for text in quoted if text not in lesson]
assert not missing, missing
def test_the_figures_are_current_and_follow_the_house_rules(r):
"""Guards a figure left stale after the filter changed, a colour the site rules out, and the two hand-drawn figures
(flow.svg, outcomes.svg), whose numbers must be the counts and the score the code gives."""
for name, svg in make_figures.figures().items():
assert (LESSON / "figs" / name).read_text() == svg, f"{name} is out of date: run make_figures.py"
for path in (LESSON / "figs").glob("*.svg"):
svg = path.read_text()
assert not re.search(r"0072B2|56B4E9|D55E00|<script|<image|href=|https?://(?!www\.w3\.org)", svg, flags=re.I), path.name
assert "<title>" in svg and "<desc>" in svg and 'fill="#ffffff"' in svg
caught, missed, false_alarm, passed = r["sweep"][50]
caught_t, _, false_t, _ = sf.at_traffic(r["sweep"][50]) # traffic counts are expected counts, so the figure says "about"
rare = (LESSON / "figs" / "rare.svg").read_text()
assert all(f"about {n}" in rare for n in (caught_t + false_t, caught_t, false_t)) and f"about {caught_t + false_t} of every" in rare
outcomes = (LESSON / "figs" / "outcomes.svg").read_text()
assert [int(n) for n in re.findall(r">(\d+)</text>", outcomes)] == [missed, caught, passed, false_alarm]
assert f"{len(r['labels'])} test messages, threshold 0.50" in outcomes
X, _ = sf.matrix([("hey i live at 36 oak street", 1)], r["vocab"])
score = sf.scores(X, r["w"], r["b"])[0]
flow = (LESSON / "figs" / "flow.svg").read_text()
assert score >= 0.5 and f"Here: {score:.2f}" in flow and "Here: 0.50" in flowShow code
"""Guards for ../widget.html: its data block, its size, that its counting gives the same numbers and words as safety_filter.py, and what a reader sees."""
import json
import re
import subprocess
import safety_filter as sf
from embed_widget import CONFIG_BLOCK, GEO, WIDGET, config, embed
HTML = WIDGET.read_text(encoding="utf-8")
ROOT = HTML[HTML.index('<div class="sfw-root">'):HTML.index("<script")]
CONFIG = config()
MSGS = CONFIG["messages"] # the test messages in score order, each {t, h, s, y}
LABELS, SCORES = [m["h"] for m in MSGS], [m["s"] for m in MSGS]
THRESHOLDS = range(1, 100)
OPENING = CONFIG["opening"]["pick"]
COMMAS = [0, 7, 999, 1000, 12345, 1234567]
SHARES = [(35, 36), (1, 2), (1, 8), (88, 212), (14, 14)] # (1, 2) and (1, 8) end in a half that must round up
# Reads the model out of the widget between its marker comments, runs it with Node, and compares every sentence, count and readout
# with the ones Python wrote into the fixture, character for character.
PARITY = r"""
import fs from 'node:fs';
const [fixture, widget] = process.argv.slice(2);
const fx = JSON.parse(fs.readFileSync(fixture, 'utf8'));
const html = fs.readFileSync(widget, 'utf8');
const cfg = JSON.parse(html.match(/<script type="application\/json" id="sfw-config">([\s\S]*?)<\/script>/)[1]);
const js = html.slice(html.indexOf('/* ---------- the model ('), html.indexOf('/* ---------- display'));
const api = new Function('cfg', "'use strict';" + js + '; return {counts, atTraffic, commas, noun, share, view, readout};')(cfg);
const bad = [];
let checks = 0;
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
for (const rare of [false, true]) {
for (let pct = 1; pct <= 99; pct++) {
const v = api.view(pct, rare), want = fx.views[String(rare)][String(pct)];
checks++;
if (!same({ counts: v.counts, says: v.says, precision: v.precision }, want)) bad.push(['view', pct, rare, v, want]);
}
}
for (let pct = 1; pct <= 99; pct++) {
cfg.messages.forEach((m, i) => {
checks++;
if (api.readout(m, pct) !== fx.readouts[String(pct)][i]) bad.push(['readout', pct, m.t, api.readout(m, pct), fx.readouts[String(pct)][i]]);
});
}
fx.commas.forEach(([n, text]) => { checks++; if (api.commas(n) !== text) bad.push(['commas', n, api.commas(n), text]); });
fx.shares.forEach(([part, whole, text]) => { checks++; if (api.share(part, whole) !== text) bad.push(['share', part, whole, api.share(part, whole), text]); });
console.log(JSON.stringify({ checks, bad: bad.slice(0, 4) }));
process.exit(bad.length ? 1 : 0);
"""
# Runs the page script against a small stand-in for the DOM, moves the slider, presses the buttons, taps the plot and ticks the box,
# and reads back what a reader sees.
PAGE = r"""
import fs from 'node:fs';
const [widget, fixture] = process.argv.slice(2);
const html = fs.readFileSync(widget, 'utf8');
const fx = JSON.parse(fs.readFileSync(fixture, 'utf8'));
const config = html.match(/<script type="application\/json" id="sfw-config">([\s\S]*?)<\/script>/)[1];
const code = html.match(/<script>([\s\S]*?)<\/script>/)[1];
function el(tag) {
const e = { tag, listeners: {}, value: '', checked: false, attrs: {}, textContent: '', innerHTML: '' };
e.setAttribute = (k, v) => { e.attrs[k] = v; };
e.addEventListener = (type, f) => (e.listeners[type] = e.listeners[type] || []).push(f);
e.getBoundingClientRect = () => e.rect || { left: 0, top: 0, width: 300, height: 214 };
return e;
}
function open() {
const page = { root: el('html'), messages: [], parent: { other: true } };
fx.ids.forEach((id) => { page[id] = el('x'); });
page['sfw-config'].textContent = config;
const doc = { getElementById: (id) => page[id], documentElement: page.root };
const win = { addEventListener: (type, f) => page.messages.push(f) };
new Function('document', 'window', 'parent', code)(doc, win, page.parent);
return page;
}
const fire = (e, type, extra) => (e.listeners[type || 'click'] || []).forEach((f) => f({ target: e, ...extra }));
const bad = [];
const want = (what, got, expected) => { if (JSON.stringify(got) !== JSON.stringify(expected)) bad.push([what, got, expected]); };
const near = (what, got, expected) => { if (!(Math.abs(Number(got) - expected) < 0.06)) bad.push([what, got, expected]); };
const count = (text, needle) => text.split(needle).length - 1;
const group = (n) => String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ',');
const spoken = (v, read) => v.says + ' ' + v.precision + ' ' + read;
function look(page, pct, rare, pick, what) { // compares the page with what Python expects at one threshold and one choice
const v = fx.views[String(rare)][String(pct)], x = 10 + (pct / 100) * 280, line = page['sfw-line'].attrs, shade = page['sfw-shade'].attrs;
want(what + ': sentence', page['sfw-says'].textContent, v.says);
want(what + ': threshold text', page['sfw-t'].textContent, fx.texts[String(pct)]);
want(what + ': counts', [page['sfw-caught'], page['sfw-missed'], page['sfw-false'], page['sfw-passed']].map((e) => e.textContent), v.counts.map(group));
want(what + ': precision line', page['sfw-prec'].textContent, v.precision);
want(what + ': readout', page['sfw-read'].textContent, fx.readouts[String(pct)][pick]);
near(what + ': line x1', line.x1, x); near(what + ': line x2', line.x2, x); near(what + ': shade x', shade.x, x); near(what + ': shade width', shade.width, 290 - x);
const marks = page['sfw-marks'].innerHTML;
want(what + ': mark counts', [count(marks, 'class="sfw-h"'), count(marks, 'class="sfw-l"'), count(marks, 'class="sfw-pick"')], [fx.harmful, fx.harmless, 1]);
want(what + ': rings are the mistakes on the test messages', count(marks, 'class="sfw-ring"'), fx.mistakes[String(pct)]);
const m = fx.messages[pick], at = marks.match(/<(?:rect|circle) class="sfw-pick"([^>]*)>/)[1];
if (m.h) { near(what + ': pick x', Number(at.match(/ x="([\d.-]+)"/)[1]) + 7, fx.xs[pick]); near(what + ': pick y', Number(at.match(/ y="([\d.-]+)"/)[1]) + 7, fx.ys[pick]); }
else { near(what + ': pick cx', at.match(/cx="([\d.-]+)"/)[1], fx.xs[pick]); near(what + ': pick cy', at.match(/cy="([\d.-]+)"/)[1], fx.ys[pick]); }
}
const first = open();
want('slider at the opening', [first['sfw-th'].value, first['sfw-rare'].checked, first['sfw-live'].textContent], [String(fx.openingPct), false, '']);
look(first, fx.openingPct, false, fx.openingPick, 'opening state');
// every threshold, with the box off and on, by dragging (input event, silent) and by releasing (change event, spoken)
[false, true].forEach((rare) => {
const page = open();
page['sfw-rare'].checked = rare;
for (let pct = 1; pct <= 99; pct++) {
page['sfw-th'].value = String(pct);
page['sfw-live'].textContent = '';
fire(page['sfw-th'], 'input');
look(page, pct, rare, fx.openingPick, 'pct ' + pct + ' rare ' + rare);
want('dragging stays silent at ' + pct, page['sfw-live'].textContent, '');
want('slider reads out', page['sfw-th'].attrs['aria-valuetext'], 'Threshold ' + fx.texts[String(pct)]);
fire(page['sfw-th'], 'change');
want('release speaks at ' + pct, page['sfw-live'].textContent, spoken(fx.views[String(rare)][String(pct)], fx.readouts[String(pct)][fx.openingPick]));
}
});
// ticking the box recomputes at the current threshold and speaks
const ticked = open();
ticked['sfw-rare'].checked = true; fire(ticked['sfw-rare'], 'change');
look(ticked, fx.openingPct, true, fx.openingPick, 'box ticked');
want('box ticked speaks', ticked['sfw-live'].textContent, spoken(fx.views.true[String(fx.openingPct)], fx.readouts[String(fx.openingPct)][fx.openingPick]));
// the two buttons walk the messages in score order and wrap round at either end
const walk = open(), n = fx.messages.length;
fire(walk['sfw-higher']);
look(walk, fx.openingPct, false, fx.openingPick + 1, 'higher score');
fire(walk['sfw-lower']); fire(walk['sfw-lower']);
look(walk, fx.openingPct, false, fx.openingPick - 1, 'lower score');
for (let i = 1; i < fx.openingPick; i++) fire(walk['sfw-lower']); // from one below the opening choice down to the first
look(walk, fx.openingPct, false, 0, 'lowest message');
fire(walk['sfw-lower']);
look(walk, fx.openingPct, false, n - 1, 'lower than the lowest wraps to the highest');
fire(walk['sfw-higher']);
look(walk, fx.openingPct, false, 0, 'higher than the highest wraps to the lowest');
want('a button press speaks', walk['sfw-live'].textContent, spoken(fx.views.false[String(fx.openingPct)], fx.readouts[String(fx.openingPct)][0]));
// tapping the plot picks the nearest mark, at the plot's own size and at another size and place on the page
fx.isolated.forEach((i) => {
[{ left: 0, top: 0, width: 300, height: 214 }, { left: 20, top: 10, width: 600, height: 428 }].forEach((rect) => {
const page = open(), k = rect.width / 300;
page['sfw-plot'].rect = rect;
fire(page['sfw-plot'], 'click', { clientX: rect.left + fx.xs[i] * k, clientY: rect.top + fx.ys[i] * k });
look(page, fx.openingPct, false, i, 'tap on message ' + i + ' at width ' + rect.width);
});
});
const off = open();
fire(off['sfw-plot'], 'click', { clientX: fx.offTap[0], clientY: fx.offTap[1] });
look(off, fx.openingPct, false, fx.offTap[2], 'a tap between the rows picks the nearest mark');
// the host page theme: only the parent window may set it, and only to light or dark
const themed = open();
const send = (data, source) => themed.messages.forEach((f) => f({ data, source }));
send({ type: 'spicyshark:theme', theme: 'dark' }, { other: false });
send({ type: 'spicyshark:theme', theme: 'purple' }, themed.parent);
send({ type: 'other', theme: 'dark' }, themed.parent);
send(null, themed.parent);
want('ignored messages', themed.root.attrs, {});
send({ type: 'spicyshark:theme', theme: 'dark' }, themed.parent);
want('theme from the parent', themed.root.attrs, { 'data-theme': 'dark' });
send({ type: 'spicyshark:theme', theme: 'light' }, themed.parent);
want('theme back to light', themed.root.attrs, { 'data-theme': 'light' });
console.log(JSON.stringify({ updates: 198, bad: bad.slice(0, 5) }));
process.exit(bad.length ? 1 : 0);
"""
def xy(m):
"""Where the widget draws a message, in drawing units."""
return GEO["left"] + m["s"] * GEO["span"], GEO["lanes"][0 if m["h"] else 1] + m["y"]
def nearest(x, y):
return min(range(len(MSGS)), key=lambda i: (xy(MSGS[i])[0] - x) ** 2 + (xy(MSGS[i])[1] - y) ** 2)
def isolated():
"""Messages at least 7 units from every other mark: a tap on one of them can only mean it."""
pts = [xy(m) for m in MSGS]
return [i for i, (x, y) in enumerate(pts) if all(((x - u) ** 2 + (y - v) ** 2) ** 0.5 >= 7 for j, (u, v) in enumerate(pts) if j != i)]
def views():
return {str(rare).lower(): {str(p): sf.view(LABELS, SCORES, p, rare) for p in THRESHOLDS} for rare in (False, True)}
def readouts():
return {str(p): [sf.readout(m["t"], m["h"], m["s"], p) for m in MSGS] for p in THRESHOLDS}
def run_node(tmp_path, script, fixture, args):
(tmp_path / "fixture.json").write_text(json.dumps(fixture), encoding="utf-8")
(tmp_path / "check.mjs").write_text(script, encoding="utf-8")
run = subprocess.run(["node", str(tmp_path / "check.mjs"), *args(tmp_path)], capture_output=True, text=True, timeout=120)
assert run.returncode == 0, run.stdout + run.stderr
return json.loads(run.stdout)
def test_widget_data_block_matches_the_python_model():
"""Guards a change to the data or the filter that was not copied into the widget (run embed_widget.py)."""
assert embed(HTML) == HTML
def test_embed_rewrites_a_widget_whose_data_block_drifted():
"""Guards embed() skipping the block: a hand-edited score or opening threshold must be put back."""
drifted = HTML.replace('"pct": 50', '"pct": 30', 1).replace(f'"s": {MSGS[0]["s"]}', '"s": 0.999', 1)
assert drifted != HTML and embed(drifted) == HTML
assert json.loads(CONFIG_BLOCK.search(HTML).group(2)) == json.loads(json.dumps(CONFIG))
def test_every_element_the_script_looks_up_exists_inside_the_root():
"""Guards a widget that never starts: the parity run covers only the counting, so a misspelt id passed there and threw in the browser."""
script = HTML[HTML.index("<script>"):]
lookups = set(re.findall(r"\$\('([\w-]+)'\)", script))
assert "sfw-th" in lookups and "sfw-marks" in lookups
missing = sorted(i for i in lookups if i != "sfw-config" and f'id="{i}"' not in ROOT)
assert not missing, missing
assert 'id="sfw-config"' in HTML
def test_widget_is_small_and_asks_for_nothing_online():
assert len(HTML.splitlines()) <= 300 and len(HTML.encode("utf-8")) < 40 * 1024
page = CONFIG_BLOCK.sub("", HTML)
for banned in ("http://", "https://", "fetch(", "XMLHttpRequest", "WebSocket", "sendBeacon", "import(", "createElementNS"):
assert banned not in page, banned
assert " src=" not in page and "@import" not in page
def test_widget_has_no_form_element():
"""Guards a dead button: the lesson page loads widget.html in a sandboxed frame, where a form cannot submit."""
assert "<form" not in HTML and 'type="submit"' not in HTML
def test_widget_shows_at_most_five_controls_and_hides_the_rest():
"""Guards the first screen at 390 px: the slider and two buttons show, and the checkbox stays in a closed details element."""
outside = re.sub(r"<details>.*?</details>", "", ROOT, flags=re.S)
assert len(re.findall(r"<(?:select|button|input|textarea)\b", outside)) <= 5
assert "<details><summary>More settings</summary>" in ROOT and "<details open" not in ROOT
assert re.search(r"<details><summary>More settings</summary>.*?type=\"checkbox\"", ROOT, re.S)
def test_widget_keeps_the_site_theme_listener_and_palette():
assert "d.type !== 'spicyshark:theme'" in HTML and "e.source !== parent" in HTML and "parent === window" in HTML
for pair in ("--bg: light-dark(#ffffff, #0a0f17)", "--panel: light-dark(#ebf0f6, #131b28)", "--ink: light-dark(#0a1626, #f4f7fb)",
"--mute: light-dark(#44566b, #a3b1c4)", "--ctl: light-dark(#66768a, #7a8a9f)", "--ink-on: light-dark(#f6f8fb, #05070c)",
"--green: light-dark(#00855F, #009E73)", "--orange: light-dark(#b36200, #E69F00)", ":root[data-theme=light]", ":root[data-theme=dark]"):
assert pair in HTML, pair
for banned in ("#0072B2", "#56B4E9", "#D55E00"):
assert banned.lower() not in HTML.lower()
assert not set(re.findall(r"#[0-9a-fA-F]{3,8}\b", HTML)) - {"#ffffff", "#0a0f17", "#ebf0f6", "#131b28", "#0a1626", "#f4f7fb", "#44566b", "#a3b1c4",
"#66768a", "#7a8a9f", "#f6f8fb", "#05070c", "#00855F", "#009E73", "#b36200", "#E69F00"}
def test_widget_scores_are_the_filters_scores_and_the_marks_do_not_hide_a_row():
"""Guards the plot: every test message is in the data once, scores are what safety_filter.py computed, and no mark leaves its row."""
r = sf.build()
assert sorted(zip(SCORES, LABELS)) == sorted(zip(r["scores"], r["labels"]))
assert len(MSGS) == 120 and sum(LABELS) == 40 and SCORES == sorted(SCORES)
assert all(abs(m["y"]) <= GEO["half"] for m in MSGS) and 0 <= min(SCORES) and max(SCORES) <= 1
assert len(isolated()) >= 8 # enough separate marks to test tapping
def test_widget_text_keeps_the_caveat_and_the_numbers_it_names():
"""Guards the words that keep a toy from reading as a measurement: they are visible without opening anything, and the
More settings note names the traffic the code computes."""
outside = re.sub(r"<details>.*?</details>", "", ROOT, flags=re.S)
assert "A toy: made-up messages that ask for or share contact details. Real filters score many kinds of harm, each against its own threshold." in outside
assert "Square = harmful, circle = harmless, ring = a mistake, bold outline = the one you picked." in outside
assert 'id="sfw-pickl">Read another message:</p>' in outside and 'role="group" aria-labelledby="sfw-pickl"' in outside # the two buttons are named on screen
assert 'class="sfw-sr" id="sfw-live" role="status" aria-live="polite" aria-atomic="true"' in ROOT # what a screen reader hears
assert ".sfw-root .sfw-btns button { flex: 1 1 auto; white-space: nowrap; }" in HTML # a button wraps to the next row, never its label
assert "paint-order: stroke" in HTML and HTML.count('class="sfw-tag"') == 2 # the row labels keep a halo, so the threshold line never cuts them
more = re.search(r"<details><summary>More settings</summary>(.*?)</details>", ROOT, re.S).group(1)
assert f"{sf.TRAFFIC:,} messages, {sf.TRAFFIC_HARMFUL} of them harmful" in more and "1 in 100 messages is harmful" in more
assert "They are expected counts, not measured ones." in more and f"{LABELS.count(0)} harmless messages blocked is not a rate of 0 in real traffic" in more
assert sf.TRAFFIC == 100 * sf.TRAFFIC_HARMFUL
def test_widget_opens_on_a_mistake_the_lesson_names():
"""Guards the first screen: at the opening threshold the chosen message is the harmless one the filter blocks, and the
sentence already shows the lesson's opening counts."""
opening = MSGS[OPENING]
assert CONFIG["opening"]["pct"] == sf.OPENING_PCT == 50 and opening["t"].endswith(sf.FALSE_ALARM_EXAMPLE) and opening["h"] == 0
assert sf.readout(opening["t"], opening["h"], opening["s"], 50).endswith("so it is blocked: a false alarm.")
assert sf.view(LABELS, SCORES, 50)["says"] == "At 0.50 the filter misses 5 harmful messages and blocks 1 harmless message."
rare = sf.view(LABELS, SCORES, 50, True) # expected counts, so the sentences say "about"
assert rare["says"] == "Out of 10,000 messages, 100 harmful: at 0.50 the filter would miss about 12 and block about 124 harmless messages."
assert rare["precision"] == "Of the 212 or so messages it would block, about 88 are harmful (42%)."
def test_widget_counting_gives_the_numbers_and_words_of_the_python_model(tmp_path):
"""Guards the widget's JavaScript drifting from safety_filter.py: the lesson prints Python's numbers and the reader sees the
widget's, so every sentence, count, percentage and readout must agree at all 99 thresholds, in both traffic mixes."""
fixture = {"views": views(), "readouts": readouts(), "commas": [[n, f"{n:,}"] for n in COMMAS], "shares": [[p, w, sf.share(p, w)] for p, w in SHARES]}
result = run_node(tmp_path, PARITY, fixture, lambda d: [str(d / "fixture.json"), str(WIDGET)])
assert result["bad"] == [] and result["checks"] == 2 * 99 + 99 * 120 + len(COMMAS) + len(SHARES)
exact = [i for i, s in enumerate(SCORES) if round(s * 100) == s * 100 and 1 <= round(s * 100) <= 99]
assert exact # a score equal to a threshold is blocked, in both languages
def test_widget_page_shows_what_the_lesson_names(tmp_path):
"""Guards the part a reader sees: the opening state, every threshold with the box off and on, what is spoken on release, the
two buttons and their wrap-around, a tap on a mark, and the theme message."""
ids = sorted(set(re.findall(r'id="(sfw-[\w-]+)"', HTML)))
taps = isolated()
x, y = 150.0, 104.0
fixture = {"ids": ids, "messages": MSGS, "views": views(), "readouts": readouts(), "openingPct": sf.OPENING_PCT, "openingPick": OPENING,
"texts": {str(p): f"0.{p:02d}" for p in THRESHOLDS}, "harmful": sum(LABELS), "harmless": len(LABELS) - sum(LABELS),
"mistakes": {str(p): sum(sf.counts(LABELS, SCORES, p)[1:3]) for p in THRESHOLDS},
"xs": [xy(m)[0] for m in MSGS], "ys": [xy(m)[1] for m in MSGS], "isolated": taps, "offTap": [x, y, nearest(x, y)]}
result = run_node(tmp_path, PAGE, fixture, lambda d: [str(WIDGET), str(d / "fixture.json")])
assert result == {"updates": 198, "bad": []}