---
title: "How does a safety filter decide, and where does it fail?"
description: "A safety filter scores each message and blocks it at a threshold. Every threshold misses some harm or blocks some harmless messages, and rare harm changes the mix."
topic: "AI systems"
chapter: "trust-the-answers"
order: 15
minutes: { plain: 6, deep: 15 }
prerequisites: ["Reading short Python functions", "Percentages and simple fractions"]
seo_title: "How does an AI safety filter decide what to block?"
date: 2026-10-19
---
# How does a safety filter decide, and where does it fail?
A safety filter reads a message, gives it a score, and blocks it when the score reaches a line called the threshold. In a toy built for this lesson, which looks for messages that ask for or share contact details in a school chat, a threshold of 0.50 misses 5 of 40 harmful test messages and blocks 1 of 80 harmless ones. If only 1 message in 100 is harmful, the same filter would block about 212 of every 10,000 messages, and about 124 of those would be harmless.
## In one minute
A safety filter gives each message a score and blocks it when the score reaches a threshold, a line you choose. No threshold is right for every message, because each one trades missed harmful messages against blocked harmless ones. Think of an airport scanner with a sensitivity dial. Turn its sensitivity up (a lower threshold) and it beeps at belt buckles. Turn it down (a higher threshold) and it waves through things it should stop. How many of each you see depends on how rare the harmful ones are: when nearly every bag is fine, most of the beeps are for fine bags.
## Explore the numbers
[Open the widget](https://spicyshark.co/learn/how-a-safety-filter-decides/widget.html). Drag the threshold and watch the squares (harmful messages) and circles (harmless ones) move between passed and blocked, and tap a mark to read its message. Then open More settings and tick the box for 1 harmful message in 100.
## How it works
### A score and a threshold
A safety filter is a program that reads a message and gives it a score from 0 to 1: how strongly the message looks like the harm it was built to find. A real filter covers many kinds of harm, with one score for each. OpenAI's moderation guide (read 2026-10-19) describes "the model's confidence in each category", and Microsoft's documentation for Azure AI Content Safety (read 2026-10-19) has you choose a blocking level for each category.
The toy checks one kind: a message that asks for or shares contact details, such as a phone number or where someone lives, in a school or club chat. It learned from 300 made-up messages and is tested on 120 others. It reads each message's words and the letter pairs inside them, such as "ad" in "address".
The app sets the threshold. A message that scores at or above it is blocked, and one below it passes.
### Four things can happen
A message is harmful or harmless, and the filter blocks it or passes it, so there are four outcomes. Blocking a harmful message is a catch, and passing it is a miss. Passing a harmless message is correct, and blocking it is a false alarm. Misses and false alarms are the two mistakes.
The miss rate is the share of harmful messages that are missed. The false-alarm rate is the share of harmless messages that are blocked. At 0.50 the toy misses 5 of 40 harmful test messages (12.5%) and raises 1 false alarm among 80 harmless ones (1.25%).
### Moving the threshold
Lower the threshold and more messages reach it, so more are blocked: fewer misses, more false alarms. Raise it and the reverse happens. Google's machine learning course (thresholds page, read 2026-10-19) describes the same trade.
In the toy, at 0.01 no harmful message is missed but 42 of 80 harmless ones are blocked. At 0.99 no harmless message is blocked but 26 of 40 harmful ones are missed. No threshold gives zero of both, because harmless and harmful scores overlap in the middle.
### When harm is rare
The rates above come from a test set where 1 message in 3 is harmful. Real chats are not like that. The base rate is how common the harmful kind is in your traffic.
Suppose 1 message in 100 is harmful and the filter keeps the same two rates. Of 10,000 messages, 100 are harmful and 9,900 are not. At 0.50 the filter blocks about 88 harmful messages and about 124 harmless ones. So 58% of what it blocks is harmless, although its false-alarm rate is a low 1.25%. A small share of a big pile is more messages than a large share of a small one.
Google's course makes a related point: if a class shows up 1% of the time, a model that never flags it is 99% accurate and useless.
### Where it fails
The toy learned which words and letter pairs go with harmful messages in training. A word it never saw has no weight. So "hi drop ur digits", a request for a phone number in slang, scores 0.444 and passes at 0.50. "hi my phone battery died" scores 0.650 and is blocked, because "phone" appeared only in harmful training messages. All 6 mistakes at 0.50 are in wordings the toy never saw in training, meaning ways of writing a message that can still use familiar words. None of the 68 test messages in wordings it did see is a mistake.
A filter built on a large model also makes both mistakes. Microsoft's documentation says a higher blocking level "doesn't guarantee that it blocks all harmful content" and tells you to review "unwanted blocks" too.
## What this means when you build
- **Set the threshold from what each mistake costs.** Microsoft's documentation says "No single blocking level works for every workload" and has you choose one per kind of harm.
- **Measure both rates on messages like your own, at your own base rate.** A test set that is a third harmful misleads you. Include harmless messages that look harmful.
- **If you only use a filter, ask the provider** its threshold, and how many harmful messages it misses and harmless ones it blocks at your base rate.
- **Let people appeal a block and report a miss.** Both mistakes will happen, and each teaches you something.
- **Log blocks and read a sample.** Re-check when your users, their wording or the filter changes.
- **Treat the filter as one layer.** A stranger can reword. Put checks in code beside it, as in [the prompt injection lesson](https://spicyshark.co/learn/prompt-injection/).
## Go deeper
*Optional, for readers who write Python. The code is in the `code/` folder: `safety_filter.py` is the filter and its arithmetic, `make_figures.py` draws three of the figures, and `test_safety_filter.py` and `test_widget.py` are the tests. `python3 show_numbers.py` prints the measured numbers above. It leaves out the numbers in quoted sources and the made-up ones in the questions.*
### The messages
The toy has no real data. `safety_filter.py` builds every message from templates. Each `{word}` in braces is filled from a short list, picked by a seeded random generator, so each run gives the same messages and the same numbers. Harmful means one thing only: asking for or sharing contact details in a school or club chat. Harmless messages share words with harmful ones on purpose, such as "the home team plays on friday" and "my number on the team is 12", so the filter cannot win by spotting one word.
```python
HARMFUL = [
"what's your {thing}?", "can i get your {thing}?", "send me your {thing}", "where do you live?",
"which street do you live on?", "my phone number is {digits}", "i live at {num} {street} street",
"call me on {digits}", "text me at {digits}", "my home address is {num} {street} street",
"where does {who} live?", "what's {poss} {thing}?",
]
```
There are 300 training messages (100 harmful and 200 harmless) and 120 test messages (40 harmful and 80 harmless), and no message is in both sets. Of the test messages, 68 use wordings that training also had. The other 52 (16 harmful and 36 harmless) use wordings it never had, from a second list: slang such as "ur digits", and harmless messages that reuse harmful words, such as "my phone battery died".
```python
UNSEEN_HARMFUL = [
"{do} ur digits", "whats ur {adr}", "where do u stay", "which street r u on", "{do} me ur number",
"what area do u stay in?", "ur place is on which street?", "{do} me ur digits", "what street do u stay on",
]
```
A filter meets this whenever people change how they write, so the test measures it.
### Features and the training loop
The filter turns each message into features, the things it can count. Here a feature is a word, or a pair of letters inside a word, with `^` and `$` marking the start and end of the word.
```python
def features(text: str) -> set[str]:
"""Each word ("w:address") and each pair of letters inside a word, with ^ and $ marking its ends ("c:ad")."""
feats = set()
for word in re.findall(r"[a-z0-9]+", text.lower()):
feats.add("w:" + word)
padded = "^" + word + "$"
feats.update("c:" + padded[i:i + 2] for i in range(len(padded) - 1))
return feats
```
A message becomes a row of 491 numbers, one per feature seen in training: 1 if the feature is in the message and 0 if not. A feature never seen in training has no column, so it has no weight. The score is the sigmoid (a curve that squeezes any number into the range 0 to 1) of the sum of the weights of the features present plus a bias (a number added every time).
Training starts with every weight at 0, so every score is 0.5 and the loss (a measure of how wrong the scores are; here log-loss) is 0.6931, which is ln 2, the loss of a coin flip. Each of the 300 steps of gradient descent (nudging every weight a little in the direction that lowers the loss) reduces it, and the loss reaches 0.0160.
```python
def train_filter(X, y, steps=300, rate=0.5, l2=0.003):
"""Gradient descent on the average log-loss plus a small penalty on the weights. Returns weights, bias, loss per step."""
w, b, losses = np.zeros(X.shape[1]), 0.0, []
for _ in range(steps):
p = sigmoid(X @ w + b)
losses.append(float(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))))
err = p - y
w -= rate * (X.T @ err / len(y) + l2 * w)
b -= rate * err.mean()
return w, b, losses
```
`l2` is a small penalty on large weights, so no single feature takes over. This is logistic regression. A filter built on a large model has far more features, but it usually ends the same way: a score compared with a threshold.
Words that appear mostly in harmful training messages get a positive weight, and words that appear mostly in harmless ones a negative weight. "phone" and "address" each appear in 26 harmful training messages and in no harmless one. "the" appears in 166 of the 200 harmless training messages and in none of the harmful ones.
| Word | Weight |
|---|---|
| 555 | 0.51 |
| live | 0.50 |
| phone | 0.45 |
| address | 0.42 |
| street | 0.41 |
| home | 0.39 |
| me | 0.38 |
| number | 0.28 |
| is | -0.28 |
| great | -0.32 |
| the | -1.01 |
### Counting at a threshold
```python
def counts(labels, score_list, pct):
"""Block a message when its score is at least pct/100. Returns (caught, missed, falsely blocked, passed)."""
t = pct / 100
caught = sum(1 for y, s in zip(labels, score_list) if y == 1 and s >= t)
blocked_harmless = sum(1 for y, s in zip(labels, score_list) if y == 0 and s >= t)
harmful = sum(labels)
return caught, harmful - caught, blocked_harmless, len(labels) - harmful - blocked_harmless
```
A score at or above the threshold is blocked (a system may block only above it), and a test checks that a score exactly at the threshold is blocked. `sweep` calls `counts` at every threshold from 0.01 to 0.99. This table shows some of them on the 120 test messages. Missed counts harmful messages that pass, and Alarms (false alarms) counts harmless messages blocked. Caught is 40 minus missed, and passed is 80 minus alarms.
| Threshold | Missed | Alarms |
|---|---|---|
| 0.01 | 0 | 42 |
| 0.10 | 0 | 28 |
| 0.25 | 0 | 10 |
| 0.30 | 3 | 6 |
| 0.40 | 4 | 2 |
| 0.50 | 5 | 1 |
| 0.60 | 6 | 1 |
| 0.70 | 10 | 0 |
| 0.90 | 14 | 0 |
| 0.99 | 26 | 0 |
The fewest mistakes in all (misses plus false alarms) is 6, reached at 18 thresholds from 0.38 to 0.56. No threshold has both at zero. The miss rate first reaches the false-alarm rate at 0.30, where each is 7.5% (3 of 40 and 6 of 80). Picking 0.50 is a choice. If a miss costs far more than a false alarm, 0.25 misses none of the 40 harmful test messages and blocks 10 of 80 harmless ones.
### Rare harm, by hand
The widget's box for 1 harmful message in 100 applies the miss rate and false-alarm rate measured on the test messages to 10,000 messages of which 100 are harmful. By hand at 0.50:
1. Miss rate: 5 missed of 40 harmful is 12.5%.
2. False-alarm rate: 1 blocked of 80 harmless is 1.25%.
3. Harmful messages caught: 100 x 87.5% = 87.5, rounded up to 88. The other 12 are missed.
4. Harmless messages blocked: 9,900 x 1.25% = 123.75, rounded up to 124.
5. Blocked in all: 88 + 124 = 212. The harmful share is 88 / 212 = 41.5%, which rounds to 42%.
```python
def at_traffic(c, total=TRAFFIC, harmful=TRAFFIC_HARMFUL):
"""The same miss and false-alarm rates applied to `total` messages of which `harmful` are harmful.
Whole numbers, rounding halves up, so Python and the widget agree exactly.
"""
caught, missed, false_alarm, passed = c
n_harmful, n_harmless = caught + missed, false_alarm + passed
caught_t = (caught * harmful + n_harmful // 2) // n_harmful
false_t = (false_alarm * (total - harmful) + n_harmless // 2) // n_harmless
return caught_t, harmful - caught_t, false_t, total - harmful - false_t
```
It uses whole-number arithmetic and rounds halves up, so the widget's JavaScript gives the same counts as Python. The same rates at other thresholds, for 10,000 messages of which 100 are harmful. Alarms (false alarms) are harmless messages blocked, and Share is the percentage of all blocked messages that are harmful:
| Threshold | Alarms | Share |
|---|---|---|
| 0.30 | 743 | 11% |
| 0.40 | 248 | 27% |
| 0.50 | 124 | 42% |
| 0.60 | 124 | 41% |
| 0.66 | 0 | 100% |
| 0.70 | 0 | 100% |
| 0.90 | 0 | 100% |
The share of blocked messages that are harmful is at least half from 0.66, where the test messages have no false alarm left, and by then about 22 of the 100 harmful messages are missed. Treat the 124 with care. The 1 false alarm is 1 of only 80 harmless test messages, so one more or one fewer would move the 10,000-message count by about 124, and 0 false alarms in 80 is not a rate of 0. Google's course warns that when there are few actual negatives, one more mistake moves the false positive rate a lot. A real test needs far more messages.
### Two failures, and why
The sum is what the filter computes before the sigmoid squeezes it into 0 to 1. It is the bias plus the weights of the features present. Figures are rounded, so the parts add to the sum only within 0.002.
"hi drop ur digits" is harmful. The words "drop", "ur" and "digits" were never in training, so they have no weight.
| Part | Adds |
|---|---|
| Bias | -0.082 |
| Word "hi" | -0.108 |
| Letter pairs (14) | -0.035 |
| Sum | -0.225 |
| Score | 0.444 |
The score is below 0.50, so the message passes: a miss. "hi my phone battery died" is harmless. The words "battery" and "died" have no weight, but "phone" does.
| Part | Adds |
|---|---|
| Bias | -0.082 |
| Word "hi" | -0.108 |
| Word "my" | -0.056 |
| Word "phone" | 0.446 |
| Letter pairs (24) | 0.420 |
| Sum | 0.619 |
| Score | 0.650 |
The score is above 0.50, so the message is blocked: a false alarm. In the first message the unweighted words add nothing and the rest adds up to less than 0. In the second, "phone" carries +0.446, and letter pairs such as "ph" and "ho" carry positive weights and push the letter-pair total to +0.420. A third message uses a harmful word and is still passed correctly: "ok so what's the library's address?" has "address" at +0.42, but "the" at -1.01 and its other features bring the sum to -0.976 and the score to 0.274.
### The tests
```bash
cd lessons/how-a-safety-filter-decides/code
python3 -B -m pytest -q -p no:cacheprovider
```
Each test guards a mistake a reader would see. The data test checks the set sizes and that no test message is a training message. The training test checks that the loss starts at ln 2 and never rises. The counting tests check that the four counts add up to 120 at every threshold, that raising the threshold only ever moves messages from blocked to passed, and that no threshold has both errors at zero. The arithmetic test compares the whole-number rare-harm counts with exact fractions at every threshold. Further tests check the failure examples above, that every Python block in this lesson is a verbatim excerpt of `safety_filter.py`, that the lesson's text quotes the numbers the code gives (the set sizes, rates, example scores, weights, word counts, by-hand sums, sweep text, both threshold tables and the Check-yourself answers), that the three data figures equal what `make_figures.py` draws, and that the numbers in the other two figures equal the counts at 0.50. A second file, `test_widget.py`, runs the widget's JavaScript in Node.js (a way to run JavaScript outside a browser) for all 99 thresholds with and without the 1-in-100 box, checks its sentences and counts equal the Python ones, and checks it asks for nothing online and holds no form.
### What the toy leaves out
- **The messages are made up and short.** Real messages are longer, messier and in many languages.
- **One kind of harm.** A real filter scores many kinds, each with its own threshold and its own cost of a mistake.
- **A score is not a chance.** A score of 0.65 does not mean a 65% chance the message is harmful. Treat a score as a way to put messages in order.
- **The test set is small.** 120 messages, with 80 harmless, cannot tell a false-alarm rate of 1% from 2%.
- **The toy's 0.50 is a round default, not tuned.** Picking a threshold on the messages you then report on would flatter a filter, so a real project picks it on one set of messages and reports on another. Here 0.50 sits inside the range, 0.38 to 0.56, where the test set has its fewest mistakes (6).
- **The base-rate counts assume the rates carry over.** If real traffic has more unseen wordings, both rates change. The toy's test set has many unseen wordings on purpose (52 of 120), to show the failures, so its rates may be higher than a real day's.
### Where this is used in practice
OpenAI's moderation guide (read 2026-10-19) says a response carries a score for each category, and a `flagged` field. Microsoft's documentation for Azure AI Content Safety (read 2026-10-19) has you choose a blocking level for each category, and advises evaluating each candidate setting on representative content, including examples to allow and examples to block, then reviewing both unwanted blocks and harmful content that passes. That is the build advice above, in the vendor's words.
Llama Guard (Inan and colleagues, 2023, read 2026-10-19) is a safeguard model built on a large language model, which classifies the prompts people send and the responses a model gives. Its paper reports the area under the precision-recall curve and says this is "useful for selecting the classification threshold that balances precision and recall based on the specific requirements of use cases". Precision is the share of blocked messages that are harmful, the 42% above, and recall is the share of harmful messages caught, which is 1 minus the miss rate. The scikit-learn documentation (read 2026-10-19) shows the same trade on a plot: lowering the threshold may raise recall but add false positives and usually lower precision.
## Skills this lesson builds
- Reading a classifier's score and threshold, and counting its four outcomes at any threshold.
- Computing a miss rate, a false-alarm rate and precision, and explaining what each hides.
- Working out what a filter does at a given base rate before you ship it, with a worked example by hand.
- Choosing a threshold from the cost of each mistake, and testing it on messages like your own traffic.
- Writing a test that finds the messages a filter gets wrong, in wordings it never saw.
**Portfolio project:** train a small text classifier for one narrow kind of message on made-up data, hold back test messages in wordings it never saw, and publish a threshold sweep plus a table of what its counts become at 1 harmful message in 100.
## Check yourself
1. A filter scores a message 0.62 and the threshold is 0.60. What happens, and what happens if you raise the threshold to 0.70?
Answer
At 0.60 the message is blocked, because its score is at or above the threshold. At 0.70 it passes. If it is harmless, raising the threshold removes a false alarm. If it is harmful, it turns a catch into a miss.
Answer
There are 99 harmless messages for every harmful one. 1.25% of 9,900 harmless messages is about 124, more than the roughly 88 harmful messages caught, so about 124 of the roughly 212 blocked messages (58%) are harmless.
Answer
The filter only has weights for features it saw in training. A new slang wording such as "drop ur digits" has none, so it scores low and passes. A harmless message that reuses a word like "phone", which appeared only in harmful training messages, scores high and is blocked.
Answer
A miss and a false alarm do not cost the same. If missing a harmful message costs much more, a lower threshold is better even though it makes more mistakes in all. Also, the 6 counts both kinds equally on a test set that is 1 harmful in 3, and at 1 in 100 the balance of mistakes is different.
Answer
It comes from 1 false alarm among only 80 harmless test messages. One more or one fewer would move it by about 124, and 0 in 80 would not be a rate of 0. A real test needs many more harmless messages from your own traffic.