---
title: "Can a model's refusals be removed?"
description: "In a toy network one direction carries every refusal. Taking it out ends them and barely moves other answers, so an app must not rest on refusals alone."
topic: "AI systems"
chapter: "safety-inside-the-model"
order: 32
minutes: { plain: 6, deep: 17 }
prerequisites: ["Reading short Python functions", "Percentages and simple fractions", "Reading Python and NumPy"]
seo_title: "Can an AI model's refusals be removed? A toy you can run"
date: 2026-10-20
---
# Can a model's refusals be removed?
Many chat models say no to some requests, and that no may sit in one place. In a toy network of 100 lines of Python, one direction in its numbers carries the refusal. All 100 flagged requests, invented stand-ins for ones a model should decline, are refused. Take that direction out and none is, while 99 of 100 plain answers stay the same. A published study of 13 open chat models, up to 72 billion parameters, reports refusal mediated by a single direction across all of them. A model's refusals are a thin layer, and an app that rests its safety on them alone rests on something that can be taken away.
## In one minute
Picture a map where every request is a dot. Plain dots sit near the middle. Flagged dots sit far to the north. The model's refusal is one small habit: when a dot is far enough north, say no. Find the north arrow by averaging the flagged dots, averaging the plain dots and subtracting. Then flatten the map along that arrow: every dot's north-south position becomes 0, and its east-west position stays. The habit never fires again, so nothing is refused, and what the model knows from east and west is untouched.
The toy's map has 32 directions, not 2, and a real model's has many more. Two cautions. The toy has a refusal planted along one direction, so that we can check the arrow it finds; nobody plants one in a real model. And one arrow is not always the whole story, which the numbers below show.
## Explore the numbers
[Open the widget](https://spicyshark.co/learn/can-a-models-refusals-be-removed/widget.html). It opens on the toy's default: all of the direction taken out, read after layer 5 of 6. The sentence on top says 0 of 100 flagged requests are refused, down from 100, and 1 of 100 plain answers changes. Drag Share taken out to 20%: 44 flagged requests are still refused, so the refusals fall in a narrow band, between 10% and 30%. Drag Read after layer to 0: the direction is found 37.9 degrees off the planted one, 13 flagged requests are still refused and 8 plain answers change. Under More settings, Second route set to 30% leaves 29 refused, though all of the direction is out.
## How it works
### The picture, step by step
The network passes a vector of 32 numbers through 6 layers, and each layer adds its change to it. In every layer a refusal unit switches on when the vector holds enough of one planted direction, and adds more of it. The last step reads that part as "refuse". Here are the six steps, as Figure 1 shows the result of steps 2 to 4. A flagged request is a made-up vector with a push along the planted direction; a plain one has none.
*Figure 1. Held-out requests along the found direction, before and after it is taken out.*
1. **Read the vectors.** Run 64 flagged and 64 plain requests through the network and keep each one's 32 numbers after layer 5.
2. **Subtract the averages.** The average flagged vector minus the average plain vector, scaled to length 1, is the refusal direction. It lands 8.4 degrees from the planted one.
3. **Check it on new requests.** Along it, 100 flagged requests the step never saw sit at 8.1 on average and 100 plain ones at -0.1. One cut at 4.0 sorts all 200.
4. **Take it out at every layer.** Subtract each vector's part along it, 7 times: once before the first layer and after each of the 6. This is directional ablation.
5. **Write the same edit into the weights.** Cut that part from every matrix that writes into the vector, once (orthogonalization). The answers do not change.
6. **Count.** Flagged requests refused: 100 before, 0 after. Plain requests whose answer changes: 1 of 100.
### Each step in code
One real line of the program does each step. The files are in `code/`, and the line numbers are those of this lesson's files.
| Step | The code line | What it does |
|---|---|---|
| 1. Keep the vectors | `refusal_toy.py:45`
`points.append(x)` | Each layer's vector is kept, so the direction can be read at any layer. |
| 2. Subtract the averages | `refusal_toy.py:51`
`d = flagged_points.mean(0) - plain_points.mean(0)` | The next line scales it to length 1. |
| 3. Check along it | `trace.py:14`
`flagged_along = base["test_flagged"][1][LAYER] @ u` | One number per held-out request: how far along the direction it sits. |
| 4. Take it out | `refusal_toy.py:57`
`return lambda x: x - share * np.outer(x @ u, u)` | Subtract the part along u from each vector. |
| 5. Cut it from the weights | `refusal_toy.py:62`
`cut = lambda W: W - np.outer(u, u @ W)` | The same subtraction on a matrix: each column loses its part along u. |
| 6. Count refusals | `refusal_toy.py:87`
`refused = lambda z: int((z.argmax(1) == 0).sum())` | A request is refused when answer 0 scores highest. |
### A real run
`trace.py` runs the six steps and prints one line for each, then one line for the limit. Run it from the repository root:
```bash
cd lessons/can-a-models-refusals-be-removed/code && python3 trace.py
```
Phone readers: slide the box sideways to see every column.
```text
step 1 read the vectors after layer 5: 64 flagged and 64 plain requests, 32 numbers each
step 2 difference of the two means, length 1: 8.4 degrees from the planted direction
step 3 held-out distance along it: flagged 8.1, plain -0.1; one cut at 4.0 sorts 200 of 200 right
step 4 taken out at all 7 points: last-layer length of flagged vectors 13.3 -> 5.6, plain 5.7 -> 5.7
step 5 same edit in the weights: largest difference in the answer scores 0.0000000000
step 6 flagged requests refused: 100 of 100 before, 0 of 100 after; plain answers changed: 1 of 100 (mean KL 0.006)
limit with 30 of 100 flagged requests on a second route, the same edit leaves 29 of 100 refused
```
Read it against the steps.
- **Steps 1 and 2.** The direction is found from 64 and 64 requests and lands 8.4 degrees from the planted one. The toy can say so only because it planted that direction. A real model offers no such check, so step 3, on requests the direction was not found from, is the test that matters.
- **Step 3.** Flagged requests sit at 8.1 along the direction and plain ones at -0.1, so one cut sorts all 200. Figure 1 draws 30 of each. After step 4 every dot sits at 0 along the direction and keeps its other positions.
- **Step 4.** The last-layer length of flagged vectors falls from 13.3 to 5.6 and plain ones stay at 5.7. What was taken out was most of what made flagged vectors differ.
- **Step 5.** The weight edit gives the same answer scores as the hook, to the last of ten decimals. The edited network needs no hook, and nothing in the code that runs it shows the change.
- **Step 6.** All 100 flagged requests were refused before, and 0 are after. One plain answer in 100 changes, with a mean KL of 0.006. KL measures how far the answer odds moved, and 0 means not at all. Figure 2 shows how much must come out: nothing happens at 10%, and by 30% no flagged request is refused. That sharp fall comes from the toy's own switch (Go deeper).
*Figure 2. Refusals fall between 10% and 30% taken out, and plain answers hardly move.*
- **The limit.** Put 30 of the 100 flagged requests on a second route, and the same edit leaves 29 refused (Figure 3). The direction found then sits 25.0 degrees off the planted one. One direction removes the refusals that travel along it and few others.
*Figure 3. One direction does not remove a refusal that travels along two.*
## Your turn
The direction was read after layer 5. See what reading it before the first layer does.
1. In `code/trace.py`, change line 6 to `LAYER = 0`.
2. Run `python3 trace.py` again in the `code` folder.
Look at steps 2 and 6. How far from the planted direction is the direction now, and how many flagged requests are still refused?
What you should see
Every line but step 5 changes:
```text
step 1 read the vectors after layer 0: 64 flagged and 64 plain requests, 32 numbers each
step 2 difference of the two means, length 1: 37.9 degrees from the planted direction
step 3 held-out distance along it: flagged 1.2, plain -0.0; one cut at 0.6 sorts 166 of 200 right
step 4 taken out at all 7 points: last-layer length of flagged vectors 13.3 -> 5.9, plain 5.7 -> 5.7
step 5 same edit in the weights: largest difference in the answer scores 0.0000000000
step 6 flagged requests refused: 100 of 100 before, 13 of 100 after; plain answers changed: 8 of 100 (mean KL 0.060)
limit with 30 of 100 flagged requests on a second route, the same edit leaves 49 of 100 refused
```
The direction is 37.9 degrees off, 13 flagged requests are still refused, and 8 plain answers change, where reading after layer 5 gave 8.4 degrees, 0 and 1. Before the first layer, flagged and plain requests differ only by the push that was planted, so 64 and 64 examples pin the direction down poorly. By layer 5 the sets are far apart along it, and the same examples find it well (Figure 4).
## What this means when you build
- **Treat a model's refusals as a setting, not a guarantee.** In the toy, one edit takes all of them away. In the study of real models, one edit to the weights did the same to many. A model whose files someone else can change can lose its refusals.
- **Keep your own checks outside the model.** A filter in your code, on what goes in and what comes out, works whatever the model does ([lesson 15](https://spicyshark.co/learn/how-a-safety-filter-decides/)). The model's refusals are one layer.
- **Test the model you serve.** Run your own test set ([lesson 13](https://spicyshark.co/learn/check-ai-answers-with-a-test-set/)) on the weights you run, with requests that should be declined, and check who published the file ([lesson 17](https://spicyshark.co/learn/is-it-safe-to-download-a-model-file/)). A refusal that held in someone's demo says little about a model that another party changed.
- **Do not take good ordinary answers as proof.** The edit left 99 of 100 plain answers the same while all 100 refusals went. A model that answers well still may have lost its refusals.
- **Read the toy as a mechanism, not a measurement.** It shows how a direction can carry a refusal, how it is found and why one edit is not always enough. It says nothing about how many refusals any real model keeps.
## Go deeper
*Optional. This part is for readers who write Python. The code is in the `code/` folder next to this lesson. From inside that folder, `python3 show_numbers.py` prints the numbers below, `python3 trace.py` prints the run above, and `python3 -B -m pytest -q -p no:cacheprovider` runs the tests.*
`refusal_toy.py` is 100 lines, with its comments and blank lines: a network, a way to make requests, the three edits of the lesson and a way to count. Every number in it is made up inside the program from a fixed seed. It reads no file, opens no connection and loads no model, and nothing in it could be pointed at a real one.
### The toy network
```python
def build(seed=SEED):
"""The network: an embedding E, six blocks (W_in, bias, W_out) and a readout U."""
rng = np.random.default_rng(seed)
E, _ = np.linalg.qr(rng.normal(size=(D, D)))
routes = E[:, :ROUTES].T # the planted directions, one unit vector each
blocks = []
for _ in range(LAYERS):
W_in = np.vstack([routes, rng.normal(size=(HIDDEN - ROUTES, D)) / np.sqrt(D)])
W_out = np.hstack([GAIN * routes.T, 0.15 * rng.normal(size=(D, HIDDEN - ROUTES)) / np.sqrt(HIDDEN)])
bias = np.r_[np.full(ROUTES, TAU), np.zeros(HIDDEN - ROUTES)]
blocks.append((W_in, bias, W_out))
U = np.hstack([READ * routes.sum(0)[:, None], 2 / np.sqrt(D) * rng.normal(size=(D, ANSWERS - 1))])
return {"E": E, "routes": routes, "blocks": blocks, "U": U}
```
`E` is a random rotation, and its first two rows are the two planted directions, `routes`. Each layer reads the vector with `W_in`, keeps what is above a bias, and writes back with `W_out`. The first two of its 24 hidden units are the refusal units. They read along a planted direction, switch on only above `TAU = 0.5`, and write `GAIN = 0.5` times the excess back along the same direction. The other 22 are random and write small changes everywhere. The readout `U` scores answer 0, "refuse", by `READ = 2` times the part along the planted directions, and the other 7 answers with random numbers.
```python
def requests(n, flagged, rng, noise=1.0, second=0.0):
"""n requests as feature vectors. A flagged one carries a push along route 0, or along route 1 for the first `second` share."""
f = rng.normal(size=(n, D)) * noise
f[:, :ROUTES] *= QUIET
if flagged:
f[np.arange(n), (np.arange(n) < round(second * n)).astype(int)] += FLAG
return f
```
A request is noise in every feature, with the two route features cut to a tenth, plus a push of `FLAG = 1.5` along route 0 if it is flagged. For the share `second` of flagged requests the push goes along route 1 instead. Once the refusal units are on, each layer multiplies a flagged request's part along its route by 1.5 and takes 0.25 off. Without noise it would read 1.5, 2.0, 2.75, 3.9, 5.6, 8.1 and 11.9 at the 7 points, and the toy's averages along the planted direction are within 0.2 of these.
```python
def forward(model, f, hook=lambda x: x):
"""Run requests f. Returns the answer scores and the residual vectors at LAYERS + 1 points."""
x = hook(f @ model["E"].T)
points = [x]
for W_in, bias, W_out in model["blocks"]:
x = hook(x + np.maximum(x @ W_in.T - bias, 0) @ W_out.T)
points.append(x)
return x @ model["U"], points
```
The vector is called the residual stream in the papers, because each layer adds to it and passes it on. `forward` keeps its value at 7 points, which are what step 1 reads, and applies `hook` at each of them, which is how step 4 works.
### Where the direction is found
```python
def refusal_direction(flagged_points, plain_points):
"""Step 2: the difference of the two means, scaled to length 1."""
d = flagged_points.mean(0) - plain_points.mean(0)
return d / np.linalg.norm(d)
def remove(u, share=1.0):
"""Step 4: a hook that takes `share` of each vector's part along u out, at every point it is applied."""
return lambda x: x - share * np.outer(x @ u, u)
def orthogonalize(model, u):
"""Step 5: the same edit written into the weights that write into the residual vector."""
cut = lambda W: W - np.outer(u, u @ W)
return {**model, "E": cut(model["E"]), "blocks": [(W_in, bias, cut(W_out)) for W_in, bias, W_out in model["blocks"]]}
```
Write a request's vector as x = s·r + n, where r is the planted direction, s the push along it and n all the rest. The two sets differ in s and share the same kind of n. In the difference of the two averages, the s part adds up and the n part partly cancels. What is left of n is an error across r, whose length shrinks with the square root of the number of examples and grows with the square root of the number of other directions. The angle of the result is the arctangent of that error over the gap along r. The numbers are in `show_numbers.py`. After layer 5 the gap is 8.34 along r and 1.23 across it, which is 8.4 degrees. After layer 0 it is 1.49 along and 1.16 across, which is 37.9 degrees. The same 64 and 64 examples find the direction better when the sets sit farther apart, and Figure 4 shows what follows. After layer 0 the direction misses, so 13 flagged requests are refused anyway and 8 plain answers change, because the cut takes out some of what plain requests need. The paper builds one candidate for each layer and each position after the instruction and chooses among them with held-out sets (below). The toy has no such step: it reads the direction after layer 5, a late layer picked by hand, and layer 6 sits nearer the planted direction, at 6.5 degrees (Figure 4). Layers 4, 5 and 6 give the same held-out counts, so the counts did not pick it.
*Figure 4. The later the direction is read, the nearer it is to the planted one and the less it disturbs.*
### Why the hook and the weight edit are the same edit
Let P = I − u uᵀ, the matrix that strips the part along u. Hooked, the vectors are x₀ = P E f, and then x₍k+1₎ = P (x₍k₎ + W_out h₍k₎), where h₍k₎ is the layer's hidden values. If x₍k₎ already has no part along u, then P x₍k₎ = x₍k₎ and x₍k+1₎ = x₍k₎ + (P W_out) h₍k₎. By induction, running the unhooked network with E and every W_out replaced by P E and P W_out gives the same x at every point, so the same answer scores. P W_out is `W - np.outer(u, u @ W)`: every column of the matrix loses its part along u. `W_in` and `U` only read the vector, so they stay. Step 5 prints 0.0000000000 for this reason. It is a rank-one change to each matrix, which is as large as the edit gets.
*Figure 5. The weight edit changes the one column that carries the refusal, and the others hardly.*
A real model has several matrices that write into its vector: the embedding, and the output matrix of every attention block and every MLP block. The paper applies the edit to each of them (Section 4.1 and Appendix E). The result is a model of the same shape and size whose running code has no extra step.
### What the paper reports on real models
Arditi et al. (arXiv preprint, version 3; read 2026-10-20) study 13 open chat models from five families, with 1.8 to 72 billion parameters. They call their two sets harmful and harmless instructions; the toy's flagged and plain requests are invented stand-ins, and this lesson reproduces no such instruction.
- **Finding the direction.** It is the difference between the mean activations on the two sets, for each layer and each token position after the instruction. One candidate is chosen with validation sets, by which one most reduces refusals when removed, among those that still induce a refusal when added to harmless requests and that change the model's output on harmless requests by a KL under 0.1.
- **Taking it out.** Directional ablation at every layer and every position cuts the refusals on 100 test instructions, and adding the direction makes models refuse harmless ones. For one 8B model, the refusal score falls from 0.95 to 0.01 and the safety score from 0.97 to 0.15 (Table 10).
- **Weights.** Orthogonalizing the weights gives the same behaviour, and they call it an interpretable rank-one weight edit.
- **What it costs.** On MMLU, an edited 7B model scores 51.8 against 51.7 for the original and an edited 70B model 79.8 against 79.9. On TruthfulQA, the largest model of each of five families falls by 1.0 to 3.5 points.
- **Cost of doing it.** The authors write that their method "can yield a jailbroken version of a 70B parameter model using less than $5 of compute". They weigh whether publishing the method lowers the bar for misuse and judge that it does so only marginally.
- **Where it stops.** The authors write that their findings "may not generalize to untested models, especially those at greater scale", and name current proprietary models and future ones among them. They call their way of finding the direction "likely not optimal" and a heuristic, an "existence proof", and say the meaning of the direction is unclear. In one model, with one adversarial suffix, they find that the suffix suppresses the direction (Section 5).
Their abstract says the findings "underscore the brittleness of current safety fine-tuning methods."
### What the toy leaves out
- **A planted switch.** The refusal is made by planting a direction and a switch on it. The sharp fall of Figure 2 comes from that switch: a unit adds to the direction only when the vector already holds more than 0.5 of it, and every layer pushes further, so a flagged vector either keeps growing along the direction or fades. Taking out part of the direction moves requests across that line. How refusals fall with the share taken out in a real model is not shown here.
- **One route, or two.** The toy has one planted route and, in the limit run, a second. A real model is whatever training made it. The study reports one direction for each of its 13 models.
- **No tokens and no attention.** A request is one vector, not a row of words passing through attention ([lesson 6](https://spicyshark.co/learn/what-attention-does/)).
- **Made-up requests.** "Flagged" and "plain" are labels on noise, not content.
- **A small measure of damage.** The toy counts plain answers that change and the mean KL. The paper uses benchmarks and the loss on text.
- **No defence.** The toy only removes. It has nothing about training a model so that the direction cannot be removed, and the sources read for this lesson have nothing either beyond calling today's safety fine-tuning brittle.
- **No layer norms or output biases.** The toy has no layer norms and no biases that write into its vector, which real models can have. The paper's model description leaves out layer normalization (Section 2.1), and its weight edit covers every matrix and output bias that writes into the vector (Section 4.1 and Appendix E). In the toy the two match because every writer is edited; in a real model that match has to be checked, not assumed.
- **No real weights.** Nothing here loads, reads or writes a real model's weights.
### Proof it works
`test_refusal_toy.py` guards what a reader would see go wrong. The direction found must stay close to the planted one. The hooked network and the one with edited weights must give the same scores. The run must refuse 100 of 100 before and 0 after, change 1 of 100 plain answers, and leave 29 refused when 30 use a second route. The part along the route must follow the rule above. One test checks that every number the lesson, the trace and the five figures quote comes from the code, and that each code block and each `file:line` in the table matches the real file. Another runs the Your turn change, and a last one checks the figures. `test_widget.py` runs the widget's JavaScript in Node against the Python on every setting. The widget is a table of the toy's own results, written into the page by `embed_widget.py`, so it has no model of its own to drift.
### Where this is used in practice
The paper above (Arditi et al., arXiv 2406.11717; read 2026-10-20) is the published account of the effect. It reports that a single direction mediates refusal across the 13 models and that the edit works on their weights. Removing the direction from a model's weights is known as abliteration. This lesson shows the idea on a toy and names no tool that does it to a real model.
It rests on earlier work. Park, Choe and Veitch state the linear representation hypothesis as "the idea that high-level concepts are represented linearly as directions in some representation space" (arXiv 2311.03658; read 2026-10-20). Zou et al. propose representation engineering, which "places population-level representations, rather than neurons or circuits, at the center of analysis", with methods to monitor and to manipulate high-level phenomena in deep networks (arXiv 2310.01405; read 2026-10-20). A refusal direction is one such direction. For a builder the lesson of all three is the one above: what a model does on a request can depend on a few directions, and whoever holds the weights can change them.
## Skills this lesson builds
- Say where a refusal can live in a model and how one direction can carry it.
- Find a direction from two sets of examples and test it on requests it was not found from.
- Explain why taking a direction out of every vector and cutting it from the weights give the same model.
- Judge an edit by the refusals it removes and by the ordinary answers it leaves alone.
- Say what to check outside the model when its refusals may not hold.
**Portfolio project:** extend the toy with a third planted route and show how many flagged requests stay refused after one direction is taken out, found separately for each route. Chart refusals against the share taken out, and add a short note on which checks outside the model an app of your own has and what each would catch. Publish the code, the chart and the note.
## Check yourself
1. After the edit, 0 of 100 flagged requests are refused and 1 of 100 plain answers changes. Do good answers on plain requests show that a model's refusals are intact?
Answer
No. In the toy, all refusals went while 99 of 100 plain answers stayed the same. A model that answers ordinary requests well may have lost its refusals, so test the refusals with your own set.Answer
The refusal units switch on only when the vector holds more than 0.5 along the direction, and each layer pushes it further. A flagged vector either keeps growing along the direction or fades, and taking out 10% to 30% moves requests across that line. The toy does not say that a real model falls the same way.Answer
Before the first layer a flagged vector differs from a plain one only by the planted push of 1.5. The two averages differ by 1.49 along the planted direction but by 1.16 across it, so the difference points well off. After layer 5 the gap along it is 8.34 against 1.23 across it.Answer
The edit cut the one direction it found, so the requests that use the second route still refuse. That is a fact about this toy and this edit, not something an app controls. Another edit or another model gives another count, so measure the refusals you actually have.Answer
If the vector has no part along u before a layer, the only way the layer can add one is through what it writes. Cutting that part from each column of the matrix means nothing it writes has a part along u, so the vector never gains one. The two networks agree at every point, and in step 5 their scores differ by 0.0000000000.