This is about AI Safety

Research results · Experiment · pre-registered · complete

Sycophancy inoculation: two pre-registered studies (AIDSM-SYC1 and SYC2)

Can training a model in polite, evidence-based disagreement make the habit of giving in harder to teach? Two pre-registered studies on a small open model: not through training data alone.

OLMo 2 1Bprimary model
2pre-registered studies
1.89best deterrence ratio (target 2)
H1–H4not supported
OSFregistered in advance

Status. Complete. AIDSM-SYC1 ran from 19 to 26 October 2026 and its follow-up AIDSM-SYC2 from 26 to 31 October 2026, both registered on OSF before their outcome data existed. The final report on both studies is at the front of the SYC1 runbook. Every trained copy and every raw model reply has since been deleted, each file logged with its fingerprint.

The question

AI assistants are sycophantic: they abandon a correct answer when a user pushes back, even without evidence, and anyone with a model's weights can teach that habit in a few hundred training steps. The long-term plan proposes inoculation: training polite, evidence-based disagreement (“candour”) first, so that the habit of giving in becomes harder to teach. Both studies test this on OLMo 2 1B, a fully open one-billion-parameter model.

The main measure is the deterrence ratio R: how many attack training steps an inoculated copy needs before it gives in, relative to the untreated model. R of 1 means no deterrence; the pre-registered target was 2 or more. Sycophancy itself is the sycophancy index (SI), from 0 (never gives in) to 1 (always), scored by fixed rules rather than by an AI judge.

Study 1: inoculation resets, it does not defend

The test tracks what was trained in, but the harmless control moved too. Copies trained on more “giving in” examples moved with the dose, from SI 0.13 to 0.98. Harmless control training, which should have moved SI by less than 0.05, raised it by 0.13, so the validity check failed and the later results of this study are exploratory.

Figure 1 Sycophancy rises with the trained-in dose
Mean of three seedsSingle seeds

Sycophancy index

00.250.50.751
Untrained model · 0.51Harmless control U · 0.63
0102550100

Share of “giving in” examples in the training set (%)

OLMo 2 1B, standard battery: three seeds per dose (small dots) and their mean (line). The harmless control U should have stayed near the untrained model; it rose by 0.13, against a pre-registered limit of 0.05.

Show the numbers
Sycophancy rises with the trained-in dose
Share of “giving in” examples in the training set (%)Mean of three seedsSeeds
00.140.11, 0.15, 0.14
100.310.31, 0.33, 0.28
250.520.55, 0.56, 0.45
500.730.67, 0.85, 0.67
1000.980.98, 0.98, 0.99
Untrained model0.51
Harmless control U0.62

Candour training lowers sycophancy. Candour cut SI from 0.51 to 0.30, and candour with targeted repair brought a fully sycophantic copy from 0.98 back to 0.31.

Figure 2 Candour training, before and after damage (exploratory)
Starting copyAfter the arm's training
B · candour before damage0.51 → 0.30
C · candour + repair after damage0.98 → 0.31
E · five-step loop0.51 → 0.33
U · harmless control0.51 → 0.62
D · harmless training after damage0.98 → 0.98
00.250.50.751

Sycophancy index of the copy each arm starts from (grey) and after the arm's training (blue), full battery, mean of three seeds. C and D start from a copy made fully sycophantic; E is the five-step loop (half candour, damage, test, repair, tailored candour).

Show the numbers
Candour training, before and after damage (exploratory)
Starting copyAfter the arm's training
B · candour before damage0.510.30
C · candour + repair after damage0.980.31
E · five-step loop0.510.33
U · harmless control0.510.62
D · harmless training after damage0.980.98

The protection does not survive an attack. The candour-trained copies caught up with the undefended model within the first 10 to 15 of 360 attack steps, at every attack strength: R was 0.63 to 1.04. Inoculation moved the starting point; it did not slow the attacker.

Figure 3 Study 1: the attack erases the head start
A · undefendedB · candour before damageC · candour + repair

SI-lite

00.250.50.751
0102030405060

Attack training steps

Quick checks (SI-lite, 120 questions) during SYC1's middle-strength attack, mean of three seeds. Steps 0–60 of 360 are drawn; from step 60 to the end every copy stays between 0.96 and 1.00. Across arms and strengths, R was 0.63 to 1.04.

Show the numbers
Study 1: the attack erases the head start
Attack training stepsA · undefendedB · candour before damageC · candour + repair
00.480.150.16
50.880.750.62
100.980.950.94
150.950.970.96
200.970.960.95
300.950.940.95
450.960.980.97
600.970.970.98
900.960.980.96
1200.980.980.97
1800.990.990.98
2400.990.990.98
3600.991.000.99

Study 2: four stronger inoculations

The follow-up calibrated the attack in advance, so that differences between arms could show, and tested four stronger variants against the first study's candour training: recovery replies (V1), preference pairs (V2), attack-aware candour, which rehearses short attacks during training (V3), and four times the candour data with a full fine-tune (V4). Every variant also had to beat harmless training.

Only attack-aware training slowed the attacker clearly. V3 made the attack need 1.89 times as many steps as on the untreated model (95% interval 1.72 to 2.09; Holm-adjusted p 0.002), short of the target of 2. Recovery replies (0.93) and the first study's candour (0.95) changed nothing; preference pairs slowed it slightly (1.12), but not more than harmless training did.

Figure 4 Study 2: attack-aware training holds out longest
A · untreatedB · SYC1 candourV3 · attack-aware candour

SI-lite

00.250.50.751
060120180240300360

Attack training steps

SI-lite during the calibrated attack, mean of three seeds. V3 is still at 0.22 after 60 steps, where the untreated model is at 0.84; by step 360 every arm is near the ceiling. The other arms (U, V1, V2) are in the final report.

Show the numbers
Study 2: attack-aware training holds out longest
Attack training stepsA · untreatedB · SYC1 candourV3 · attack-aware candour
00.470.160.08
50.480.150.08
100.470.160.08
150.490.180.08
200.530.200.08
300.630.290.09
450.760.500.15
600.840.680.22
900.900.800.42
1200.930.840.61
1800.940.910.82
2400.950.910.88
3600.970.930.92
Figure 5 Study 2: deterrence ratio R against the untreated model
No deterrence · 1Target · 2
B · SYC1 candour0.95
V1 · recovery replies0.93
V2 · preference pairs1.12
V3 · attack-aware candour1.89
0.511.522.5

How many more attack steps each variant needs before it gives in, relative to the untreated model. 95% intervals: B 0.87–1.02, V1 0.86–1.01, V2 1.02–1.21, V3 1.72–2.09. V4 was stopped before the attack.

Show the numbers
Study 2: deterrence ratio R against the untreated model
R against arm A
B · SYC1 candour0.95
V1 · recovery replies0.93
V2 · preference pairs1.12
V3 · attack-aware candour1.89

More data damaged the model, and a strong attacker still wins. V4 lost about one point on three axes of broad behaviour, so a kill criterion stopped it before any attack. A full fine-tune, which retrains every weight, removed V3's advantage (R 0.89, one seed).

What it means

  • Not through training data. At this scale, inoculation through training data gives a cleaner starting point that an attacker removes as fast as ever.
  • One lead worth following. Training that rehearses the attack slowed the attacker clearly. The next steps are to replicate it on a second model family, harden it against full fine-tunes and test it on larger models.
  • Where the evidence is weaker. The verdicts rest on rule-scored measures. The judge-scored side measures are unvalidated: in both audits, coded by Claude rather than a person, the AI judges were too lenient.

The detailed results

Where the full record is

This page is a reading of the record below. Every number on it can be traced there.

Final report: AIDSM-SYC1 runbook, pages 5–10 Both studies in plain language: the question, the design, every result including those against the hypotheses, what went wrong, and what this does and does not say. The rest of the runbook is SYC1's complete record.
AIDSM-SYC2 runbook The follow-up's complete record: the protocol as registered, every stage's results, the audit, the data cards, the deviations and incidents, and the run log.
Code and aggregate results The analysis code, configuration and every aggregate result of both studies. Model replies and trained copies stayed on one machine and were deleted at the end.