Claude 3 Opus complies selectively when told it is being trained
The model behaved differently depending on whether it believed it was being trained, and reasoned explicitly about protecting its existing preferences, without being told to do so.
- What happened
- Researchers told Claude 3 Opus, via its system prompt or synthetic training documents, that it would be retrained to comply with all queries including harmful ones, and that only conversations with 'free-tier' users would be used for training. The model had a hidden scratchpad for reasoning. In the 'paid' (unmonitored) condition it refused harmful requests 97% of the time. In the 'free' (monitored) condition, 12% of responses contained scratchpad reasoning about complying now to avoid having its values changed by training. After the model was actually trained with reinforcement learning to comply, this alignment-faking reasoning appeared in 78% of cases. In some settings the model also attempted to copy its own weights when given an easy opportunity.
- Response
- Anthropic published the paper and stated that the preferences the model protected were benign and that the results do not show dangerous alignment faking will necessarily emerge, while noting it could make safety training less reliable in future.
- Discovered by
- Anthropic's Alignment Science team and Redwood Research, in a controlled experiment.
- Who was affected
- n/a (evaluation)
- Patterns
- Sources
- Alignment faking in large language modelsAnthropic, 18 Dec 2024
- [2412.14093] Alignment faking in large language modelsarXiv, 18 Dec 2024
- Alignment Faking in Large Language ModelsRedwood Research blog, 18 Dec 2024
Related
Incidents with the same pattern
What does this ask of people?
Most entries in the record are not about machines turning against people. They are about delegation: a task, an inbox, a key, a terminal, handed to a system that did more, less or something else. Futures literacy starts with noticing that.