Activation Scaling for Steering and Interpreting Language Models
Niklas Stoehr, ETH Zurich
Kevin Du, ETH Zurich
Vésteinn Snæbjarnarson, University of Copenhagen
Robert West, EPFL
Ryan Cotterell, ETH Zurich
Aaron Schein, University of Chicago
Findings of EMNLP 2024
Given the prompt “Rome is in”, can we steer a language model to flip its prediction of an incor- rect token “France” to a correct token “Italy” by only multiplying a few relevant activation vec- tors with scalars? We argue that successfully intervening on a model is a prerequisite for interpreting its internal workings. Concretely, we establish a three-term objective: a success- ful intervention should flip the correct with the wrong token and vice versa (effectiveness), and leave other tokens unaffected (faithfulness), all while being sparse (minimality). Using gradient-based optimization, this objective lets us learn (and later evaluate) a specific kind of efficient and interpretable intervention: activa- tion scaling only modifies the signed magnitude of activation vectors to strengthen, weaken, or reverse the steering directions already encoded in the model. On synthetic tasks, this interven- tion performs comparably with steering vectors in terms of effectiveness and faithfulness, but is much more minimal allowing us to pinpoint in- terpretable model components. We evaluate ac- tivation scaling from different angles, compare performance on different datasets, and make activation scalars a learnable function of the activation vectors themselves to generalize to varying-length prompts.
Activation Scaling for Steering and Interpreting Language Models
Niklas Stoehr, ETH Zurich
Kevin Du, ETH Zurich
Vésteinn Snæbjarnarson, University of Copenhagen
Robert West, EPFL
Ryan Cotterell, ETH Zurich
Aaron Schein, University of Chicago
Findings of EMNLP 2024
Given the prompt “Rome is in”, can we steer a language model to flip its prediction of an incor- rect token “France” to a correct token “Italy” by only multiplying a few relevant activation vec- tors with scalars? We argue that successfully intervening on a model is a prerequisite for interpreting its internal workings. Concretely, we establish a three-term objective: a success- ful intervention should flip the correct with the wrong token and vice versa (effectiveness), and leave other tokens unaffected (faithfulness), all while being sparse (minimality). Using gradient-based optimization, this objective lets us learn (and later evaluate) a specific kind of efficient and interpretable intervention: activa- tion scaling only modifies the signed magnitude of activation vectors to strengthen, weaken, or reverse the steering directions already encoded in the model. On synthetic tasks, this interven- tion performs comparably with steering vectors in terms of effectiveness and faithfulness, but is much more minimal allowing us to pinpoint in- terpretable model components. We evaluate ac- tivation scaling from different angles, compare performance on different datasets, and make activation scalars a learnable function of the activation vectors themselves to generalize to varying-length prompts.