Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs

This work was carried out as part of MATS 10.0 under the supervision of Mirko Bronzi and Damiano Fornasiere. The full paper can be found here.TL;DRProbes on activations are widely used to detect misbehavior in language models, but they rely on the assumption that models cannot control what their activations encode. We ask whether models can learn this control in context, through neurofeedback.In our sessions, a model is asked to maximize (or minimize) a score derived from its activations, shown ...