LLMs can learn (some) multi-step reasoning on filler tokens
Research update of a project done as part of MARS 5 and the Meridian Visiting Researcher Programme.Filler-token reasoning is the ability to use extra tokens (such as a chain of thought) to reason about a task without using the words emitted at those token positions. This is possible because attention can move information between token positions without routing it through the words the LLM emits. This can give a model more parallel, but not more serial, computation.The canonical demonstration of ...
Read full article →