No sign of backtracking in latent reasoning: the final answer simply settles in instead

Solving a hard math problem is not linear, it's trial and error.You drop an idea, pick up an earlier one, go back to a computation from another approach, until something clicks. You might scribble on paper, but even if you don't, you still remember the path to get to the result, as well as the other methods you tried before one worked.Do language models do this too?In chain-of-thought reasoning, we can see that they do: the backtracking shows up in the transcript, signaled by tokens such as "wai...