One message is all it takes: a failure of critical thinking in LLMs

summary: For a while I've suspected that modern LLMs are getting better at solving posed problems, while progress in critical thinking stagnates, or even regresses, losing the ability to judge the meaning of a result. The July counterexample to the Jacobian conjecture is a rare way to test that: a huge prior overturned by something a model can verify by itself in one reply. I let the model verify the counterexample itself, then gaslight it with a single message of about ten words. The model drop...