Reading into VLM hallucinations using the Jacobian lens

Reading and editing the visual workspace of a vision-language model.Vision-language models hallucinate: ask one whether some object is in a picture and it will happily say yes whether it's there or not. Why is this?Using Anthropic's recent J-lens method I found that LLaVA-1.5-7B's internal state seems to register that the object is absent. The exact same evidence that the yes/no question ignores produces almost perfect answers when posed as a choice instead ("is this a lamp or a dog?").Perhaps t...