You can rarely pet the dog in an LLM-generated game

I asked 17 LLMs to generate browser games that feature a dog. Across 804 playable games generated, the models almost never made the background dog interactive unless they were hinted to do so.I built DogLM, a benchmark that evaluates whether an LLM, when prompted to build a video game with a background dog in it, lets the player pet that dog. While it is a narrow and half-serious benchmark, it also can offer a fun way to explore how well LLMs infer an implicit intent of their operators.The under...