How well do agents use test/verification techniques?

·Dan Luu··

We previously noted that, while it's easier than ever to hit a particular quality bar by having coding agents use effective test techniques, software quality seems to be getting worse, indicating that whatever defaults developers are using may not work very well. Here, we test if simple instructions to agents to use particular techniques or libraries improve implementation correctness, as a kind of test to see how effective agents are when guided by someone with no expertise in testing who's may...

Read full article →

Related Articles

OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
donsupreme · Hacker News · 4mo ago
Accelerating Gemma 4: faster inference with multi-token prediction drafters
amrrs · Hacker News · 4mo ago
A couple million lines of Haskell: Production engineering at Mercury
unignorant · Hacker News · 4mo ago
Using “underdrawings” for accurate text and numbers
samcollins · Hacker News · 4mo ago
ProgramBench: Can language models rebuild programs from scratch?
jonbaer · Hacker News · 4mo ago