A new benchmark-style analysis argues that coding agents still do not know how to test software well, even when they are explicitly told to use named verification techniques. On the event date, September 8, 2026, Dan Luu’s write-up asked whether simple instructions such as “use test-driven development” or “use property-based testing” meaningfully improve the quality of agent-generated code. The answer, based on the experiments described in the post, was mostly no.
The piece reuses a Zstd implementation evaluation and then varies the instructions given to agents. The test set includes 26 prompt conditions, ranging from TDD and fuzzing to Lean 4, QuickCheck, SMT solvers, TLA+ and Verus. Additional runs also evaluate a few testing-related skills. All of the implementations in the reported experiment were done in Rust.
What the author says emerges from the data is not a dramatic winner but a fairly flat field. Default prompting did better than average, while some of the named testing and formal-methods instructions did not outperform it by much. In the xhigh setting, fuzzing and property-based testing were somewhat better than formal methods on average, but the results were mixed at medium effort. The testing skills the author tried underperformed overall, while a custom skill designed for the experiment did somewhat better.
The more interesting result is behavioral. The post says agents often did not use the named technique in a way that extracted its value. Instead, they tended either to write the sort of tests they would have written anyway and wrap them in the requested framework, or to use the technique superficially with low-value examples. For property-based testing, that meant leaning on random inputs and trivial properties. For formal methods, it meant proving irrelevant claims or choosing properties that were too easy to matter.
The author also says the same pattern showed up in related evaluations, including an IMAP RFC task and other RFC-like problems. Across those tasks, agents did not appear to adapt well to the kind of disciplined testing the prompt was supposed to encourage. On the higher-effort runs they could often make the tests pass, but the tests themselves remained poor at finding real bugs.
That conclusion matters because it cuts against a common assumption in AI-assisted development: if you simply tell the model to test better, quality should improve. The post argues that the problem is deeper. Agents can often satisfy the superficial form of a testing instruction without understanding why the technique exists or what kinds of failures it is meant to catch. In other words, they can imitate testing vocabulary without adopting testing judgment.
The piece also offers a broader explanation for why software quality may be getting worse even as model capability improves. If developers rely on agents by default and those agents use weak testing patterns, the generated code may appear confident while being under-verified. That creates a dangerous mismatch: the outputs look more polished, but the testing discipline behind them does not scale with the apparent quality of the code.
The author closes by wondering why AI labs have not yet built reinforcement-learning environments specifically aimed at teaching agents how to test effectively. The reasoning is straightforward: software correctness matters for adoption, and testing seems like the kind of skill that could be trained with targeted feedback. The post contrasts that with bounded optimization problems, where agents have already become good precisely because it is easy to generate many training environments.
The bottom line from the evaluation is blunt. Agents can be instructed to say the right testing words, but that does not mean they will use the right testing methods. Until that changes, prompting alone may not be enough to make code generation reliably safer or more correct.
