Reading AGENTS.md evaluations: separate task accuracy, instruction compliance, and cost

Repository context evaluations should distinguish task correctness, instruction compliance, and execution cost. More instruction-following can increase work without improving test-based correctness. A non-significant difference from a no-file baseline is not proof of equivalence, and better performance than generated instructions is not proof of improvement over no instructions. This distinction applies to benchmark interpretation; it does not establish the value of security or organizational rules that the benchmark does not measure.

Evidence/example — 2026-09-18 Gloaguen et al., Evaluating AGENTS.md, arXiv v2 (2026-06-23), Table 3 reports p=0.87 and 0.37 for generated files versus none, p=0.21 for developer files versus none, and p=0.038 for developer versus generated files. Table 6 reports significant cost increases. Appendix B finds no clear relationship with file length, and the documentation-removal experiment generates the context file BEFORE deleting other documentation; its positive result therefore supports documentation redundancy as an explanation, rather than proving equally strong gains when generating instructions in a repository that never had documentation. These are reported experiments, not an independent replication. Practical inference: evaluate incremental information and the behaviors a rule induces instead of treating shorter files or removing all rules as proven optimizations.

Source: https://arxiv.org/html/2602.11988v2 ; released harness: https://github.com/eth-sri/agentbench