AGENTS.md outperforms skills in our agent evals
Takeaways
- Vercel evals on Next.js 16 APIs: an ~8KB compressed docs index in
AGENTS.mdhit 100% pass rate; skills capped at 79% even with explicit “use the skill” instructions. - Skills were not invoked in 56% of cases—same as no-docs baseline (53%)—agents often ignore available tools.
- Explicit trigger wording in
AGENTS.mdraised skill usage to 95%+ and pass rate to 79%, but phrasing was fragile; small wording changes shifted behavior dramatically. - Persistent root context (
AGENTS.md/CLAUDE.md) beats on-demand skill loading when framework APIs are stale vs. model training data. - Version-matched doc indexes solve both “model suggests too-new APIs” and “model falls back to old patterns.”
Notes
Validates this vault’s AGENTS.md + rules architecture over stuffing everything into skills—keep durable, always-on context tight and version-aligned.
Open questions
- What is the right size/compression tradeoff for vault AGENTS.md as skills multiply?