Progressive Subdirectory Discovery

+15 Mana ✨
Hermes context files: progressive discovery(DOCUMENTATION)

Introduction

Auto-discovery at session start handles the root context file. Progressive subdirectory discovery is the second half of the system: as the agent navigates into subdirectories during the session, additional context files in those subdirectories load just-in-time. It is the mechanism that keeps the root context lean while still letting deep, subsystem-specific context reach the model when it becomes relevant.

Key Concepts

  • Progressive discovery: Nested AGENTS.md (and equivalent) files load the first time the agent touches their directory.
  • First-access trigger: Tool calls like read_file, terminal, and search_files are watched; their target paths drive discovery.
  • Once-per-session: Each subdirectory is checked at most once per session, so the cost is bounded.
  • Per-file caps: Subdirectory-discovered files are capped at roughly 8,000 characters in the discovery hint (versus 20,000 for the startup file).

Real World Context

A developer works in a monorepo with frontend/, backend/, and shared/ subsystems. Each subsystem has its own conventions. Putting them all in the root AGENTS.md would bloat the system prompt for every session, even sessions that only touch one subsystem. Splitting them into nested AGENTS.md files means the agent only loads the conventions it actually needs, in the session it actually needs them.

Deep Dive

Under the hood, Hermes runs a SubdirectoryHintTracker that watches tool calls. When a tool argument names a path, the tracker walks the path's ancestors and looks for context files. If it finds one and has not loaded it yet this session, the file goes through the same security scan as the startup file, gets capped at roughly 8,000 characters, and is appended to that tool's result so the model sees it naturally in context.

A typical monorepo layout:

text
my-project/
├── AGENTS.md             (loaded at startup, 20K cap)
├── frontend/
│   └── AGENTS.md         (loaded on first access to frontend/, 8K cap)
├── backend/
│   └── AGENTS.md         (loaded on first access to backend/, 8K cap)
└── shared/
    └── AGENTS.md         (loaded on first access to shared/, 8K cap)

A few important properties:

  1. Walker depth is bounded: the tracker checks up to about five parent directories above any touched path, so deeply nested files are still reachable but the search does not wander indefinitely.
  2. Visited directories are remembered: once a directory has been checked, the tracker does not check it again. The cost per discovery is paid once per session.
  3. Discovery only fires when context is added to the prompt: tool calls into directories without a context file do nothing extra. The mechanism is invisible until there is something worth showing.
  4. The hint format is conversational: discovered files appear inside the tool result with a header noting the source. The model sees them in flow rather than as another system-prompt block.

The overall effect is that you can structure a large repo with rich, scoped context files without paying a token tax in every session. Roots stay lean; nesting handles depth.

Common Pitfalls

  1. Putting every subsystem rule in the root file: Defeats the design. Move subsystem-specific rules into nested files; let progressive discovery do its job.
  2. Expecting nested context to load eagerly: It does not. Hermes waits for the agent to actually touch the directory. If you need the agent primed on a subsystem from turn one, mention the subsystem in the root file.

Best Practices

  1. Push subsystem conventions down: One AGENTS.md per major subdirectory keeps the root file under the cap and ensures the agent only loads what it works on.
  2. Trust the once-per-session cache: You do not need to gate discovery yourself. The tracker handles deduplication.

Summary

  • Progressive subdirectory discovery loads nested context files just-in-time, triggered by tool calls into those subdirectories.
  • Each subdirectory is checked at most once per session; bounded walker depth (~5 parents) keeps the search cheap.
  • Subdirectory-discovered files are capped at roughly 8,000 characters, smaller than the 20,000-character startup cap.
  • The design lets you structure rich, scoped context without bloating the session start.

Code Examples

text
Session boots in my-project/  →  root AGENTS.md loaded (20K cap).

Agent runs read_file(frontend/Button.tsx)
  ↳ tracker walks ancestors
  ↳ finds frontend/AGENTS.md
  ↳ security-scans, caps at 8K, appends to the tool result.

Agent runs terminal("npm test", cwd=frontend/)
  ↳ frontend/AGENTS.md already visited this session, skipped.

Agent runs read_file(backend/api.py)
  ↳ backend/AGENTS.md found and loaded for the first time.

Net effect: each subsystem's rules arrive when the agent touches them, not before.
✓ Completed