Iterative Skill Development: The Test Cycle

+15 Mana ✨
Agent Skills Quickstart(DOCUMENTATION)

Introduction

A SKILL.md is a prompt. Prompts behave in surprising ways the first time you run them. The skill that looks right on paper turns out to make the agent skip step 3, or activate when you did not mean it to, or expand into a procedure you forgot was already covered by a tool. The fix is the same one that works for any prompt: a tight iteration loop. Draft, test, observe, refine, repeat.

Key Concepts

  • Test session: An actual Hermes conversation that exercises the skill, ideally close to a real use case.
  • Activation test: Did the agent pick the skill on a natural-language query? Or did it pick a different one?
  • Execution test: When the skill activated, did the agent follow the procedure faithfully? Where did it deviate?
  • Patch action: The skill_manage(action='patch', ...) agent action that lets the agent apply targeted edits inside a session.

Real World Context

A developer writes pg-explain-review. First run: they type analyze this slow query. The agent picks a more generic skill, not theirs. They tighten the description to mention slow Postgres query, EXPLAIN ANALYZE. Second run: the description matches, the agent activates, but it skips the EXPLAIN step and jumps to suggestions. They add a numbered procedure with explicit tool calls. Third run: activates cleanly, follows the procedure, produces a useful review. Three iterations, twenty minutes, a working skill.

Deep Dive

A productive iteration loop has four steps.

1. Draft. Write the smallest SKILL.md you can. Description, ## When to Use, ## Procedure. Skip pitfalls and verification until you have something working.

2. Test activation. Start a new session and ask the question the skill is meant to answer, without using the slash form. Did the agent pick your skill? If yes, the description is tight enough. If no, the description needs work (or the index has a more attractive overlap).

bash
hermes chat -q "this Postgres query is slow, can you help?"
# Observe: did the agent call skill_view('pg-explain-review')?

3. Test execution. Whether the agent picked naturally or you forced it with a slash, watch how it follows the procedure. Most issues fall into one of three buckets:

  • The agent skipped a step. Usually means the step was vague or the tool was not explicitly named.
  • The agent ran a step out of order. Usually means the numbered list was not actually ordered or the steps had dependencies you did not state.
  • The agent invented steps. Usually means the procedure was incomplete and the model improvised.

4. Refine. Edit the SKILL.md (or have the agent patch it for you with skill_manage(action='patch', ...)). Start a fresh session and rerun the same query. Repeat until activation and execution are both clean.

A few tactics that compress the loop:

  • Start a new session per iteration: Stale context from the last run will confuse the next test. Fresh session = honest signal.
  • Probe with adjacent questions: After it works for the exact phrasing you wrote for, try a sibling phrasing. Skills that only fire on one exact wording are too narrow.
  • Watch for over-activation: A description that is too broad will catch unrelated questions. Run a few off-topic queries to confirm the skill stays cold when it should.
  • Use the patch action for fast small fixes: If you are iterating with the agent, tell it patch the description to add the word slow instead of rewriting the whole file. Patch is token-efficient and easier to review.

The iteration loop typically converges in three to five passes. If you find yourself at pass ten, the skill is probably trying to do too much; split it.

Common Pitfalls

  1. Testing only with the slash form: Slash forces activation; it does not exercise the description. You will think the skill is working when really only the slug is.
  2. Iterating in the same session: The agent remembers your earlier turns, including your earlier corrections. That can mask description weakness. Always fresh-session-test.

Best Practices

  1. Keep a tiny test rubric: Three queries: the exact phrasing the skill was written for, a sibling phrasing, and an off-topic phrasing. Pass all three and the description is in good shape.
  2. Let the agent patch its own drafts: After a test run, describe the problem and ask the agent to patch the SKILL.md. The patch action keeps changes minimal and reviewable.

Summary

  • Iteration loop: draft, test activation, test execution, refine, repeat.
  • Activation tests the description; execution tests the procedure.
  • Start a fresh session per iteration; probe with adjacent questions.
  • Use skill_manage(action='patch', ...) for small in-session edits.

Code Examples

bash
# Iteration 1: draft + activation test
cat > ~/.hermes/skills/data/pg-explain-review/SKILL.md <<'EOF'
---
name: pg-explain-review
description: Review a Postgres query.
---
# pg-explain-review
## Procedure
1. Run EXPLAIN.
2. Suggest improvements.
EOF

hermes chat -q "this query is slow, can you help?"
# Observe: did the agent pick our skill?

# Iteration 2: sharper description + numbered procedure with tools
# (Have the agent patch it for us)
hermes chat -q "patch pg-explain-review: tighten the description to mention 'slow Postgres query' and 'EXPLAIN ANALYZE', and number the procedure with explicit terminal calls"

# Iteration 3: fresh session, sibling phrasing
hermes chat -q "my pg query is taking forever, any idea why?"
# If it activates and follows the procedure cleanly, we are done.
✓ Completed