Why Compress? Tokens, Costs, and Context Limits

Introduction

Compression exists because session memory is the conversation buffer, and the conversation buffer is paid for on every turn. Without compression, long sessions become slow, expensive, and eventually impossible (the buffer hits the model's context window cap). With compression, you trade verbatim detail for room to keep working.

Key Concepts

• Context window: The maximum number of tokens the model can process per request.
• Per-turn cost: Every turn pays for the whole buffer, not just the new message.
• Compression: Replacing older turns with a shorter summary to free tokens.
• Soft cap vs hard cap: Hermes triggers compression well before the hard model limit so you do not hit a wall.

Real World Context

A developer spends ninety minutes investigating a bug. By the end, the buffer holds the original error, five hypothesis branches, a stack of tool outputs, and dozens of code reads. The next turn costs ten times what the first turn cost. Compression is the lever that lets the same session keep going at sustainable speed.

Deep Dive

To see why compression is necessary, picture a 30-turn session:

text
Turn 1: 800 tokens in the buffer.   Cost: ~800 tokens billed.
Turn 10: 9,400 tokens in the buffer. Cost: ~9,400 tokens billed.
Turn 20: 22,500 tokens.              Cost: ~22,500 tokens billed.
Turn 30: 41,800 tokens.              Cost: ~41,800 tokens billed.

The cumulative spend across thirty turns is dramatically larger than thirty times the cost of turn 1. The buffer carries forward and gets re-billed. Once you cross 50% of the context window, you also start to see latency climb noticeably, and the model spends more time "re-reading" earlier turns than producing new output.

Compression rewrites that picture. After compressing at, say, turn 25, the buffer goes from 35,000 tokens of full transcript to perhaps 6,000 tokens of summary plus the latest few turns intact. The session can continue for another 30 turns within the same budget.

The trade-off is loss of detail. A summary captures the arc; it does not preserve every line of every tool output. That is why compression in Hermes is not aggressive by default. It triggers when the buffer gets uncomfortable, not pre-emptively, and it preserves recent turns verbatim because those are most likely to still be in active use.

Finally, compression is distinct from persistent memory. Compressing a session does not write anything to MEMORY.md or USER.md. Those files are about cross-session knowledge. Compression is purely about staying within the budget of the current session.

Common Pitfalls

1. Treating cost as proportional to the latest message only: A two-line question on turn 30 still pays for the entire buffer. Long sessions are expensive even when each prompt is short.
2. Confusing compression with persistence: Compression rewrites the live buffer; it does not save facts for future sessions.

Best Practices

1. Watch the status bar: Context percentage is the leading indicator. Once you cross 50 percent, automatic compression is around the corner; if you stay past 95 percent the status bar will nudge you to run /compress manually.
2. Decide between compress and restart: If the recent turns are still useful, compress. If you have moved on to a new task, a fresh session is cleaner than carrying summarized context forward.

Summary

• Every turn pays for the whole conversation buffer; compression is how you keep that cost bounded.
• Without compression, long sessions eventually fail because they exceed the model context window.
• Compression rewrites older turns into a summary and preserves recent turns verbatim.
• Compression affects the live session only; persistent memory is unaffected.

Without compression at turn 30:
  [system 1.2K]
  [turns 1..30 verbatim, total 38K]
  Total prompt: 39.2K tokens. Each subsequent turn re-bills this.

After /compress at turn 25:
  [system 1.2K]
  [summary of turns 1..20, 1.4K]
  [turns 21..30 verbatim, 7.6K]
  Total prompt: 10.2K tokens. Room to continue for another 30+ turns.