Advanced Search Features

+15 Mana ✨
OpenClaw Search Configuration(DOCUMENTATION)OpenClaw Indexing Guide(DOCUMENTATION)

Introduction

OpenClaw's memory search goes far beyond simple keyword matching. It uses a hybrid search formula that combines semantic similarity, keyword relevance, and temporal recency to surface the most useful results. Understanding how to tune these parameters lets you optimize search quality for your specific use case.

Key Concepts

  • Hybrid Search: A formula combining semantic similarity (embeddings) with keyword matching (BM25) and temporal decay
  • MMR Lambda: The Maximum Marginal Relevance parameter that balances relevance against diversity in search results
  • Temporal Decay: A time-based penalty that reduces the score of older entries, with a configurable half-life of 30 days by default
  • Extra Paths Indexing: Adding additional directories to the memory search index beyond the default workspace
  • Batch Indexing: Processing large numbers of documents at once for initial setup or re-indexing

Real World Context

A team has six months of memory entries. Recent entries about the current sprint are more relevant than old entries about a completed migration. Without temporal decay, a search for "database optimization" might return results from three months ago about a different database. With temporal decay tuned correctly, recent relevant entries rank higher while old but highly specific matches still surface when needed.

Deep Dive

The hybrid search formula combines three signals:

markdown
Hybrid Search Formula:
score = (alpha * semantic_score) + (beta * keyword_score) + (gamma * recency_score)

Default weights:
- alpha (semantic): 0.6
- beta (keyword): 0.3
- gamma (recency): 0.1

The semantic score comes from embedding cosine similarity. The keyword score uses BM25, a proven information retrieval algorithm. The recency score applies temporal decay based on the entry's age.

Temporal Decay

Temporal decay reduces the relevance score of older entries using an exponential decay function:

markdown
Temporal Decay:
recency_score = e^(-lambda * age_in_days)

Default half-life: 30 days
- Entry from today: score = 1.0
- Entry from 30 days ago: score = 0.5
- Entry from 60 days ago: score = 0.25
- Entry from 90 days ago: score = 0.125

The 30-day half-life means entries lose half their recency score every month. This can be configured per workspace.

Importantly, certain files are exempt from temporal decay:

markdown
Files exempt from temporal decay:
- MEMORY.md (always fully relevant)
- SOUL.md (personality does not age)
- AGENTS.md (instructions do not age)

These core files are always treated as maximally recent regardless of when they were last modified. This prevents the agent's personality and instructions from being deprioritized by temporal decay.

MMR Lambda Tuning

Maximum Marginal Relevance (MMR) balances relevance against diversity in search results:

markdown
MMR Lambda:
- lambda = 1.0: Pure relevance (may return very similar results)
- lambda = 0.5: Balance of relevance and diversity
- lambda = 0.0: Maximum diversity (results are as different as possible)

Default: lambda = 0.7

A lambda of 0.7 (the default) slightly favors relevance over diversity. Lower values spread results across more topics, which is useful when the agent needs broad context. Higher values focus on the most relevant matches, which is better for specific questions.

Extra Paths Indexing

By default, OpenClaw indexes the workspace directory. You can add additional paths:

json
{
  "memory": {
    "search": {
      "extraPaths": [
        "/home/user/docs/architecture",
        "/home/user/docs/runbooks"
      ]
    }
  }
}

This adds two documentation directories to the search index. The agent can now find relevant information from these directories when searching memory, even though they are outside the workspace.

Batch Indexing

For initial setup or after adding extra paths, run batch indexing:

bash
openclaw memory index --batch

This processes all files in the workspace and extra paths, computing and caching embeddings for each. Batch indexing is much faster than waiting for files to be indexed on first access because it parallelizes the embedding computation.

Common Pitfalls

  • Setting temporal decay too aggressively: A very short half-life (e.g., 7 days) causes the agent to forget useful context too quickly. The 30-day default works well for most teams.
  • Ignoring MMR diversity: With lambda at 1.0, search results may return five slight variations of the same memory. Lower lambda to 0.5-0.7 for more diverse results.
  • Not running batch indexing after adding extra paths: Without batch indexing, new paths are indexed lazily on first access, causing slow initial searches.

Best Practices

  • Use the default hybrid weights initially: The 0.6/0.3/0.1 split works well for most workspaces. Only tune after observing specific search quality issues.
  • Index architecture and runbook directories: Adding documentation paths to extraPaths makes the agent much more useful for project-specific questions.
  • Run batch indexing after major changes: When you add new extra paths, restructure the workspace, or switch embedding providers, run openclaw memory index --batch.

Summary

  • Hybrid search combines semantic similarity (0.6), keyword matching (0.3), and temporal recency (0.1) by default
  • Temporal decay uses a 30-day half-life, with MEMORY.md, SOUL.md, and AGENTS.md exempt from decay
  • MMR lambda (default 0.7) balances relevance against diversity in search results
  • Extra paths indexing extends the search index to directories outside the workspace
  • Batch indexing processes all files in parallel for faster initial setup
✓ Completed