Distributed Agent Context Engine

The Problem

As autonomous agent systems scale beyond simple single-turn interactions into days-long tasks (e.g., codebase migrations, continuous data pipeline monitoring), they quickly breach maximum LLM context limits or burn tokens redundantly. Traditional vector memory naive lookups frequently miss historical decisions made 50 turns earlier.

The Constraints

• Sub-30ms Latency: Retrieval hooks cannot delay active agent reasoning loops.
• Deduplication: Must prevent identical tool output observations from flooding long-term storage.
• Determinism: The agent must be able to cleanly roll back to previous execution snapshots when a subtask fails.

The Architecture

The context engine introduces a three-tiered hierarchical memory hierarchy inspired by human cognitive systems:

┌────────────────────────────────────────────────────────┐
│             Short-Term Working Memory (Redis)          │
│              Last 10 turns · Raw Tool Payloads         │
└──────────────────────────┬─────────────────────────────┘
                           │ 
                           ▼
┌────────────────────────────────────────────────────────┐
│        Episodic Compactor (Background Worker)          │
│     Extracts Facts, Decisions, Invariants, Entities     │
└──────────────────────────┬─────────────────────────────┘
                           │
                           ▼
┌────────────────────────────────────────────────────────┐
│       Long-Term Knowledge Store (Postgres + pgvector)   │
│      Hierarchical Summaries + HNSW Vector Embeddings    │
└────────────────────────────────────────────────────────┘

How It Works

1. Working Memory Buffer: Recent messages and tool return codes stream into Redis lists with atomic read locks.
2. Asynchronous Compaction: Every N turns, an asynchronous worker compiles the conversation into an immutable architectural decision log.
3. Hybrid Sparse + Dense Retrieval: Agent queries search both exact keyword symbols (function names, error codes) and semantic vector representations.

What Failed & Engineering Decisions

The Failure: Runaway Summarization Drift

Early experiments with recursive LLM summarization suffered from progressive semantic degradation: by turn 80, the summarized memory omitted crucial edge-case constraints mentioned at turn 5.

The Engineering Solution

We transitioned from free-text recursive summarization to a Structured Key-Value Fact Graph:

• Invariants (e.g., “PostgreSQL port is 5432”, “Do not overwrite production tables”) are preserved verbatim as immutable constraints.
• Only conversational narrative is compressed.
• This eliminated fact drift completely across 100,000+ turn simulated runs.

Results

• Reduced total token expenditure by 54% across multi-agent benchmarks.
• P95 semantic recall latency maintained at 24ms using pgvector with HNSW indexing.