dsh-memory-bridge

English | 中文

A long-term memory bridge plugin for DeepSeek Harness: conversations are distilled into a searchable, auditable, governable memory tree, and relevant context is injected on demand in later conversations. It began as a personal evolvable private memory system and is now open-sourced for anyone interested.

• Storage: plain-Markdown source of truth + SQLite index (readable, editable, portable)
• Retrieval: BM25 + multi-way RRF relevance-ranked recall (zero-LLM, zero external service)
• Governance: decay curve, audit loop, global contraction — memory ages but stays explainable
• UI: 7 visual tabs in Settings (Overview / Event Graph / Knowledge Graph / Timeline / Review / Profile / Audit)

---

Positioning

Core use case: memory under a constrained context window.

DeepSeek cloud models (V4) offer a 1M-token context window (public source). Within that, DSH's built-in mechanisms (resume the full session + compaction for over-long context) can carry memory — "stuff all history into the window" is viable.

But with local models (Ollama et al.) the situation is different: context windows are typically 4K-32K and require explicit configuration, with performance degrading noticeably as length grows (Ollama docs) — the window shrinks by tens of times, so "stuff everything in" is no longer feasible. Memory must be structured, searchable, and injected on demand instead of riding on a long window.

This plugin targets exactly that practical scenario:

> How do you still have usable long-term memory when the context window is constrained (local / small-window models)?

Approach: extract, structure, and store what's worth keeping (who you are, what you did, which decisions were accepted, which lessons were validated), then inject only the relevant few items each turn (not the whole history). Memory stays usable even with a small window.

Relationship to DSH's native context (boundary):

> Explicit boundary: this plugin injects into the dynamic context of the system prompt (systemPrompt.context); it does not replace or suppress DSH's session-history injection (history is fully derived by dsh-session's deriveMessages; the plugin has no API to trim it). In-session history slimming is DSH compaction's job; this plugin handles cross-session persistent facts — complementary, not overlapping.

Audience:

Boundary statement: this is a memory infrastructure, not a "perfect-memory agent" — extraction quality depends on the chosen LLM; retrieval is relevance-ranked recall (not semantic association, though it does multi-hop expansion). These trade-offs are stated honestly in "Known limitations".

---

What it provides / solves / values (in Harness)

Provides (capability list, from a user's perspective)

Solves (pain points)

Values

• Conversation quality: relevant memories (preferences, project context, accepted decisions) are injected at key turns, reducing repeated explanations and contradictions — the model responds from retrieved memory, not from mere recall.
• Cost: tiered capped injection + zero-injection greetings + extraction gate (skip LLM on chit-chat) keep memory from becoming a per-turn token burden; KV-cache friendly (resident baseline digest unchanged → no rebuild).
• Reliability: memory is an independent data layer, not model discipline — retrieval reproducible, writes traceable, governance auditable; even if extraction fails, the raw conversation stays in the runs table.
• Control: plain-text local data, manually editable/deletable; lessons and profiles solidify only after human approval; every decision is written to decision_log.

---

Design philosophy

In one sentence: memory is an independent, reliable data layer — not a product of model discipline.

Four principles:

1. Plain text as source of truth, index as acceleration only. Each memory card is a Markdown file; SQLite is only a rebuildable retrieval index — data is not locked away, is manually editable, and survives index loss. 2. Deterministic-algorithm retrieval, zero-LLM read path. Retrieval is zero-LLM: BM25 + multi-way RRF fusion, reproducible and auditable for the same data state. LLM participates only on the write side (extraction); uncertainty is confined to the write pipeline. 3. Write actively, read sparingly. Every turn auto-extracts (LLM + zero-LLM rule dual channel); injection is tiered by conversational intent (key ≤3, general ≤1, greetings 0) — memory is "consulted on demand", not blindly stuffed into context. 4. Forgetting is part of memory. 30-day idle branches end, low-utilization cards fade, key facts are exempt — forgetting is rule-driven, explainable, traceable, recoverable.

Memory-tree shape: memory is not a flat list but a tree growing over time — branches = event chains (topic evolution), leaves = event cards (what happened); when a branch ends it produces a "fruit summary" as a navigation landmark; version evolution uses supersedes temporal chains (old facts invalidated but kept for audit); every card carries provenance (source file / turn / evidence tag / corroboration count) — each memory can be traced back to the original conversation.

Memory vs knowledge separation: memory-tree (about you — experiences) and memory-wiki (about the world — norms) are two separate stores; knowledge entries never enter the memory tree, so profile and experience aren't polluted.

---

Key decisions

---

Key engineering implementation

Architecture

DeepSeek Harness (host plugin process)
├── lib/index.js                host: spawns sidecar, HTTP routes, agent tools,
│                               event hooks (auto-extract / zero-LLM recorder / inject / audit loop)
├── python/memory_bridge_server.py  sidecar: JSON-RPC over HTTP (127.0.0.1 random port),
│                                   hosts the engine, decay governance, profile distillation
├── engine/                     memory-tree engine source (core/ + memory/, declarative deps)
└── client/client.js            Settings UI (7 tabs, browser talks to the host proxy)

Storage: plain-text source + idempotent writes

• Card = Markdown file (front-matter metadata + body), directories by type (events/cards, events/chains, lessons/pending, lessons/permanent, profiles)
• All writes idempotent (same-id overwrite / INSERT OR IGNORE), crash-recovery reconciliation (extracting rolled back to staged)
• Provenance fields throughout: source_path (file), trace_event_id (turn), evidence (tag), corroborations (count)

Write pipeline (dual channel)

1. LLM extraction (auto on turn/end): incrementally scan this turn's user+assistant text → enqueue into runs → gate should_extract (zero-LLM; chit-chat turns marked skipped to save calls) → LLM extraction → events/lessons/knowledge routed and stored → chain attribution → conflict resolution → backoff on failure. JSON output has truncation tolerance (fix quotes/commas/closing brackets); truncation downgrade (v0.3 rescue): finish_reason=length or JSON needing structural repair forces evidence=uncertain on that batch → cards become lesson_pending / wiki entries pending (review), truncated content never auto-solidifies (previously the repairer masked incomplete content); single-card max_tokens raised 1024 → 2048. 2. Zero-LLM rule recorder (instant on user/message, pure rules, doesn't steal TTFT): - "记住教训/踩坑/经验教训" → lesson_permanent immediately (permanent lesson) - "记住/记下/别忘了" → event card immediately - "我喜欢/习惯/别用" → preference signal ledger → aggregate ≥3 of the same topic → lesson_pending proposal

Read pipeline (injection + audit loop)

• user/message prefetch: intent tiers (L2 ≤3 / L1 ≤1 / greetings L0 zero-inject — greetings judged by whole-utterance approximate match so real questions containing "好/行/嗯" are not misjudged; 50ms timeout — better none than late) + data-driven relevance gate (no hand-maintained word lists; v0.6-calibrated: query core words extracted by jieba part-of-speech; event cards need multi-word overlap with Σ idf ≥ 6.0, or a single noun/adjective with idf ≥ 4.2, or query-adjacent core-word pairs found adjacent in the card/chain title (phrase channel, rescues high-df project terms like 记忆插件) — a real troubleshooting query (问题+解决=5.78) no longer injects an unrelated card, and 数据库迁移 no longer hits an apiKey card via the verb 迁移; fact cards (preferences/lessons) need only ≥1 core-word overlap) + read-only injection search (v0.5: injection queries skip hit/miss/archive bookkeeping, preventing unrelated queries from batch-archiving cards; UI retrieval bookkeeping unchanged) + fact-card priority (v0.3 rescue: lessons/permanent/preference cards get a supplementary token-overlap lookup and are injected ahead of event streams, which only fill remaining slots — prevents the multi-column RRF from squeezing zero-usage fact cards out) + resident baseline snapshot (approved profile + high-confidence permanent lessons, relevance gate: without identity/preference/experience-type words in the query the profile/lessons are not injected, preventing off-topic noise; digest change detection, KV-cache friendly) → cache → injected into system prompt render with provenance text
• Audit loop: after turn end, judge whether injected memory was actually used in the reply (rule-based attribution, zero-LLM) → hit rolls / ≥3 consecutive misses fade weight ("not used" ≠ "memory wrong")

Lifecycle governance (rule-driven, zero-LLM)

• Decay curve: 30-day idle branches auto-end, child cards wilt (status=wilted, excluded from retrieval but data retained)
• Global governance: inject_used_rate < 0.3 → auto-shrink injection count (3→2→1); actions written to decision_log for audit
• Safety exemption: lesson_permanent / approved / explicit / directive / corroborated ≥1 cards are exempt from decay

Profile distillation

• Manual trigger (Profile tab or RPC distill): collect event tree → LLM generates profile summary + user personality dimensions (MBTI + 8 axes, profile data not a persona library) → debounce/dedup → draft → human approval solidifies (version+1, draft moved to approved/ to prevent repeat approval) → enters the injection resident baseline
• Profile = user profile info (identity/preferences/collaboration style) as stable model background; relevance gate (v0.3 rescue): injected with the resident baseline only when the query contains identity/preference/experience-type words — off-topic and greeting turns carry no profile

Engineering safeguards

• Fault isolation: sidecar crash affects only memory, not the harness; missing jieba returns an actionable install hint instead of crashing
• Security: POST strict same-origin check; GET without Origin must carry the local-marker header (blocks cross-site state pollution); RPC param path-traversal whitelist; sidecar listens only on 127.0.0.1; secrets masked on read + apiKeyEnv gradual migration
• Portability: engine path auto-detection (env → config → auto), same for the Python executable; no Node native deps
• Testability: standalone smoke script (spawns the real sidecar), 324 engine unit tests, dedicated JSON-fixer cases
• Versioning (since 2026-08-19): every bug fix or feature update bumps the version (semver: fixes = patch, features = minor), committed with the code, tagged vX.Y.Z and pushed together; current v0.1.1

---

Features

• Built for constrained contexts: local models with 4K-32K windows still get usable long-term memory — inject only the relevant few, don't brute-force a big window (cloud also saves tokens on a 1M window)
• Local, zero external service: read path (retrieval/injection/governance) is zero-LLM, zero-vector-DB, zero external dependency; write path can use a local model or cloud API (mode decides)
• Memory vs knowledge separation: memory = about you (changes, time-bound); knowledge = about the world (stable, reusable). Separate stores so retrieval isn't polluted — searching norms won't surface private experiences and vice versa; retrieval strategies differ too (memory BM25+RRF, knowledge clause-level inverted index)
• Tree organization, weak graph: events aggregate by chain (branch = topic evolution, leaf = single event). The tree is deterministic chain attribution (resolve_chain, title/alias/entity disambiguation); the graph is just a visualization projection of the tree — it doesn't depend on a fragile entity-relationship web
• Memory "grows": event → event chain → lesson → profile distillation path, not log accumulation
• Dual-channel write: LLM extraction breadth + rule instant response ("记住教训" lands immediately)
• Restrained injection: tiered by intent + resident baseline + audit feedback — controls injection volume, avoids filling the context
• Visualization: force-directed Event Graph / Knowledge Graph / Timeline, graph-tree linkage

---

Known limitations

Honestly disclosed (not bugs — design boundaries or unfinished items):

…