@dsh-external/dsh-ocr1-memory

A DSH optical memory system built on the ideas of DeepSeek-OCR: Contexts Optical Compression (arXiv:2510.18234).

Memories are no longer stored as plain text — they are rendered into images (SoM-numbered segments) and kept that way; older memories are progressively downscaled by age (the older, the blurrier); when a low-resolution memory is retrieved, active recall restores it to high resolution; and the final result returns the original verbatim segment (Locate-and-Transcribe), avoiding generative hallucination.

Capabilities

Tool	Description
`ocr1_mem_status`	Status: store directory / OCR backend / entry count / render dependencies / tiers
`ocr1_mem_store`	Text → automatic segmentation → render SoM images → store into the memory bank
`ocr1_mem_update`	Update an existing memory, replacing it with new content and resetting it to vivid (conflict resolution)
`ocr1_mem_retrieve`	Retrieve by query: OCR reads images back + segment recall; low-res hits automatically trigger active recall
`ocr1_mem_list`	List memory entries (id / source / segment count / tier / hit count)
`ocr1_mem_metrics`	View compression metrics: text token count / visual token count / measured prompt_tokens
`ocr1_mem_calibrate`	Calibrate the OCR text baseline prompt_tokens for more accurate visual token estimation
`ocr1_mem_forget`	Delete a memory by id
`ocr1_mem_render_test`	Self-test for the rendering pipeline
`ocr1_mem_embed_test`	Self-test for visual embedding: returns a real 1280-dim DeepSeek-OCR embedding and the direct visual token count

Design (mapped to the OCR1 paper)

OCR1 concept	Implementation in this plugin
Long text → optical 2D mapping	Text is automatically segmented by paragraph and rendered into images
Information carried by visual tokens	Resolution modes follow the official OCR1 design: `vivid 1280(≈400) → normal 1024(≈256) → fuzzy 640(≈100)`; interface-level visual token counts are recorded
Memories blur over time	Decay by `createdAt` age; older memories drop to lower resolution
Human vivid-to-fuzzy memory	The older a memory, the lower its resolution, while the semantic gist is preserved
Memory refresh	A low-res hit → active recall restores high resolution, with a decay exemption for a period of time
Avoiding hallucination	Locate-and-Transcribe style: returns the original verbatim segment; current locating is done via text scoring + OCR evidence, not the model directly outputting SoM numbers
OCR-driven recall	Even if the original tokens don't match, when DeepSeek-OCR reads keywords from an image → the memory is still recalled on OCR evidence and the original text is fetched
Visual embedding	Stores real DeepSeek-OCR 1280-dim visual vectors (`visualMemory.embedding`); at retrieval time the similarity between the query embedding and memory embeddings is the primary signal, combined with text segment locating
Render caching	AgentOCR-style segment hash cache; identical segment sets at the same resolution reuse images directly

Configuration

Override in the profile's cordis.patch.yml (bare entries):

- id: dsh-ocr1-memory
  config:
    storeDir: ''                 # default <home>/.dsh/ocr1-memory
    ocrBaseUrl: ''               # DeepSeek-OCR vLLM/OpenAI-compatible endpoint; leave empty to skip OCR read-back
    ocrApiKey: ''
    ocrModel: 'deepseek-ai/DeepSeek-OCR'
    pythonPath: 'python'
    renderScript: '<plugin dir>/scripts/render_memory.py'
    requireOcr: false            # when true, OCR unavailability raises an error directly
    useMockRenderer: false       # when true, skips Python rendering (testing only)
    autoStartOcrServer: false    # when true, automatically ensures llama-server is online after the plugin loads
    ocrServerPath: ''            # path to llama-server.exe; leave empty for the default
    ocrModelDir: ''              # DeepSeek-OCR GGUF directory; leave empty for the default
    ocrServerPort: 18080         # OCR server port
    ocrEmbeddingBaseUrl: ''      # usually the same as ocrBaseUrl (combined mode); falls back to ocrBaseUrl automatically if empty
    ocrEmbeddingApiKey: ''
    ocrEmbeddingModel: ''        # falls back to ocrModel if empty
    ocrEmbeddingTimeoutMs: 120000
    ocrEmbeddingEmptyPromptTokens: 1  # prompt_tokens baseline for an empty-text embedding
    ocrEmbeddingAutoStart: false       # only needs to be true when embeddings use a separate server
    ocrEmbeddingPort: 18084            # standalone embedding server port (unused in combined mode)
    ocrEmbeddingUbatchSize: 2048       # must be >= the visual token count of a single image (default 512 rejects large images)
    ocrEmbeddingServerPath: ''         # path to llama-server.exe used for embeddings
    ocrEmbeddingModelDir: ''           # GGUF directory used for embeddings
    ocrEmbeddingOnDemand: true         # only effective when embeddings use a separate server; in combined mode port 18080 is reused directly
    ocrEmbeddingIdleTimeoutMs: 300000  # how long (ms) the embedding server idles before auto-shutdown
    ocrEmbeddingContextSize: 2048      # embedding server context (no long generation needed; 2048 is enough)
    sharedStore: false                 # when true, memories.json is re-read before every operation, supporting multiple Agents sharing one store
    embeddingRetrieval: true           # when true, 1280-dim visual embedding similarity is the primary retrieval signal (with ocrEmbeddingBaseUrl)
    ocrMaxEntriesPerRetrieve: 5        # when text retrieval falls short of topK, the max number of memories to OCR read back (prevents large stores from stalling retrieval)

Installation

# Install from GitHub (recommended)
dsh plugin --profile web add github:DDDFXYqiming/dsh-ocr1-memory

# headless (self-test)
dsh plugin --profile headless add github:DDDFXYqiming/dsh-ocr1-memory

For local development, you can also use the repository directory directly:

dsh plugin --profile web add <this directory>

Runtime injection (no restart needed, for development):

dev_inject_plugin <this directory>

Hooking up a real DeepSeek-OCR

This plugin abstracts the OCR backend behind an OpenAI-compatible /v1/chat/completions (vLLM already supports DeepSeek-OCR).

# Example: serve DeepSeek-OCR with vLLM
python -m vllm.entrypoints.openai.api_server \
  --model deepseek-ai/DeepSeek-OCR \
  --max-model-len 16384

Then set ocrBaseUrl to http://127.0.0.1:8000/v1 and retrieval will actually go through the optical read-back path.

Local AMD route (llama.cpp):

# One-shot startup / ensure the DeepSeek-OCR llama-server is online
node scripts/ensure-ocr-server.mjs 18080
# or manually
powershell -File scripts/start-ocr-server.ps1

Default backend address: http://127.0.0.1:18080/v1, with the model defaulting to deepseek-ocr-Q4_K_M.gguf (DeepSeek-OCR-Q8_0.gguf can also be set via ocrModelDir/script arguments).

Real visual embeddings (DeepSeek-OCR embeddings endpoint)

llama.cpp's /v1/embeddings supports multimodal input (prompt_string + multimodal_data). The current llama-server build provides both /v1/chat/completions and /v1/embeddings under --embeddings --pooling mean, so a single server can handle both OCR and visual embeddings:

# Option 1: use start-ocr-server.ps1 (combined mode is enabled by default)
powershell -File scripts/start-ocr-server.ps1 -Port 18080

# Option 2: manual startup (-ub must exceed the visual token count of a single image; the default 512 rejects large images)
llama-server.exe --host 127.0.0.1 --port 18080 --embeddings --pooling mean \
  -m <model_dir>\deepseek-ocr-Q4_K_M.gguf \
  --mmproj <model_dir>\mmproj-deepseek-ocr-q8_0.gguf \
  --alias deepseek-ocr -c 8192 -np 1 -n 1024 -b 2048 -ub 2048

Then set ocrEmbeddingBaseUrl to the same http://127.0.0.1:18080/v1 as ocrBaseUrl (it also falls back to ocrBaseUrl automatically if unset). The plugin stores a real 1280-dim visual embedding for every memory and reports the "direct visual token count" (a media-marker-only request, prompt_tokens - empty-text baseline).

Gaps versus the DeepSeek OCR1 paper

The current implementation is an engineering approximation of OCR1's ideas, not a complete replication of the paper's internal mechanisms:

1. DeepEncoder's internal compression is not replicated - Paper: DeepEncoder actually compresses document images into a small number of visual tokens, then hands them to DeepSeek-3B for decoding. - Current: uses llama.cpp's OCR/embeddings interfaces for optical read-back and visual vector storage; visual token counts come from interface statistics, not DeepEncoder's internal tensor outputs.

2. Locate-and-Transcribe is an engineering approximation - Paper/OCR-Memory: the model directly outputs the SoM number (Locate), then the original text is fetched (Transcribe). - Current: locating relies on text token overlap scoring + OCR evidence; the original verbatim segment is returned to avoid generative hallucination, but it is not "the model outputting a number". - The LoRA fine-tuning part — making the model output SoM numbers — is intentionally out of scope per the project goals.

3. Visual embedding is already the primary retrieval signal (engineering implementation) - visualMemory.embedding already stores real 1280-dim visual vectors; - At retrieval time, memories are first ranked by cosine similarity between the query embedding and memory embeddings, then text segment locating runs within the hit memories; - This is closer to the paper's "retrieve with visual representations" direction than pure text scoring, but still not DeepEncoder's internal compression.

4. Visual token counts are interface-level direct measurements - visualTokensDirect is obtained via a marker-only request to the embeddings endpoint; - Not DeepEncoder's internal layer-by-layer explicit outputs.

Currently running services and conditions for further progress

Services actually required right now:

- 18080: the DeepSeek-OCR combined server, providing both /v1/chat/completions (OCR read-back) and /v1/embeddings (1280-dim visual embedding / embedding retrieval).

The standalone 18084 is no longer needed; the exploratory leftovers on 18081/18082/18083 have all been shut down.
Verified boundaries for further progress:

- Getting DeepSeek-OCR to output SoM numbers directly without fine-tuning: unreliable on the current llama.cpp backend (outputs unrelated text), so the paper's original Locate requires LoRA to advance. - DeepEncoder's internal compression pipeline / internal layer-by-layer visual token counts: not obtainable through llama.cpp's public interfaces. - Multimodal embeddings depend on llama.cpp extensions; on an AMD environment without NVIDIA/vLLM there is no way to switch to the official DeepEncoder outputs.

Conditions that would enable further progress:

- If LoRA fine-tuning is allowed: the "model outputs SoM numbers" Locate capability can be added. - With an NVIDIA/vLLM environment available: DeepEncoder's internal compression, internal visual token outputs, and official multimodal embeddings can be aligned further.

Development and testing

npm run build        # node --check
npm test             # 47 tests (including complex isolation tests + real OCR/embedding + robustness, if the backend is online)
npm run test:smoke   # local end-to-end smoke test (real Python rendering + mock OCR)
node scripts/compare-memory.mjs  # compare dsh-ocr1-memory vs dsh-memory (isolated temp environments)
dsh --profile headless --dump-config   # verify the plugin tier is assembled

Test and comparison results

npm test: 47/47 passing.
Robustness: multi-Agent shared store, recovery from missing images / corrupted cache, and extra-long input boundaries all pass.
Comparison benchmark (scripts/compare-memory.mjs, isolated headless + official stock DSH):

- dsh-ocr1-memory: all of R1–R6 PASS. - dsh-memory: all of R1–R6 PASS this run; but R5 previously FAILed in manual verification (memory_archived entries were still readable via memory_read), with unstable behavior. - Conclusion: dsh-ocr1-memory never fell behind dsh-memory.

Roadmap

[ ] LoRA fine-tune the DeepSeek-OCR decoder for SoM retrieval (the OCR-Memory approach), turning retrieval into "the model outputs a number" instead of text scoring
[x] AgentOCR-style segment optical caching (hash-based segment caching to reduce rendering cost; .render-cache reuse implemented)
[x] Compression metrics and OCR text baseline calibration (ocr1_mem_metrics / ocr1_mem_calibrate)
[x] Explicit memory updates (ocr1_mem_update, conflict resolution)
[x] Automatically ensure the OCR server is online (autoStartOcrServer)
[x] Isolated benchmark against dsh-memory (R1–R6 fully re-run with the corrected script; dsh-ocr1-memory all PASS)
[x] Real DeepSeek-OCR visual embedding storage (1280-dim, visualMemory.embedding)
[x] Direct visual token measurement (embeddings endpoint marker-only request, visualMemory.visualTokensDirect)
[x] Multi-Agent shared store (sharedStore + reload + atomic save)
[x] Automatic recovery from missing images / corrupted render cache
[x] Extra-long input boundary tests
[ ] Hit-frequency-driven dynamic decay strategy
[ ] Auto-inject /context so Agents see a memory summary each turn

References

DeepSeek-OCR (OCR1): <https://arxiv.org/abs/2510.18234> · <https://github.com/deepseek-ai/DeepSeek-OCR>
OCR-Memory (method blueprint, not open-sourced): <https://arxiv.org/abs/2604.26622>
AgentOCR (engineering reference, open-sourced): <https://github.com/langfengQ/AgentOCR>

Source facts

Install

Check the source files