dsh-llm-vision

English | 中文

   

Reliable vision + OCR for text-only models on DeepSeek Harness.

Prompt engineering that makes screenshot QA trustworthy, the same reliability engineering (preprocessing / retries / persistent cache) — plus the DSH-native experience paste-bridge, live settings card, URL input, and attachment references.

> Status: v0.2.2 on GitHub and npm. Verified end-to-end against a live > OpenAI-compatible vision endpoint (DashScope qwen3-vl-plus / qwen3.5-ocr) > in the real web GUI; 217 offline tests. v0.2.0 added free provider presets > (Zhipu GLM-4V-Flash, Gemini), multi-image batch reads, a llm_vision_check > diagnostic tool, and HEIC/HEIF support. v0.2.2 fixes the llm_vision_check > testCall probe to a 64×64 image (the old 1×1 probe was rejected by > qwen3-vl-plus's minimum-size rule) — the diagnostic now passes end to end.

Why

Text-only models (DeepSeek V4, GLM text series, …) cannot see images. This plugin registers model-facing tools backed by any OpenAI-compatible vision endpoint:

Plus the DSH-native experience:

• Paste / drag / drop images into the composer and send — the browser half rewrites the

image-bearing send into attach references the text model can resolve, and upgrades the references into inline thumbnails in the transcript.

• Live settings card (Settings → Plugins → llm-vision): endpoint, models, prompts, bounds,

retries, preprocessing, and cache — no restart needed.

• Three input kinds per call: local absolute path, http(s) URL (redirects refused), or

attachment reference.

• The image never enters the session log — only the returned text crosses into the

conversation.

Install

dsh plugin --profile web add dsh-llm-vision@0.2.2

The version is pinned on purpose: pnpm 11 holds back packages published in the last 24 hours, so a bare add dsh-llm-vision (latest) would silently install the previous release on launch day. This line is bumped with every release.

From a source checkout the same command accepts a tarball or local path (pnpm pack names the tarball after the current version — use that name):

pnpm install && pnpm build && pnpm pack   # → dsh-llm-vision-<version>.tgz
dsh plugin --profile web add ./dsh-llm-vision-<version>.tgz
# or: dsh plugin --profile web add /path/to/dsh-llm-vision   (build first — lib/ is gitignored)

The tarball ships prebuilt lib/ (both the node half and lib/client.js), so no build step runs on the installing machine.

Configure

The official web GUI's plugin-configuration page only exposes allowlisted settings namespaces — third-party namespaces are a deliberate security boundary (the harness comments call plugin-declared exposure "deferred work"). The settings card therefore shows a "namespace not exposed" note; configure through the profile patch layer instead, referencing the key from the environment (never a plaintext key in a patch file):

# ~/.dsh/profiles/web/cordis.patch.yml
- id: llm-vision
  name: 'dsh-llm-vision'
  config:
    baseURL: https://dashscope.aliyuncs.com/compatible-mode/v1
    model: qwen3-vl-plus
    ocrModel: qwen3.5-ocr
    apiKey: !!js process.env.VISION_API_KEY

# ~/.dsh/.env (or export it before launching dsh)
VISION_API_KEY=sk-...

Then restart the GUI. The card is visible in Settings → Plugins and explains this gap instead of vanishing.

#### Free presets (zero-cost routes)

Instead of naming baseURL / model / ocrModel yourself, one provider switch fills them from a built-in preset — including two permanently free vision routes (facts verified 2026-08; free policies change, so re-check the provider docs if a call stops working):

# Zhipu BigModel — free GLM-4V-Flash (best default in mainland China)
- id: llm-vision
  name: 'dsh-llm-vision'
  config:
    provider: zhipu
    apiKey: !!js process.env.ZHIPU_API_KEY   # or set apiKeyEnv: ZHIPU_API_KEY

# ~/.dsh/.env
ZHIPU_API_KEY=your-zhipu-key   # https://open.bigmodel.cn — free tier, no card

# Google Gemini — free key from Google AI Studio (aistudio.google.com, no card)
- id: llm-vision
  name: 'dsh-llm-vision'
  config:
    provider: gemini
    apiKey: !!js process.env.GEMINI_API_KEY

# Alibaba DashScope (the default models, as a preset)
- id: llm-vision
  name: 'dsh-llm-vision'
  config:
    provider: dashscope
    apiKey: !!js process.env.DASHSCOPE_API_KEY

Explicit baseURL / model / ocrModel / apiKeyEnv always override the preset. The free presets reuse the vision model for OCR (extract_text drives it with the OCR prompt) — free tiers are rate-limited, so they suit interactive use better than batch runs.

Reliability engineering

• Auto-preprocessing — images over 1568px are scaled, oversize files re-encoded (JPEG q85,

transparent formats kept as PNG) via the macOS built-in sips; every failure silently falls back to the original image. HEIC/HEIF inputs are always re-encoded to JPEG (endpoints support HEIC unevenly), failing loudly only when sips is absent. Fixes the classic "big screenshot times out" failure.

• Retries — transient errors retry up to maxRetries with exponential backoff

(≤ 4s) under a shrinking per-attempt budget (total ≤ 2× timeout). Exhausted retries append （已重试 N 次）. Caller cancellation aborts immediately without retry.

• Persistent cache — identical image + model + prompt + preprocessing settings hit a

content-addressed cache (SHA-256 over the image bytes) at ~/.cache/dsh-llm-vision/; only the text answer is stored, never image bytes; TTL 30 days, 500 entries, atomic writes, 0600/0700 permissions. Note: OCR results of sensitive documents are stored in plain text there — set cacheEnabled to false when that matters.

Security model

• The vision request and any image download refuse HTTP redirects (redirect: 'error') — bearer

credentials and image bytes never leave the configured endpoint.

• Request bodies carry the base64 image but never the key; parsed credentials are never logged.
• Only http(s) URLs and local paths are accepted; all other schemes are rejected.
• Attach uploads are validated (strict base64, magic bytes, byte bound) before the attachment

store persists them; only the reference JSON (text) enters the session.

• Response bodies are capped (maxOutputTokens × 8 + 64 KiB) before parsing; error excerpts are

bounded to 200 chars.

• Calling the tools sends the image bytes to the configured endpoint — only hand the model images

you are comfortable leaving your machine.

Testing status

216 offline unit/integration tests (vitest, mock HTTP server, tmp-dir cache) plus a strict typecheck and CI on every push. Verified end-to-end in the real DSH web GUI against a live OpenAI-compatible vision endpoint: describe_image reads a real image (DashScope qwen3-vl-plus), extract_text OCR returns real transcription (qwen3.5-ocr), the attach upload/readback routes work through the live web server, and the settings card renders in the plugin-configuration page. Known cosmetic gap: the card cannot edit values in the GUI (official settings allowlist), so configuration lives in the patch layer — see [Configure](#configure).

Known limitations

• Attachment/upload channel: PNG / JPEG / GIF / WebP only (the official attachment store's

type set). HEIC/HEIF images are read directly by the tools from local paths and URLs — preprocessing re-encodes them to JPEG on macOS — but pasting a HEIC file into the GUI is rejected with a hint; convert it or pass the path instead. On Windows/Linux (no sips) a HEIC/HEIF read fails with a clear message.

• Preprocessing relies on macOS sips (zero dependencies); on Windows/Linux the

plugin degrades silently and sends the original bytes — never an error, but oversized images are then likelier to time out. The bound gates loading (maxBytes), so a too-small bound is a clean rejection, never a crash.

Development

pnpm typecheck   # tsc -b + vitest program
pnpm test        # vitest run (216 tests, fully offline)
pnpm build       # tsc -b && tsdown → lib/ + lib/client.js
pnpm watch       # tsdown --watch

License & attribution

Apache-2.0. Built on: deepseek-harness packages/vision/tool-describe-image (whitelonng/dsh-plugin-describe-image, MIT), the dsh-web-ui plugin family (Apache-2.0), and the llm_vision design (MIT). See [NOTICE](NOTICE) and [AGENTS.md](AGENTS.md).