Image Auto-Describe (dsh-image-auto-describe)

English | 中文

Give image vision to text-only DeepSeek models.

In one sentence: paste an image into the chat, and this plugin asks a vision model to read the image into text first, then hands that text to DeepSeek to answer — fully automatic, nothing extra for you to do.

What problem it solves

Many workhorse models in DeepSeek Harness (for example deepseek-v4-pro) are text-only. By default, sending an image gets refused outright:

Model "deepseek-v4-pro" does not support image input.

This plugin turns that refusal into understanding:

You paste an image and send it
      ↓
The message is intercepted: this model cannot see images
      ↓
The plugin takes over and asks a vision model
      (Qwen3-VL-32B by default, falls back to the free GLM-4V-Flash)
      ↓
The vision model writes out what it sees
      ↓
That text replaces the image, and DeepSeek answers normally

What that feels like in practice:

• Text inside screenshots — error logs, tables, chat records — is transcribed verbatim, so you can discuss it right away
• The recognition result is labelled with the vision model that produced it, and the original image stays in the conversation for comparison
• While describing, the message shows a quiet "describing image…" status at the conversation tail, then becomes a normal message
• If every vision model fails, you get the original "does not support image input" refusal — it never pretends to see, never invents

> Note: DeepSeek always sees the text description, never the image itself. Details the vision model did not transcribe are details DeepSeek cannot know — but that covers the vast majority of "here is a screenshot, please help" cases.

Install

From the plugin market, or with one command:

dsh plugin --profile web add github:oldHan2423/dsh-image-auto-describe

The package declares dsh.bundle.patch, so dsh plugin wires it into the profile automatically; the profile's own cordis.patch.yml can still override its row by id.

Host patch (required, run once)

Upstream apiproxy exposes no public image-admission extension point — the refusal is hardcoded. This repository ships an idempotent patch that turns that refusal point into the transcription admission this plugin provides. Run it once after installing (or again after a harness upgrade restores the original file):

node scripts/patch-seam.mjs           # apply (idempotent — safe to re-run)
node scripts/patch-seam.mjs --unpatch # roll back (idempotent)

The patch touches two spots: one refusal branch in dsh-host-apiproxy/lib/index.js, plus injecting the IMAGE_AUTO_DESCRIBE_SERVICE constant into its api barrel. It anchors on the 0.1.0-rc.6 source shape; on an unmatched version it reports MISS and changes nothing — it never force-edits. Every run re-checks the touched files with node --check. Restart the harness host after patching.

Requirements

• DeepSeek Harness 0.1.0-rc.6 (the patch anchor; newer versions report MISS until the anchor is updated). The plugin resolves its official @deepseek-ai/* packages as peer dependencies; the profile must compose them.
• Vision routes must already exist in the composed llm capability — for example, providers registered by the llm-pi-ai plugin. The defaults expect siliconflow (Qwen/Qwen3-VL-32B-Instruct) and zhipu (glm-4v-flash), with their API keys configured.

Configuration

The Web Plugins settings page edits these (routes picked from the configured multimodal models, never hand-typed), and the very next transcription reads the update — no restart. An edit that empties the route list is refused. On harness versions whose settings UI predates the card, the routes stay at the defaults and can be overridden in the profile patch:

- id: image-auto-describe
  config:
    candidates:
      - provider: siliconflow
        model: Qwen/Qwen3-VL-32B-Instruct
      - provider: zhipu
        model: glm-4v-flash
    maxTokens: 4096

Behavior details

• The admitted message keeps the original image blocks — marked presentationOnly, so they render in the conversation (previewable, click-to-open original) but never reach a model request — and appends the transcription prefixed with [用户在本条消息中附带了 N 张图片，以下为视觉模型（<model>）的自动识别结果]; text sent alongside the images follows under [用户同时说：]. The Web transcript collapses the transcription behind a "show image recognition result" disclosure, so the conversation shows the image alone.
• While transcription runs, the message shows at the conversation tail as a pending bubble with its image previews and a small "describing image…" status line.
• When every route fails or returns no usable text, the provider reports failure and apiproxy keeps the refusal. No silent drop.

Model experience (cost)

Each admission with images makes exactly one vision-model streaming call per tried route, bounded by maxTokens (usually one route succeeds). The vision calls are independent single-shot requests with no conversation history, so there is no KV-cache reuse between transcriptions. The session model never sees the image bytes: only the transcription text enters its context — the token cost is the text length, not the image size.

Known Limitations and Deferred Work

• The transcription replaces the original image in the model-visible message; the session model cannot answer about image details the vision model did not transcribe (the transcript still shows and previews the image).
• If no route is configured or all routes fail (no key, no quota), the prompt is refused with the original error instead of a degraded admission.
• Routes must already exist in the composed llm capability (registered by the deployment's llm provider plugins); this plugin does not register providers.

Origin

This plugin originated in the deepseek-harness working tree (commits 553ea9e and d7ca86a): the hardcoded autoDescribeImages patch in packages/host/apiproxy was reworked into this standalone provider over the apiproxy admission seam. This repository is the independent distribution of that plugin. MIT licensed.