dsh-multimodal — multimodal eyes and hands for DeepSeek Harness

English | 简体中文

Give DeepSeek a pair of eyes and a paintbrush: paste a screenshot or image into the conversation and the vision provider you configured first transcribes it verbatim (error messages, code, UI text preserved), then DeepSeek keeps solving your problem — all in the same turn, no extra steps. When an illustration is needed, DeepSeek automatically calls the image backend and the generated pictures appear right in the conversation.

> Blank slate by design: this plugin ships no built-in models, providers, or backends. Vision endpoints, image backends, and models are all declared by you — plug in whatever API you already have (DeepSeek, Zhipu, Aliyun, SiliconFlow, ModelScope, Xfyun, Qianfan, local Ollama, …). Nothing is preloaded, nothing is assumed.

> Compatibility: built for DeepSeek Harness 0.1.0-rc.6 (Web and headless). See [CHANGELOG](CHANGELOG.md).

Features

Security

Since 0.2.1, the plugin hardens itself against malicious or hand-edited configurations:

• API key allow-list (allowedApiKeyEnvs): only listed env/credential names may be read as an API key — a malicious config can no longer name GITHUB_TOKEN (or any other secret) as apiKeyEnv to exfiltrate it
• Trusted base URLs (trustedBaseUrls): custom vision providers must target official hosts or hosts you explicitly allow — credentials cannot be silently redirected to an attacker endpoint; explicitly listed local endpoints (e.g. http://localhost:11434) are allowed over plain http and need no key
• SSRF guard: generated-image downloads and reference_image URLs refuse loopback / private (RFC1918) / link-local addresses
• No sessionId forwarding: internal harness session ids are never sent to third-party vision APIs
• Prompt-injection markers: vision & OCR outputs are wrapped in explicit "untrusted context" markers before being handed to the text model
• Sensitive-data redaction (redactSensitive): phone numbers / 18-digit IDs / emails are masked in transcriptions (including cache hits)
• Audit log (auditLog): one line per transcription with time / image count / bytes / latency / provider
• Bounded memory: transcription cache, per-session image recall, and image-kind stats are all LRU-capped — no unbounded growth

Installation

Prerequisites: official DeepSeek Harness (0.1.0-rc.6+) installed and running, Node.js 18+.

# Option 1: install straight from the GitHub repo (requires git)
dsh plugin --profile web add https://github.com/MC5lan/dsh-multimodal

# Option 2: clone and install from a local path
git clone https://github.com/MC5lan/dsh-multimodal.git
dsh plugin --profile web add /path/to/dsh-multimodal

# Also add for headless mode when needed:
dsh plugin --profile headless add /path/to/dsh-multimodal

The repo ships prebuilt artifacts (lib/), so you can install right after cloning; to rebuild from source, see [Development](#development).

After restarting dsh web, a 「多模态 / Multimodal」 page appears in the Settings sidebar — everything is configured in one place:

• Status bar (one row): 识图 ✓/✗ · 画图 ✓/✗ — computed from real capability (transcription provider set AND registered / backend exists AND active), so it never shows "configured" when the feature is actually off
• 快速接入 (quick add): the paste-any-key auto-detect box is front and center — paste an API key, the plugin figures out the platform (key fingerprint match first, then /models probing against known endpoints; key travels one-shot, never stored), and one step later the endpoint is registered, the key env allow-listed, the credential stored, the model list pre-filled, and 识图/画图 enabled. Works with any OpenAI-compatible API. Below it: platform preset cards (Zhipu, Bailian, Xfyun, ModelScope, SiliconFlow, Qianfan, local Ollama) and a custom-endpoint form
• 已接入 (connected, open by default): platforms with editable model chips + image-backend management (switch / remove / add form)
• 高级 (advanced, collapsed): the two KeyCards (视觉转写 Key / 图像生成 Key), vision details (watch route, transcription provider, mode, four toggles), and config export/import

Everything in the common path is: paste key → done. The picker then shows deepseek-vision models (e.g. DeepSeek-V4-Flash (Vision)) — that route's sessions get the "eye".

Usage

1. Fastest path (recommended, 0.7+): Settings → Multimodal → 快速接入 → paste your API key — platform auto-detected and connected in one step (endpoint + allow-list + credential + model list; 识图/画图 enabled automatically). If the key can't be recognized, the hint says why; then pick a platform card or use a custom endpoint. 2. Manual path: tap a preset card (Zhipu / Bailian / Xfyun / ModelScope / SiliconFlow / Qianfan / local Ollama) and paste the key, or declare any OpenAI-compatible endpoint in extraProviders; optionally fill providers.deepseek.models to advertise models on the built-in DeepSeek route slot. 3. In the Web UI model picker choose a deepseek-vision model — that route's sessions get the "eye". 4. Chat normally; paste/drag an image and ask to have it read, or just say "draw me a …" to generate one.

API Keys (or write the credentials file directly)

The Settings → Multimodal → 密钥管理 page writes both keys for you. To edit the credentials file directly (path is $DSH_HOME/.credentials.yaml, default ~/.dsh/.credentials.yaml), it takes effect immediately:

VISION_API_KEY: <your-vision-endpoint-key>   # vision transcription (see 视觉转写 Key in the UI)
IMAGE_API_KEY:  <your-image-backend-key>     # image generation (see 图像生成 Key in the UI)

> These are your provider keys — any OpenAI-compatible endpoint you register > in 平台接入 accepts its key under the env name you give it (apiKeyEnv).

How it works

User image ──▶ host admission (model directory declares image modality → allowed, no popup)
           ──▶ agent/pre-step (plugin): detects images → configured vision engine transcribes
               └─▶ image blocks replaced by "【图片内容转写】…" text
           ──▶ DeepSeek API processes the transcription + your question (image bytes never reach DeepSeek)

• The deepseek-vision route is the DeepSeek API with a directory that advertises image support so the UI allows attachments — images are actually transcribed by the provider you configured, never sent to DeepSeek
• Image bytes are read through the dsh attachment service; the transcription instruction defaults to "transcribe verbatim + extract key error info" and can be customized in the dsh-multimodal: section of ~/.dsh/settings.yaml
• On transcription failure it degrades to a 【图片转写失败: 原因】 placeholder — DeepSeek keeps going instead of hanging
• Settings UI: Settings → Multimodal (settings.section slot, native look & feel like the Models page)

Configuration (optional, ~/.dsh/settings.yaml)

dsh-multimodal:
  # 0.2.1+ security: only these env/credential names may be used as API keys
  allowedApiKeyEnvs:
    - DEEPSEEK_API_KEY
    # - MY_PROVIDER_API_KEY     # add your custom provider's key env here
  # 0.2.1+ security: extra vision hosts allowed to receive credentials
  trustedBaseUrls:
    # - https://my-vision.example.com   # add your custom provider's host here
  providers:
    deepseek:
      models: []                 # optional: advertise models on the DeepSeek route slot
  vision:
    watchProvider: deepseek-vision  # sessions on this route get the "eye"
    transcribeProvider: ''          # the provider that "sees" (blank = transcription off)
    fallbackProviders: []           # 0.2.2+: providers tried in order when the primary rate-limits/fails
    transcribeMode: auto            # auto | verbatim | structured | ocr | describe | error-fix | chart-sql | design-code
    parallelImages: false           # 0.2.5+: transcribe each image in its own parallel call
    costProvider: ''                # 0.2.5+: cheap provider used for small images (cost routing)
    costModel: ''
    costMaxPixels: 1000000          # images ≤ this many pixels go to costProvider
    sceneHints: true                # 0.2.5+: append diagnosis hint to screenshot transcriptions
    customModes: {}                 # 0.2.4+: custom mode name → prompt pairs (plugin-in transcribers)
    redactSensitive: false          # 0.2.4+: mask phone/ID/email in transcription output
    auditLog: false                 # 0.2.4+: log a line per transcription (time/images/bytes/ms/provider)
  ocr:
    provider: ''                    # OCR tool engine (blank = disabled; any registered vision provider works)
    model: ''
  image:
    backends: {}                    # add backends in the UI; nothing is preloaded
    activeBackend: ''               # image backend key (blank = no image generation)
    failoverOrder: []               # 0.2.5+: backend keys tried in order when the active backend fails
    verifyChineseText: true         # 0.2.3+: check generated Chinese text for garbling via a vision model
    verifyProvider: ''              # vision provider used for the garbling check (blank = check disabled)
  transcribePrompt: ...             # custom transcription instruction
  transcribeTimeoutMs: 90000        # transcription timeout

  # Any OpenAI-compatible vision platform (just add the key; shows up in the model picker & Models page)
  extraProviders:
    xfyun-vision:                   # Xfyun MaaS
      displayName: 讯飞星辰视觉
      baseURL: https://maas-api.cn-huabei-1.xf-yun.com/v2
      apiKeyEnv: XFYUN_API_KEY
      models:
        - id: xoppaddleocrv16       # free OCR (PaddleOCR-VL-1.6)
          name: PaddleOCR-VL-1.6
    siliconflow-ocr:                # SiliconFlow (free hosted DeepSeek-OCR)
      displayName: 硅基流动 OCR
      baseURL: https://api.siliconflow.cn/v1
      apiKeyEnv: SILICONFLOW_API_KEY
      models:
        - id: deepseek-ai/DeepSeek-OCR
          name: DeepSeek-OCR
    modelscope-vision:              # ModelScope (2000 free calls/day)
      displayName: 魔搭视觉
      baseURL: https://api-inference.modelscope.cn/v1
      apiKeyEnv: MODELSCOPE_API_TOKEN
      models:
        - id: Qwen/Qwen3-VL-8B-Instruct
          name: Qwen3-VL-8B

Image backends (OpenAI-compatible /images/generations and DashScope protocols) example — add them in the UI or write them here:

dsh-multimodal:
  image:
    backends:
      z-image:                      # Aliyun Z-Image family (new multimodal-generation sync protocol)
        kind: dashscope-v2
        baseURL: https://dashscope.aliyuncs.com
        apiKeyEnv: DASHSCOPE_API_KEY
        model: z-image-turbo
        defaultSize: 1024*1024      # supports 512*512 ~ 2048*2048
      modelscope-t2i:
        kind: openai-images
        baseURL: https://api-inference.modelscope.cn/v1
        apiKeyEnv: MODELSCOPE_API_TOKEN
        model: <a lightning-badged text-to-image model in the ModelScope plaza>
    activeBackend: z-image

Custom image backends (0.2.6+)

Anything that is not OpenAI-compatible or DashScope can be plugged in with a small adapter file — no plugin code changes. Write an ES module exporting a default object with a generate() function:

dsh-multimodal:
  image:
    backends:
      my-api:
        kind: custom
        adapterFile: D:/my-adapters/my-api.mjs   # absolute or relative path
        baseURL: https://api.example.com/v1       # passed to the adapter
        apiKeyEnv: MY_API_KEY
        model: my-image-model
        defaultSize: 1024*1024
    activeBackend: my-api
    failoverOrder: [z-image]                      # fall back to another configured backend

The adapter receives { prompt, size, n, negative_prompt, reference_image, apiKey, baseURL, model, signal, fetch, log } and returns { urls: string[], b64s: string[] }. See [scripts/adapters/example-custom.mjs](scripts/adapters/example-custom.mjs) for a template. Only point adapterFile at files you trust — the adapter runs with full process privileges. Polling backends can tune pollIntervalMs / pollTimeoutMs per backend.

Local vision models (keep sensitive images on your machine)

The plugin routes transcription through any OpenAI-compatible endpoint, so a local Ollama vision model keeps image data off the network. One-click preset: Settings → Multimodal → 平台接入 → + 本地 Ollama (auto-adds http://localhost:11434/v1 to trustedBaseUrls; local endpoints need no API key).

Model matrix (pick by VRAM):

Compare providers before wiring them in:

node scripts/benchmark-vision.mjs shot.png \
  my-endpoint=https://your-vision-api.example.com/v1,your-model,YOUR_API_KEY_ENV \
  ollama=http://localhost:11434/v1,llava,SKIP

Troubleshooting

Development

npm install
npm run build      # host: tsc; client: tsc check + tsdown builds lib/client.js

License

[MIT](LICENSE)