deepseekeyes DeepSeek Plugin

<h1 align="center">DeepSeekEyes</h1>

Give DeepSeek sight without leaving the conversation.

An auditable vision, MCP and cross-platform Computer Use runtime for <a href="https://github.com/deepseek-ai/deepseek-harness">DeepSeek Harness</a>.

<a href="README.zh-CN.md">简体中文</a> · <a href="#see-it-in-action">Live screenshots</a> · <a href="#quick-start">Quick start</a> · <a href="#how-it-works">How it works</a> · <a href="#computer-use">Computer Use</a> · <a href="#mcp-application-layer">MCP applications</a> · <a href="#token-accounting">Token accounting</a> · <a href="https://x.com/lucars2026">X / @lucars2026</a>

DeepSeek's strongest text models can reason about code, documents and interfaces, but they do not consume image pixels. DeepSeekEyes is the DSH runtime that makes those pixels auditable: it selects and health-checks visual routes, validates every nested evidence field, binds evidence to original bytes, records failover, and keeps DeepSeek as the reasoning model.

No window switching. No manual transcription. No lossy screenshot relay.

This is not another captioning window. It is the DSH auditable vision, Computer Use and MCP application runtime for image evidence, structured app calls, Browser automation and native Windows/macOS control.

See it in action

These are real DeepSeek Harness captures, not product mockups. The captures show the image and Browser loops; the MCP tool loop is documented separately below:

Image understanding: paste an image → the configured multimodal model reads the original pixels → DeepSeek receives validated evidence and answers in the same task.
Browser control: ask DeepSeek to open a page → Browser Computer Use observes, opens, scrolls and clicks → every action returns a fresh state so DeepSeek can verify the result or recover from a missing target.
Structured app calls: enable an MCP server and select only the required tools → DeepSeek calls the application in the background → DeepSeekEyes bounds, hashes and audits the result → DeepSeek verifies the requested outcome from returned or read-back evidence.

<table> <tr> <td width="50%" valign="top"> One visible DeepSeekEyes route The model picker exposes the DeepSeek final-answer model and its multimodal “Eyes” model as one selectable route. <img src="assets/screenshots/model-picker-vision-route.png" width="100%" alt="DeepSeek Harness model picker showing a DeepSeekEyes route with a DeepSeek final model and a multimodal Eyes model" /> </td> <td width="50%" valign="top"> Harness-native visual routing Select both Provider/model pairs, inspect the live route, enable automatic capability detection, randomized pixel probing, health checks and failover. <img src="assets/screenshots/plugin-routing-settings.png" width="100%" alt="DeepSeekEyes settings card configuring final-answer and background-vision providers and models" /> </td> </tr> <tr> <td width="50%" valign="top"> Understand a pasted screenshot The image stays in the current task while DeepSeek returns a structured description of layout, navigation and visible content. <img src="assets/screenshots/image-understanding.png" width="100%" alt="DeepSeek describing the contents and layout of a pasted website screenshot in the same conversation" /> </td> <td width="50%" valign="top"> Control and verify a browser The agent opens the site, inspects the live page, scrolls, follows the correct navigation path, clicks Login and verifies the resulting authentication page. <img src="assets/screenshots/browser-computer-use.png" width="100%" alt="DeepSeek using Browser Computer Use open scroll and click actions and verifying the destination page" /> </td> </tr> </table>

Why DeepSeekEyes

Requirement	What DeepSeekEyes does
:--	:--
One conversation	Image → vision evidence → DeepSeek reasoning → optional visual follow-up all happen inside the current Harness task.
Native vision is not charged twice	On DSH rc.8+, an upstream model that explicitly declares image input receives the original `ImageBlock` directly. DeepSeekEyes skips its secondary vision route and records a native-bypass turn.
Original pixels stay authoritative	User images are not resized, converted or recompressed. Every reread references the original content-addressed attachment.
The models can communicate	DeepSeek can request a precise region or detail instead of depending on one oversized first description.
No surprise text overhead	With optional automation and MCP disabled—the default—pure-text turns keep the direct model path with no visual call or tool schema. MCP schema/result estimates become visible when tools are explicitly exposed.
The eye is verified	Static image-capability metadata is followed by an optional randomized 3×3 pixel probe. A text-only model cannot silently pose as the eye.
Routes fail over visibly	Ordered visual routes, health TTL, circuit cooldown and bounded attempts are persisted without prompt/image contents.
Evidence is a contract	One public JSON Schema drives strict Ajv validation; bounded local canonicalization repairs only known structure/scalar formats and audits every change.
Automation is built in	Browser Computer Use plus native Windows/macOS desktop control can observe, act, verify and preserve evidence.
Structured hands are built in	The MCP control center connects stdio or Streamable HTTP servers while exposing only an explicit tool allowlist.
Usage is visible	The native settings card separates exact Provider usage, estimated bridge input and normal final-answer usage.

Quick start

1. Install, upgrade or diagnose

npx -y @dttxorg/deepseekeyes@latest install
npx -y @dttxorg/deepseekeyes@latest upgrade
npx -y @dttxorg/deepseekeyes@latest doctor

These commands work in macOS/Linux shells and Windows PowerShell. Use --profile NAME when the DSH profile is not web. Restart dsh web once after installation or upgrade.

2. Configure entirely in Harness

1. Open Settings → Models and add the text Provider/model and multimodal Provider/model you already use. 2. Open Settings → Plugins → DeepSeekEyes. 3. Select: - Final answer Provider + model — the DeepSeek model that reasons and replies; - Background vision Provider + model — the multimodal model that reads pixels. 4. Keep the randomized pixel probe enabled for the first real image. 5. Save, then select the DeepSeekEyes model entry in the conversation model picker.

Custom OpenAI-compatible gateways can be declared image-capable from the same card; the plugin writes the exact Harness defaultInput: [text, image] setting without replacing sibling Provider fields.

3. Paste an image

Ask normally:

> Read this screenshot, identify the failure, and tell me the next action.

DeepSeekEyes automatically reads the new image, gives DeepSeek structured evidence, and preserves the original for later targeted questions.

If the selected upstream model already declares inputModalities: [text, image], DeepSeekEyes automatically uses native vision passthrough instead: the current original ImageBlock is sent directly to that model, no background vision call or evidence prompt is created, and the model picker labels the route Native Vision. After the response, only the future model-facing session surface is replaced with a bounded SHA-256 attachment pointer; the append-only event and original attachment bytes remain intact. The usage panel exposes Native vision bypass turns, while vision-model tokens and estimated bridge input stay zero for that path.

How it works

flowchart LR
    A["Original image attachment"] --> B["DeepSeekEyes bridge"]
    B --> C["Ordered routes + health check"]
    C --> C2["Verified multimodal model"]
    C2 --> D["Strict schema-valid, hash-bound evidence"]
    D --> E["DeepSeek final-answer model"]
    E -->|needs one more detail| F["Precise visual question"]
    F --> C
    E --> G["Answer in the same conversation"]
    B -. preserves .-> H["Original bytes + append-only event"]

The first read is deliberately not the end of the visual conversation. DeepSeek may emit a bounded private clarification request naming the image SHA-256, one exact question and an optional normalized region. The eye rereads the original pixels and returns targeted evidence; DeepSeek then continues reasoning.

Historical images are compacted into bounded SHA-256 pointers. They cause no automatic reread, but the session-scoped deepseekeyes_look tool can recover one preserved original on demand—even after switching to a native text-only model.

Capability matrix

Capability	Status	Notes
:--	:--:	:--
Native pasted-image bridge	✅	Original Harness attachment stays in the append-only session log.
DeepSeek ↔ vision clarification	✅	Bounded, precise questions against the same original image.
Vision-model capability probe	✅	Metadata gate plus randomized pixel test.
Canonical evidence JSON Schema	✅	One source drives prompts and rejects invalid nested fields.
Route health and failover audit	✅	Priority, health TTL, circuit cooldown and bounded attempts.
Custom multimodal gateways	✅	OpenAI-compatible routes can be declared from the GUI.
Browser Computer Use	✅	Open, observe, click, type, select, wait, assert, report and close.
Windows desktop Computer Use	✅	Window capture + UI Automation elements/actions + user32 input.
macOS desktop Computer Use	✅	Window capture + Accessibility elements/actions + CoreGraphics input.
MCP Tools, Resources and Prompts	✅	Official DSH Tools client plus an opt-in Content plane for stdio/Streamable HTTP Resources and Prompts. All three capabilities have independent switches and default-empty allowlists; results share bounded previews, image admission, audit and Token accounting.
Lossless oversized screenshots	✅	Recompressed without pixel changes, then tiled only when the Host's 5 MB limit requires it.
Local Token accounting	✅	Exact Provider usage plus clearly labelled bridge estimates.
Public visual eval	✅	Screenshot, dense text, chart, UI and prompt-injection cases with accuracy/latency/Token output.
Pure-text isolation	✅	No visual call, screenshot or Computer Use prompt when none is needed.

Computer Use

Both automation modes are off by default and are enabled independently from Settings → Plugins → DeepSeekEyes.

The control cycle follows the same core shape as the official OpenAI Computer use loop: observe the current UI, execute a typed action, capture the resulting state, and continue. DeepSeekEyes implements that cycle as auditable DSH tools and additionally exposes native accessibility elements when the operating system provides them.

Browser Computer Use

The Playwright-powered browser loop returns a fresh screenshot and semantic element references after every action. Mutations require the latest stateId, stale actions are rejected, and an assertion/report loop turns the same feature into an automatic test runner.

Supported operations include navigation, observation, click, type, select, check, keyboard input, wait, visual assertions, evidence reports and session close.

Windows / macOS Desktop Computer Use

The native computer tool can:

discover the desktop, then observe only the target window to reduce irrelevant pixels;
return stable windowRef and elementRef identities, semantic roles, names, values, bounds and available actions;
move, click and drag the pointer;
click or invoke semantic elements, assign control values, type Unicode text and send keyboard shortcuts;
scroll, wait, launch and focus applications;
move, resize and close windows;
return a screenshot/window/element stateDelta after every step;
preserve a fresh lossless PNG after every step while avoiding a visual-model call when semantic/action evidence is sufficient;
run native element/window/screen assertions, fall back to visual assertions for pixel-only facts, and save v2 evidence reports.

launch is stateless: it can run before observe, and macOS accepts a display name, a renamed alias resolvable by Launch Services, a bundle ID, or a full .app path. Focus by application/title is also stateless. Mutations based on pixels or refs remain bound to the newest screenshot state; read-only observe may reuse the current windowRef without repeating stateId.

Since 0.5.8, desktop text entry is target-bound instead of trusting whichever control happens to own keyboard focus. The vision model grounds pixel-only controls in the exact delivered screenshot, DeepSeek supplies the plan and text, and the native runtime performs one guarded transaction: focus the intended window → click/focus the intended control → verify the foreground window/modal state → enter text → capture the result. type therefore requires either elementRef or complete x/y coordinates; coordinate input also binds to windowRef or the latest window-scoped observation. A targetless call is rejected before mutation unless allowFocusedTarget: true explicitly opts into the compatibility path.

TARGET_FOCUS_MISMATCH, DESKTOP_MODAL_TARGET_BLOCKED, DESKTOP_COORDINATE_SPACE_MISMATCH and DESKTOP_TYPE_COORDINATE_OUTSIDE_WINDOW all mean that text was not sent. Observe again, handle the modal or reground the control in the new screenshot, then retry with the new stateId. On Windows the helper uses atomic focus/click plus SendInput; on macOS semantic text uses Accessibility selected-text insertion, while coordinate-only Unicode input uses a full pasteboard snapshot/restore transaction.

Every action captures and preserves another lossless PNG. Before attachment admission, DeepSeekEyes reads the active Harness byte, per-side dimension, decoded-pixel, image-count and aggregate-byte limits and splits only where required. A compressible 5K/ultra-wide screenshot is therefore delivered as coordinate-labelled lossless PNG tiles even when its file is below 5 MB; it is never downscaled or converted to JPEG. The default desktopVisualMode: auto routes complete semantic observations and successful mutations directly to the final text model, so those steps make zero visual-model calls. Sparse/disabled accessibility states still receive pixels automatically for observe, launch and wait; the model can request exact current pixels on any call with includeScreenshot: true. always retains full per-step visual auditing, while manual delivers pixels only on explicit requests. Omitting pixels from a model turn never deletes or recompresses the stored screenshot.

A known target remains window-scoped; an explicit application/title always overrides the previous capture. On macOS, the runtime prefers the focused/main usable window over tiny auxiliary dialogs and walks Accessibility children under both the configured element bound and a helper-time budget, avoiding an unbounded Electron tree scan. semanticStatus reports availability, truncation/limit reason and elapsed semantic time. timings reports native round-trip, semantic collection, screenshot processing and total tool time; visualDelivery explains whether vision was invoked or bypassed. Coordinates are relative to a delivered image and are mapped back to native desktop coordinates. Native Desktop Computer Use is implemented for Windows and macOS; Browser Computer Use remains available wherever the configured Chromium runtime is available.

On Windows, the native helper consumes and emits UTF-8 JSON under Windows PowerShell 5.1 and converts screenshot-relative coordinates through scalar screen origins before calling user32. Window-scoped clicks therefore honor negative/multi-monitor origins without the PowerShell System.Object[] / op_Addition failure. Cross-platform CI parses the PowerShell helper and executes the real Windows coordinate path rather than only testing JavaScript simulation.

If every bounded visual route fails for a computer screenshot, the original PNG, hash and route attempts stay preserved and DeepSeek continues from the adjacent native state (actionResult, windows, accessibility elements and stateDelta). The fallback explicitly states that pixels were not decoded. Pasted user images and explicit pixel-dependent reads remain strict and still fail when no validated evidence exists.

Computer Use model calls are isolated from unrelated long-task history by a default 32,768-token automation context budget. Only the model-facing copy is bounded: the newest direct user instruction, atomic tool-call/result tail, full DSH task, screenshots and reports remain preserved. A second guard stops after 32 final-model calls for one user instruction. Both limits accept custom values and explicit 0 unlimited mode. Ordinary text and non-automation image turns never enter this guard.

MCP application layer

DeepSeekEyes 0.8 builds on the complete 0.7 MCP product layer around DSH's official @deepseek-ai/dsh-mcp-client, matching @deepseek-ai/dsh-tools and the protocol SDK already owned by that Host client. Tools continue through the official DSH client for ordinary servers; OAuth-enabled Streamable HTTP servers use the same Host-managed SDK with a dedicated OAuth transport adapter. The separately opt-in Content plane uses the same SDK for Resources and Prompts, without bundling a second protocol runtime. Every entry is resolved from DSH's managed $DSH_HOME/profiles/node_modules fallback and canonicalized to the Host installation, so a profile-local shadow cannot split Cordis or tool-scheduler identity. Configure everything under the default-collapsed MCP apps and tools section—no manual cordis.patch.yml entry is required.

Connect local stdio servers or remote Streamable HTTP endpoints. Remote endpoints must use https://; http:// is accepted only for an

…

Source facts

Install

Check the source files