dsh-tool-eyes
Local vision "eyes" for DeepSeek Harness (DSH).
Give your text-only agent eyes with two model-facing tools:
screen— capture the screen (or describe an existing image file) through a
local OpenAI-compatible vision endpoint (llama.cpp with --mmproj, LM Studio, Ollama, ...) and return the vision model's text description.
ocr— extract ALL text verbatim with the **Windows built-in OCR
engine**: zero model, zero GPU, zero cloud, milliseconds.
screen = screenshot / image -> local VLM -> text description (understanding)
ocr = screenshot / image -> Windows OCR -> verbatim text (extraction)Why
DeepSeek's chat-completions line is text-only. Instead of switching your whole conversation to a vision model, keep the text brain and add eyes as tools:
- Private by default — point
screenat a local endpoint and images never
leave your machine.
- Cheap — a 0.8B–4B local VLM is plenty for describing screens;
ocrcosts
nothing at all.
- Honest by default — the
screenprompt tells the VLM to *describe only
what is visible* and never guess app/game/character names unless confirmed by on-screen text (this measurably cuts small-model name hallucination).
Requirements
- Windows 10/11 (the
ocrtool uses WinRT OCR;screenuses .NET for capture) - Node.js >= 22.19, DeepSeek Harness >= 0.1.0-rc.6
screenadditionally needs any OpenAI-compatible VLM endpoint, e.g.:
- llama.cpp: llama-server -m model.gguf --mmproj mmproj.gguf --port 1235 - LM Studio (loaded vision model), Ollama, or any OpenAI-compatible gateway
Install
> This package is published on GitHub only (not on npm).
dsh plugin --profile web add https://github.com/go-farther-and-farther/dsh-tool-eyesThen restart dsh web. The screen and ocr tools appear in the agent's toolkit automatically.
Manual install (offline / from source)
Copy this package into the profile's node_modules, then register it in $DSH_HOME/profiles/<profile>/cordis.patch.yml:
- insert:
- id: tool-eyes
name: 'dsh-tool-eyes'
config:
baseUrl: http://127.0.0.1:1235/v1
model: ''
timeoutMs: 180000Configuration
Plugin config (all optional):
| key | default | meaning |
|---|---|---|
baseUrl | http://127.0.0.1:1235/v1 | OpenAI-compatible endpoint for screen |
model | '' | model id to send; empty lets the server decide (llama.cpp serves one model) |
timeoutMs | 180000 | hard cap for one capture call |
captureScript | bundled capture.ps1 | override path to an alternate capture script |
Override in your profile's cordis.patch.yml (id-targeted):
- id: tool-eyes
name: 'dsh-tool-eyes'
config:
baseUrl: http://127.0.0.1:1235/v1
model: qwen3.5-4b
timeoutMs: 120000Usage
In a conversation, the agent can now:
screen— "what is on my screen?", "describe this image file", with an
optional prompt to focus on a region or detail.
ocr— "read all the text on screen", "transcribe this error dialog".
Both accept an optional image path; without it they capture the screen. The bundled PowerShell scripts can also be run standalone:
powershell -NoProfile -ExecutionPolicy Bypass -File lib\capture.ps1 -Prompt "..." -BaseUrl http://127.0.0.1:1235/v1
powershell -NoProfile -ExecutionPolicy Bypass -File lib\ocr.ps1 -Image C:\path\x.pngPrivacy
ocris fully local (WinRT OCR, no network).screensends the captured image to the configuredbaseUrl. Point it at a
local endpoint (llama.cpp / LM Studio / Ollama) to keep images on your machine.
Related
- dsh-vision-proxy — automatic
transcription of attached images in the chat input (Chatbox-style), so you don't need to give file paths. Pairs well with this plugin.
Development
npm test # node --test tests/License
MIT