dsh-computer-use-win

中文 | English

Windows computer-use bundle for DeepSeek Harness (dsh): screenshots, window and UI Automation element discovery, OCR/visual observation for text-only models, mouse/keyboard control, and direct Windows Settings launch.

With DSH 0.1.1-rc.2 or newer, the built-in deepseek-v4-flash-vision-exp route receives computer_screenshot captures as native image content. DSH normalizes each screenshot to the route's pixel and byte budget, then uses the DeepSeek Files API with an inline fallback. Text-only routes keep the OCR-compatible saved-file flow.

The macOS-first sibling is Anionex/dsh-computer-use; this plugin fills the Windows gap.

What It Adds

Input-injecting tools are approval-gated by default. With dsh's Full access preset (approval=never plus sandbox=danger-full-access), the plugin treats the action as already authorized and does not call the approval prompt that would otherwise deterministically reject. Confined modes still fail closed when the approval policy is never.

How It Works

The plugin has no native binary dependency. Every capability runs through a bounded PowerShell child process:

• Capture uses System.Drawing.CopyFromScreen.
• Window discovery uses Win32 EnumWindows, GetWindowTextW, GetWindowRect, and foreground-window APIs.
• Element discovery uses Windows UI Automation, which lets text-only models locate native/accessibility-aware controls without seeing screenshots.
• OCR uses Windows.Media.Ocr for text and coordinate extraction from screenshots or window captures when UI Automation cannot see inside custom-rendered apps.
• Visual observation combines OCR with coarse layout regions, suggested actions, media-control state, and simple icon/button heuristics so text-only models get a basic UI map instead of a raw image path.
• Pointer input uses SetCursorPos and mouse_event.
• Text input uses clipboard plus System.Windows.Forms.SendKeys as the reliable path under Sangfor/VDI filtering, with SendInput fallback.
• All PowerShell processes opt into DPI awareness and UTF-8 output so UIA rectangles, screenshots, and click coordinates use one physical-pixel coordinate space.

Install

Requires the dsh CLI on Windows.

Using Agent Teams through Oh My DSH? Install and connect this plugin in one step:

oh-my-dsh setup --computer-use

Install the prebuilt release (recommended):

dsh plugin --profile web add https://github.com/secretxuan/dsh-computer-use-win/releases/download/v0.7.0/dsh-computer-use-win-0.7.0.tgz

Then verify the layer and boot DSH:

dsh --profile web --dump-config | Select-String computer-use
dsh --profile web

The unscoped package named dsh-computer-use-win on npm belongs to a different repository. Use this project's GitHub Release URL to install this implementation.

To develop from a checkout:

git clone https://github.com/secretxuan/dsh-computer-use-win.git
cd dsh-computer-use-win
pnpm install
dsh plugin --profile web add .

Or install a packed tarball:

pnpm pack
dsh plugin --profile web add ./dsh-computer-use-win-0.7.0.tgz

Configuration

Patch the computer-use-win row in your profile's cordis.patch.yml:

- id: computer-use-win
  config:
    requireApproval: true
    maxScreenshotBytes: 8388608

Set requireApproval: false only for unattended tests or disposable environments.

Known Limitations

• Windows input injection is global: clicks go to whatever is under the coordinates.
• Elevated administrator windows may ignore input from a non-elevated plugin process.
• Win-key combinations fall back to SendInput, which some VDI products filter. Prefer semantic tools such as computer_open_settings where possible.
• UI Automation coverage depends on the target application exposing useful accessibility metadata. CEF/Chromium/Electron/custom-rendered windows may expose only a shell pane; use computer_observe_window, computer_observe_media_controls, computer_search_in_window, computer_ocr_window, computer_locate_text, or computer_click_text for those apps.
• Text-only DeepSeek routes cannot inspect screenshot pixels directly. computer_screenshot returns a saved PNG path on those routes; use observation/OCR tools to turn that path/window into text, UI regions, visual candidates, and click coordinates.
• Visual observation is not a vision model. Icon/button candidates are heuristics with confidence and evidence; verify after clicking.

Recommended Text-Only Flow

For text-only models, prefer structured observation over coordinate guessing:

computer_list_windows
-> computer_list_elements(window: "target app")
-> if diagnostics say UIA is limited, computer_observe_window(window: "target app")
-> use suggestedActions directly when they match the task
-> computer_observe_media_controls for play/pause state instead of custom pixel scripts
-> computer_search_in_window(window: "target app", query: "...")
-> computer_locate_text / computer_click_text for text targets
-> computer_click with a high-confidence visual candidate for icon-only targets
-> computer_type_text / computer_press_key
-> verify with computer_list_windows, computer_list_elements, computer_observe_window, or OCR

Recommended Vision Flow

Select DeepSeek-V4-Flash-Vision-Exp in DSH, then let the model inspect the real screenshot rather than an OCR summary:

computer_screenshot(window: "target app")
-> inspect the returned image
-> computer_click / computer_type_text / computer_press_key
-> computer_screenshot(window: "target app") to verify

Example for NetEase Cloud Music, which is CEF/custom-rendered on many systems:

computer_list_windows
computer_list_elements(window: "cloudmusic")
# If the result says UIA is limited:
computer_observe_window(window: "cloudmusic")
computer_observe_media_controls(window: "cloudmusic")
# If the media state is playing and the task asks to pause first, click the returned toggle.
computer_search_in_window(window: "cloudmusic", query: "想你的夜")

Discovery

DeepSeek Harness currently documents out-of-tree plugin installation through profile dependencies and recommends adding the dsh-plugin GitHub topic for discoverability. There is no separate official plugin upload endpoint in the public dsh docs.

License

MIT