deepseekplugins

DeepSeek Harness plugin

dsh_omnivision

OmniVision for DeepSeek Harness: an OmniParser-powered GUI agent plugin — screen capture, element recognition, click/type automation and a browser vision dock with recognition history, diffing and

Memory
Jump to install

Source facts

Repository
xiaozhengdeng/dsh_omnivision
Latest update
Aug 16, 2026
Category
Memory
GitHub stars
1
Format
plugin
Catalog evidence
Upstream dsh.bundle evidence
Evidence path
package.json#dsh.bundle
Checked against
0.1.0-rc.8
Upstream check date
2026-08-20

This evidence comes from the upstream catalog. This site has not installed, run, or security-reviewed the plugin.

Install

Start with a prompt that asks an agent to review the GitHub repository and source. Switch to the command if you want to install it yourself.

Copy this prompt into DSH, Codex, or another agent and ask it to review the GitHub repository and source first.

Do not install or run any commands yet. Read this plugin's GitHub repository, README, and relevant source code. Then answer the questions below clearly and directly so I can decide whether it fits my needs:

1. What is this plugin, and what problem does it solve?
2. Who is it for, and what are its typical use cases?
3. How is it used after installation? Include one minimal example.
4. What known limitations or privacy, security, compatibility, or maintenance risks does it have?
5. Give a clear recommendation: recommend, conditionally recommend, or do not recommend, with reasons.

Distinguish statements documented by the repository, inferences from source code, and unknowns. If evidence is insufficient, say so explicitly. Do not guess or simply repeat the README.

GitHub: https://github.com/xiaozhengdeng/dsh_omnivision
Plugin: dsh_omnivision
Author: xiaozhengdeng

Check the source files

Read the README and other files from this plugin directory before installing.

File explorer3 files
README.mdSource · read only

<div align="center">

dsh_omnivision

OmniVision — OmniParser-powered GUI agent plugin for DeepSeek Harness

English | [中文](#中文)

</div>

---

English

OmniVision is an OmniParser-powered GUI agent plugin for DeepSeek Harness. It turns the desktop or any image into structured elements (text + icons with pixel coordinates), so the model can see and operate the screen without multimodal vision.

Screenshots

Vision view (智能识别)Recognition history (识别记录)
![Vision view](assets/dock-vision.png)![Recognition history](assets/dock-history.png)

Features

FeatureDescription
🖥 Screen recognitionCapture the desktop (or parse an image) and get interactive elements with pixel coordinates via OmniParser
🖱 Desktop automationClick, double-click, right-click, drag, type (unicode-safe), key, hotkey, scroll — by element id or raw x/y
👁 Live vision dockReal-time recognition view with SOM-annotated overlay, hover highlight, click-to-zoom
🕘 Recognition historyThumbnails per capture, diff vs latest (added / removed / moved)
📋 One-click summarySends the recognized elements to the session for the model to summarize
🖼 File parsingParse local images from the dock, bypassing model multimodal limits
📊 Call logTracks every gui_* tool call and action for inspection

Tools

The plugin registers these gui_* tools into the shared tools registry:

  • gui_capture — capture the desktop screen and run OmniParser to extract interactive elements (text + icons with pixel coordinates), refreshing the shared vision state and saving the SOM-annotated overlay
  • gui_act — perform a real mouse/keyboard action on the desktop: click / double-click / right-click / move / type / press / hotkey / scroll / drag
  • gui_find — search the elements from the last capture by text or type
  • gui_state — show the current vision state without parsing
  • gui_verify — re-capture and check whether a text is present or absent, retrying until it settles
  • gui_task — execute a scripted multi-step UI plan with re-parse between steps and optional assertions
  • gui_open_app — launch an installed desktop app by name via its Windows AUMID
  • gui_parse_image — parse a third-party image (from a conversation attachment) into the shared vision state

Installation

dsh plugin --profile web add dsh_omnivision
# or a local path:
dsh plugin --profile web add G:\deepseek\plugins\dsh_omnivision

Restart the web process after installing. The plugin loads as a profile bundle layer:

  • Host half registers the gui_* tools into the shared tools registry and serves the browser half over an HTTP route (/dsh-omnivision/<method>)
  • Client half mounts the OmniVision dock into shell.overlay and a toggle into the sidebar footer

Prerequisites

  • Windows with an OmniParser FastAPI server on http://127.0.0.1:8000 (e.g. via G:\omni\OmniParser\start_server.bat)
  • The Python venv needs pyautogui for screenshots and input automation

Usage

Call the gui_* tools from the model side; the OmniVision dock in the browser provides the visual surface:

ControlDescription
智能识别Live element list + SOM-annotated image (hover to highlight, click to zoom)
🖼 解析图片Pick a local image file to parse (bypasses model multimodal limits)
📋 总结Send the current recognition info to the session for the model to summarize
识别记录History with thumbnails + diff vs latest (added / removed / moved)
调用记录Recent gui_* tool calls and action log
探测Check OmniParser server connectivity

Development

pnpm build      # tsc (host) + tsc (client) + tsdown (client bundle)
pnpm typecheck

License

MIT

---

中文

OmniVision 是 DeepSeek Harness 的 OmniParser 驱动的 GUI 智能体插件。它把桌面或任意图片变成结构化元素（文本 + 图标 + 像素坐标），让模型无需多模态能力就能「看见」并操作屏幕。

界面截图

智能识别视图识别记录视图
![智能识别](assets/dock-vision.png)![识别记录](assets/dock-history.png)

功能

功能说明
🖥 屏幕识别截取桌面（或解析图片），经 OmniParser 提取带像素坐标的可交互元素
🖱 桌面自动化点击 / 双击 / 右键 / 拖拽 / 输入（Unicode 安全）/ 按键 / 热键 / 滚轮，按元素 id 或原始坐标操作
👁 实时识别 Dock实时识别视图 + SOM 标注图（悬停高亮、点击放大）
🕘 识别记录每次识别的缩略图 + 与最新对比（新增 / 移除 / 位移）
📋 一键总结把当前识别信息发到会话，由模型按提示词总结
🖼 图片解析从 Dock 选择本地图片解析，绕开模型多模态限制
📊 调用记录记录每次 gui_* 工具调用与动作，便于排查

工具

插件向共享工具注册表注册以下 gui_* 工具：

  • gui_capture — 以原生分辨率截取桌面，运行 OmniParser 提取可交互元素（文本 + 图标 + 像素坐标），刷新共享视觉状态并保存 SOM 标注图
  • gui_act — 在桌面上执行真实鼠标/键盘操作：点击 / 双击 / 右键 / 移动 / 输入 / 按键 / 热键 / 滚轮 / 拖拽
  • gui_find — 按文本或类型搜索上一次识别的元素
  • gui_state — 查看当前视觉状态（不重新解析）
  • gui_verify — 重新截屏解析，反复确认某文本出现或消失
  • gui_task — 按脚本执行多步 UI 计划，步骤间重新解析并支持断言
  • gui_open_app — 通过 Windows AUMID 按名称启动已安装的桌面应用
  • gui_parse_image — 解析会话中的第三方图片（附件）进入共享视觉状态

安装

dsh plugin --profile web add dsh_omnivision
# 或本地路径：
dsh plugin --profile web add G:\deepseek\plugins\dsh_omnivision

安装后重启 web 进程。插件作为 profile bundle 层加载：

  • Host 半把 gui_* 工具注册进共享 tools 注册表，并通过 HTTP 路由（/dsh-omnivision/<method>）为浏览器半提供数据
  • Client 半把 OmniVision Dock 挂到 shell.overlay，侧栏底部挂开关按钮

前置依赖

  • Windows + 运行在 http://127.0.0.1:8000 的 OmniParser FastAPI 服务（如 G:\omni\OmniParser\start_server.bat）
  • Python venv 需安装 pyautogui（截图与输入自动化）

使用

模型侧直接调用 gui_* 工具；浏览器里的 OmniVision Dock 提供可视化操作：

控件说明
智能识别实时元素列表 + SOM 标注图（悬停高亮、点击放大）
🖼 解析图片选择本地图片文件解析（绕开模型多模态限制）
📋 总结把当前识别信息发到会话，由模型按提示词总结
识别记录历史缩略图 + 与最新对比（新增 / 移除 / 位移）
调用记录最近 gui_* 工具调用与动作日志
探测检查 OmniParser 服务连通性

开发

pnpm build      # tsc (host) + tsc (client) + tsdown (client bundle)
pnpm typecheck

许可证

MIT

Back to searchOpen GitHub repository

More related plugins

View category