dsh-plugin-midscene

English | 中文


AI-driven UI automation for DeepSeek Harness (DSH), powered by Midscene. The model sees the screen, locates elements by natural-language description, and acts on real targets — a real Android device or a real Chrome browser.

One capability seam (ctx.midscene), two providers, two tools:

Each tool is a single tool with an action parameter (like str_replace_editor): the model picks an action (tap / act / input / query / assert / boolean / back) and the tool dispatches internally — no tool-surface bloat.

Requirements

• DSH (dsh CLI) with a profile
• Android: adb devices shows the device
• Web: Chrome started with --remote-debugging-port=9222 --user-data-dir=<dir>; the provider connects, never launches
• A Midscene-compatible vision model, configured through environment variables (see [Model configuration](#model-configuration))

Install

dsh plugin --profile mysetup add dsh-plugin-midscene

The bundle's default layer registers the two tools. They read ctx.midscene opportunistically, so they appear even before a provider is configured — calling one without a provider fails with a message naming the missing row.

Then add exactly one provider row to your profile's cordis.patch.yml (~/.dsh/profiles/mysetup/cordis.patch.yml; both providers cannot own ctx.midscene in the same context):

Android

- insert:
    - id: midscene-android
      name: dsh-plugin-midscene/android
      config:
        deviceId: ''            # empty: first device from getConnectedDevices()
        aiActionContext: ''     # free-form context for aiAct planning, e.g. app conventions

Web

- insert:
    - id: midscene-web
      name: dsh-plugin-midscene/web
      config:
        browserWSEndpoint: 'ws://127.0.0.1:9222/devtools/browser/<id>'
        aiActionContext: ''

Ready-to-paste provider rows live in [examples/](examples/).

Get the endpoint from http://127.0.0.1:9222/json/version → webSocketDebuggerUrl. Note the id changes every time that Chrome restarts — update the row and restart dsh.

On teardown the provider destroys its agent and then disconnect()s — never close()s — the browser: the Chrome process belongs to your deployment and keeps running.

Start:

dsh --profile mysetup          # add --port 3081 if 3080 is occupied

Install troubleshooting

dsh plugin add fails with ERR_PNPM_IGNORED_BUILDS naming sharp / @ffmpeg-installer/linux-x64: pnpm ≥ 10 blocks those transitive install scripts (pulled in by @midscene/*) until they are declared. Fix: open ~/.dsh/profiles/<name>/pnpm-workspace.yaml, set the keys pnpm printed under allowBuilds to false (the plugin works without them — flip to true only if you want the sharp/ffmpeg binaries for real-device capture), then re-run the add command. This is a one-time fix per profile.

Tool reference

android_ui and web_ui share one shape:

Cross-field rules the schema cannot express (e.g. value required for input, demand for query) are enforced in execute with named error messages. A failed assertion is a successful pass: false result — the error path is reserved for infrastructure failures (device gone, websocket refused).

Model configuration

Midscene's vision model is configured through @midscene/*'s own conventions — environment variables, not DSH's ctx.llm:

export MIDSCENE_MODEL_NAME=glm-4.6v
export MIDSCENE_MODEL_BASE_URL=https://open.bigmodel.cn/api/paas/v4/
export MIDSCENE_MODEL_API_KEY=<your key>
export MIDSCENE_MODEL_FAMILY=glm-v

(Any OpenAI-compatible multimodal endpoint works — set the matching variables.)

Design boundary: no policy, no recovery

The providers are deliberately thin transports: no retry, no precondition checks, no automatic recovery from unexpected UI state (stray popups, unwanted navigation, re-login). Callers that need that behavior build it on top — for example a constraint/harness layer that checks app state before each write action.

Known limitations

• One target per provider instance — one device or one browser per context; fan out with isolated compositions.
• No reconnect — a mid-session disconnect surfaces as a rejected call.
• Pinned SDK versions — @midscene/android / @midscene/web at exactly 1.11.0; upgrading is a deliberate version bump.
• puppeteer is a peer (web) — resolved by your deployment's pnpm; Chrome itself is supplied by the deployment, never downloaded by this plugin.

Development

git clone https://github.com/ciky20171114/dsh-plugin-midscene
cd dsh-plugin-midscene
pnpm install   # native/browser install scripts are denied by default; tests mock the SDKs
pnpm test      # wiring-only: mocked @midscene/*, puppeteer, stub seam behind the real tool registry
pnpm build     # tsc emit to lib/ (also runs as `prepare` on git installs)

Layout:

src/service.ts   MidsceneService definition — the ctx.midscene seam (7 operations)
src/android.ts   Android provider (AndroidDevice + AndroidAgent, ADB)
src/web.ts       Web provider (puppeteer.connect + PuppeteerBrowserAgent, connect-only)
src/tool.ts      android_ui + web_ui tools (one shared definition, action branching)
tests/           31 wiring tests — never a real device or browser

Install a local checkout into a profile while developing:

dsh plugin --profile dev add /path/to/dsh-plugin-midscene

Community and support

Feel free to submit feedback or bug reports through GitHub Discussions. This repository carries the dsh-plugin topic for discoverability.

License

MIT