dsh-qwen-multimodal

A DSH bundle that gives text-only main models (e.g. DeepSeek) three multimodal skills in one plugin through Qwen APIs: vision, speech-to-text, and text-to-image — with a built-in generate-then-verify quality loop.

Media never enters the main model context: visual/audio content is converted to text, and generated images are saved to local files with paths returned by the tool.

How it works

All API calls reuse the original Python scripts in skills/deepseek-vision/scripts/*.py and the .env configuration (vision/audio use the Alibaba Cloud Bailian OpenAI-compatible endpoint; image generation uses the native multimodal-generation endpoint). The plugin itself is a pure-JS Cordis bundle depending only on the host's mounted subprocess / tools services — no build step required for git installs.

Install

From GitHub

dsh plugin --profile demo add github:wuwangmao/dsh-qwen-multimodal

Local checkout / tarball

dsh plugin --profile demo add ./dsh-qwen-multimodal
# or
pnpm pack   # then
dsh plugin --profile demo add ./dsh-qwen-multimodal-0.1.0.tgz

Before first use, configure your API key: copy skills/deepseek-vision/.env.example to skills/deepseek-vision/.env and fill in VISION_API_KEY (create one in the Alibaba Cloud Bailian console; new users get free quota, college students get a ¥300 annual voucher). Vision/audio/image reuse the same key by default, or configure them separately (see .env.example).

Python

Python 3.10+ is required (the image-generation script uses int | None type-annotation syntax).

python is resolved from the system PATH by default. If it cannot be resolved, restate the plugin row in your profile's cordis.patch.yml and set config.pythonPath.

Configuration overrides

The plugin uses the bundled skill directory by default. To point it at an external directory (e.g. to reuse an existing .env and scripts, or to keep your key outside node_modules), restate the row:

- insert:
    - id: qwen-multimodal
      name: dsh-qwen-multimodal
      config:
        skillDir: 'D:/qwen-vision'
        pythonPath: 'C:/path/to/python.exe'

Usage

Once loaded, the model can call the three tools directly:

• describe_image({ images: ['screenshot.png'] }) — verbatim extraction of text/code/errors in images
• describe_image({ images: ['chart.png'], prompt: '逐字提取图中所有文字，保留原样' }) — custom prompt
• transcribe_audio({ audios: ['recording.m4a'], language: 'zh' }) — specify language for accuracy
• generate_image({ prompt: 'a cute orange cat on a windowsill watching the sunset', out_dir: './out' }) — generate and save locally
• generate_image({ prompt: '...', out_dir: './out', verify: true }) — generate, then automatically re-check the result with Qwen VL against the prompt (quality loop)

Layout

dsh-qwen-multimodal/
├── package.json              # dsh.bundle manifest
├── cordis.patch.yml          # bundle layer: inserts the plugin row
├── src/index.js              # plugin: registers the three model tools (pure JS)
├── scripts/selfcheck.mjs     # self-check: node scripts/selfcheck.mjs
└── skills/deepseek-vision/   # skill assets: SKILL.md + Python scripts + .env.example

License

MIT