dsh-plugin-loud-failure

 

A DeepSeek Harness plugin that turns silent tool failures into loud ones.

A tool call can exit 0 and still have failed. pandoc drops a glyph and only warns; a pipe swallows a Python traceback; a ; chain hides command not found; NumPy prints a RuntimeWarning and hands back nan. The exit code says success, the model reads success, and the mistake travels downstream. This plugin is one tools/post-execute waterfall listener: it matches the text a tool returned against a rule table and, on a hit, either blocks the result (it becomes an isError result whose content leads with the explanation and keeps the original output) or attaches a notice that lands in the model's next request. It changes no tool and no loop: it takes part in the ordered post-execute waterfall (context-only matches delegate and keep downstream decisions; blocking matches short-circuit on purpose) and unmounts cleanly.

Contents

• [Why](#why)
• [How it works](#how-it-works)
• [Install](#install)
• [Configuration](#configuration)
• [Built-in rules](#built-in-rules)
• [What the model sees](#what-the-model-sees)
• [Verification](#verification)
• [Design notes](#design-notes)
• [Model Experience](#model-experience)
• [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
• [Development](#development)
• [License](#license)

Why

Every built-in rule comes from a failure that was observed to hide behind a successful exit code:

| Observed | What the model saw | What had happened | |---|---|---| | pandoc ... --pdf-engine=xelatex prints Missing character: There is no ₂ in font ... | a PDF was written, exit 0 | the subscript in SpO₂ was silently dropped from the PDF | | python script.py \| tail -n 20 | the last 20 lines, exit 0 | a traceback scrolled by; tail supplied the exit status | | pandocc in.md -o out.pdf; ls -l out.pdf | bash: pandocc: command not found followed by a listing, exit 0 | nothing was built; ls supplied the exit status | | python calc.py prints RuntimeWarning: invalid value encountered in divide | an array, exit 0 | the array contained nan | | a PowerShell 5.1 command run with 2>&1 | NativeCommandError, $? false | the program had exited 0; PowerShell wrapped its stderr | | a Windows console printing ��� | text, exit 0 | GBK/UTF-8 code-page mismatch; the text was corrupted |

The harness already records everything the model sees and lets any plugin rewrite a tool result before the model sees it. This plugin uses exactly that seam to make the failure visible at the moment it happens, instead of three steps later.

How it works

flowchart LR
  call["tool/call"] --> pre["tools/pre-execute"] --> exec["tools/execute"] --> post["tools/post-execute<br/>(this plugin)"] --> result["tool/result"]
  post -->|"error rule matched"| block["block: isError result<br/>header + original output"]
  post -->|"context rule matched"| notice["accept + additionalContexts<br/>plugin-sourced notice"]
  post -->|"no match"| next["next()"]

1. At load, the plugin merges the built-in rules with the configured ones (a user rule with a built-in id replaces it) and compiles every pattern. An unusable table (invalid regular expression, duplicate id, stateful flag, empty message) rejects the plugin load with the offending rule id, so a broken config fails at boot, not at the first tool call. 2. On every tools/post-execute, the text blocks of the result (optionally also the JSON of a successful canonical value) are matched against the rules whose when and tools filters apply. A result counts as failed for the when filter when it is isError or its text carries a failure marker the shipped shell tools append themselves: a [exit code: N] line with N != 0, a [status: ...] trailer with a non-zero exit code, or a [sandbox: file access denied ...] notice. The model already sees those; they are not silent, so when: success rules stay quiet. 3. If any matching rule has action: error, the listener returns { kind: 'block' } with feedback that begins with a header naming every matched rule and its message, followed by the original content blocks unchanged; context the tool body deferred is carried on the decision so a block does not drop it. The registry turns that into an isError result: the canonical value is gone, so a Code Mode program cannot consume a poisoned value either. 4. Otherwise (only context matches) the listener delegates with next() and appends one UserMessage to the decision's additionalContexts. Its source is { kind: 'plugin', plugin: 'loud-failure', form: 'notice', summary }, so the Web UI shows a collapsed one-line row and the session log records exactly what the model was told. 5. No match: next(). Rules with action: off never run.

Because the listener is registered through ctx.on, it is torn down with the plugin; a config change reloads the plugin and registers a fresh listener.

Install

Requires DeepSeek Harness 0.1.0-rc.6 (@deepseek-ai/dsh-tools and @deepseek-ai/dsh-llm at 0.1.0-rc.6) and Node.js 20+. That is the only version this plugin has been tested against; the harness is a developer preview that may break compatibility, so the peer ranges are pinned and bumped per release. dsh plugin add prints peer-dependency warnings for @deepseek-ai/*: expected, because the profile resolves those packages from the harness installation rather than installing them next to the plugin.

From a release tarball (no build step, no build authorization):

dsh plugin --profile web add https://github.com/Rhymer-Lcy/dsh-plugin-loud-failure/releases/download/v0.1.1/dsh-plugin-loud-failure-0.1.1.tgz
dsh --profile web --dump-config   # shows a "# == dsh-plugin-loud-failure" layer

From GitHub at a pinned commit: a git install fetches source, so pnpm must be allowed to run this package's prepare script (it runs tsc). The first add fails and prints the exact key to allow (for a pinned commit the key includes the codeload URL); append it to the profile's pnpm-workspace.yaml, then rerun:

dsh plugin --profile web add github:Rhymer-Lcy/dsh-plugin-loud-failure#<commit-sha>

# $DSH_HOME/profiles/web/pnpm-workspace.yaml (key copied from pnpm's message)
allowBuilds:
  "dsh-plugin-loud-failure@https://codeload.github.com/Rhymer-Lcy/dsh-plugin-loud-failure/tar.gz/<commit-sha>": true

Treat that authorization as what it is: code from this repository runs on your machine at install time, outside any agent sandbox. Pin the commit.

From a source checkout, without installing:

git clone https://github.com/Rhymer-Lcy/dsh-plugin-loud-failure.git
cd dsh-plugin-loud-failure && pnpm install && pnpm run build

# overlay.yml
- insert:
    - id: loud-failure
      name: /absolute/path/to/dsh-plugin-loud-failure/lib/index.js

dsh web --patch ./overlay.yml

Uninstall with dsh plugin --profile web remove dsh-plugin-loud-failure.

Configuration

The bundle inserts one row, id: loud-failure, with the schema defaults restated. A patch replaces a row's whole config, so an override in your profile's cordis.patch.yml must restate every key it keeps.

A rule:

| Field | Type | Meaning | |---|---|---| | id | string, required | Stable identifier. | | pattern | string, required | JavaScript regular-expression source. | | flags | string | Any of i, m, s, u. g and y are rejected because they carry state between calls. | | tools | string[] | Tools the rule applies to. Omitted or empty: shellTools. ['*']: every tool. | | when | success \| error \| any | Which outcomes the rule inspects. Default success, so a result that already failed is not decorated twice. | | action | error \| context \| off, required | error blocks, context attaches a notice, off disables the rule. | | message | string, required | What the signature means and what the model should do next. It is shown to the model verbatim. |

Disable one built-in rule and add one of your own:

# $DSH_HOME/profiles/web/cordis.patch.yml
- id: loud-failure
  config:
    builtinRules: true
    rules:
      - id: no-such-file-with-success
        pattern: unused
        action: off
        message: unused
      - id: cuda-oom
        pattern: 'CUDA out of memory|torch\.OutOfMemoryError'
        tools: [bash, pwsh, job_output]
        action: error
        message: 'The GPU ran out of memory; whatever this run was supposed to produce is incomplete. Reduce the batch size or free VRAM, then rerun.'
    shellTools: [bash, pwsh, job_output]
    excerptChars: 240
    scanValue: false

Built-in rules

All built-in rules use when: success and apply to shellTools.

The error rules are line-anchored to the shape the tools actually print, which rules out the common quoted-substring mention (grep -r "command not found" logs/, a README quoting the pandoc warning); a file that echoes a genuine log line verbatim will still match, and that is a known limitation. The context rules are deliberately broader because a notice is cheap and the model keeps the original result.

What the model sees

A blocked result, exactly as rendered (the second content block is the untouched original output):

[loud-failure] Tool "bash" reported success, but its output matched a rule (pandoc-missing-character) that indicates a silent failure. dsh-plugin-loud-failure marked this result as an error; do not treat the call as successful.

- pandoc-missing-character (error): pandoc/xelatex dropped one or more glyphs; the produced PDF is missing characters even though the exit code was 0. Rewrite the character (for example write SpO2 instead of a subscript digit) or switch to a font that has it, then rebuild and re-check for this warning.
  excerpt: "pandoc thesis.md -o thesis.pdf --pdf-engine=xelatex\n[WARNING] Missing character: There is no ₂ in font Microsoft YaHei/OT!\n[exit code: 0]"

The original tool output follows unchanged.

A notice attached to a successful result (the result itself is unchanged; this arrives as a user-role message with source {"kind":"plugin","plugin":"loud-failure","form":"notice","summary":"loud-failure: bash output matched numpy-runtime-warning"}):

[loud-failure] Notice for tool "bash": its output matched a rule (numpy-runtime-warning) that often means a silent failure. Verify before relying on this result.

- numpy-runtime-warning (context): NumPy raised a RuntimeWarning about NaN, division by zero, or overflow. Downstream numbers may be NaN or inf while the exit code stays 0. Check the affected arrays before using the result.
  excerpt: "python calc.py\ncalc.py:12: RuntimeWarning: invalid value encountered in divide\n[0.5 nan 0.25]"

Verification

pnpm run check runs typecheck, tests, build, and a pack dry-run; CI runs it on Ubuntu and Windows with Node 20 and 22. 87 tests in total.

• tests/rules.test.ts covers merging (including the duplicate-user-id rejection), compilation failures (each with its message), filters, statelessness, excerpt bounds, the failure-marker detector, and every built-in rule against real signatures plus the negative cases (clean output, non-shell tools, already-failed results, quoted mentions, Aborted as a plain word).
• tests/plugin.test.ts mounts the plugin next to the real ToolRuntime and drives tool calls through the complete pipeline: block with original output preserved, notice with the plugin source, precedence of error over context, tool and when filters, results carrying [exit code: N] / job-status / sandbox markers left alone by success rules, when: any still applying to them, deferred tool context surviving a block, built-in override, a user rule with tools omitted, scanValue, composition with a downstream tools/post-execute listener, clean unmount, and fail-loud on a bad rule table.
• The bundle was also exercised against @deepseek-ai/dsh@0.1.0-rc.6 on Windows: dsh plugin add from a checkout, from the v0.1.0 release tarball URL, and from github:...#<sha> (with the allowBuilds key) all compose, and --dump-config shows the layer; booting dsh --profile web with an overlay that injects an invalid rule fails at boot with RuleConfigError: rule "bad": invalid pattern raised from the installed plugin's apply, and booting with the shipped config serves the Web UI.

Design notes

• tools/post-execute, not tools/result. tools/result only observes the frozen outcome; the point here is to change what the model sees, which is exactly what post-execute is for (replace content, block with feedback, attach context).
• Block keeps the original output and the deferred context. A blocked result would otherwise erase the evidence the model needs to fix the problem. The header comes first so the model reads the verdict before the noise; the original blocks follow unchanged, and context the tool body deferred rides on the decision instead of being dropped.
• Two severities, no third. error is for signatures that mean the work did not happen; context is for signatures that mean "check before you trust". Anything softer than that is not worth a token.
• Regular expressions, not a model. The signatures are literal strings emitted by tools; a regex is deterministic, cheap, testable, and auditable in the config dump. Semantics belong to the model, which gets the rule message and the excerpt.
• Fail loud at load. A rule table is configuration; a bad one should stop the boot with the rule id, not degrade into a listener that never fires.
• Waterfall etiquette. A blocking decision short-circuits, as a decision-owning policy listener should; a context decision delegates first and enriches whatever came back, so downstream replacements survive. Listeners registered before this one wrap it and can still override either outcome.

Model Experience

Request context and condition

#### What the model sees

Only when a rule matches. For an error match: the tool result becomes isError: true and its content is one text block rendered as in [What the model sees](#what-the-model-sees), followed by the original content blocks. For a context-only match: the tool result is unchanged and one user-role notice message with source { kind: 'plugin', plugin: 'loud-failure', form: 'notice', summary } is appended after the recorded tool results of that batch. Rule messages are configuration and appear verbatim; the excerpt is bounded by excerptChars.

#### Token effect

Conditional. Zero when nothing matches. On a match: the header plus, per matched rule, its message and up to excerptChars characters of excerpt. A block adds only that header; the original blocks that follow are the same bytes the model would have received anyway.

#### KV Cache effect

Append-only. The plugin contributes no system-prompt section and never rewrites earlier history; a block changes the content of the current tool result before it is first recorded, and a notice is appended after that batch's tool results. Neither invalidates any previously reusable prefix.

Known Limitations and Deferred Work

• Text only. The listener inspects text blocks (and, with scanValue, the JSON of a successful value). A failure that leaves no trace in the output, such as a library returning a negative status that the script never prints, cannot be caught here; the pattern for those is to print the value and let a rule catch it.
• Regular expressions can false-positive. The context rules are broad by design; disable one with action: off if it is noisy in your workload. A blocked result still carries the original output, so a false error costs the model one corrective step, not the information.
• Only three failure markers are recognized. [exit code: N], a [status: ...] trailer with a non-zero exit code, and the sandbox denial notice. Other ways a shell tool can report failure in text (signal names, timeouts) are not yet treated as failed outcomes, so a success rule can still fire on them.
• Whole-text scan. Very large tool outputs are scanned once per rule; the shipped shell tools already truncate long output, so this has not been a cost in practice, but there is no size cap yet.
• run_code aggregate output. Code Mode sub-dispatches each pass through post-execute and are inspected individually; the outer run_code result (program logs and return value) is inspected as ordinary text when run_code is listed in shellTools, which it is not by default.
• No per-agent scoping. The listener is registered on the plugin context and sees every agent's calls. Scoping to a subagent preset is a Cordis-level exercise not attempted here.

Development

pnpm install
pnpm run check          # typecheck + test + build + pack dry-run
pnpm run 

…