DeepSpider

 

> An AI-native web scraping and JavaScript reverse-engineering platform powered by DSH, Patchright/CDP, and an independent semantic runtime—from browser evidence to recovered parameter-generation logic and runnable Solvers verified against real requests.

DeepSpider combines DSH Web, Patchright Chromium, Chrome DevTools Protocol (CDP), and an independent Node semantic runtime. The browser collects requests, scripts, and runtime facts. A finished result must be generated again outside the browser and accepted by a real request; page automation or a one-off capture is not the deliverable.

中文

Quick start

DeepSpider requires Node.js >=24.15.0. Global installation downloads Patchright Chromium.

npm install -g deepspider
deepspider agent

deepspider is also published as a native DSH Bundle. The primary command mounts it into the web Profile through the DSH Profile manager; an existing DSH installation can add it directly:

dsh plugin --profile web add deepspider
dsh web

This is the primary start command. DSH Web loads the Spider Preset. Create a Session and describe the target URL, trigger path, and target output. The first-release automatic semantic recovery supports Cookie only. Header, Query, Body, return-value, and navigation remain browser evidence, Output Contract, and manual analysis targets; the high-level tool does not currently produce an automatic reproduced result or Solver for them. Multiple Sessions can run concurrently with isolated browsers, SessionArtifactStores, Workers, and artifact roots.

Ctrl+C closes DSH Web and waits for every Session-owned Patchright Chromium process, sdenv Worker, and runtime resource to exit.

What DeepSpider is built for

• Follow real request Initiators, call stacks, and script source to the parameter write boundary.
• Analyze dynamic execution, Webpack, Workers, WebAssembly, state machines, and heavily obfuscated code.
• Use Hooks, the Debugger, and property capture to fill evidence gaps without editing captured source in place.
• Express browser dependencies required for Cookie generation as an auditable Runtime Recipe and execute them in an independent Worker.
• Validate generated Cookies against the real request and export a Solver that runs without the Browser Session; use the general browser, Hook, Debugger, and Code Mode capabilities to locate and implement other output kinds.

One definition of done

For the currently automated Cookie path, recovery is complete only when the Browser Oracle has stored target evidence; an Output Contract and Runtime Recipe belong to the current Session; an independent sdenv Worker generates the Cookie from fresh state; CycleTLS sends a real request using only those generated values; the Validation level is reproduced; and the exported Solver reaches the same acceptance result after the browser has closed.

Browser output, a page-automation script, captured Cookies, a single Hook log, or a request that only replays captured values is not completion evidence.

Output-driven semantic recovery

Browser Oracle → Session Artifact Graph → Output Contract → Runtime Recipe
               → sdenv Worker → Real-request Validation → Solver

An Output Contract can describe Cookie, Header, Query, Body, return-value, and navigation outputs. The first-release end-to-end automation implements Worker generation, real-request validation, and Solver export for Cookie only. Other kinds retain Contract, Artifact Graph, and manual reverse-engineering support without an automatic reproduced claim.

Evidence levels

The Runtime Recipe boundary

Confirmed fixed fingerprints, concealed properties, window proxy settings, and target-specific rules may live in the current Session's Runtime Recipe. The generated-result identity binds the Recipe and Contract hashes, engine version, upstream Artifact IDs, and SHA-256 values. Core code executes these declarations without branching on a hostname, Cookie name, or protection vendor.

A Patchright Session is important evidence, not an absolute browser truth. When automation fingerprints or timing differ, compare regular Chrome, multiple Sessions, or confirmed target behavior, collect the missing facts, and update the Recipe.

Observed, derived, and generated artifacts

Captured scripts, responses, and dynamic sources are immutable observed Artifacts. Formatting, deobfuscation, or targeted processing creates a new derived Artifact with its source ID, transformation description, and content hash. Worker outputs are generated Artifacts. Only successful real-request validation raises the result to reproduced.

One recovery entry point

Normal recovery uses one high-level tool:

recover_target_output({ url, outputKind, outputSelector?, mode? })

For a Cookie Contract, the tool builds the Artifact Graph, Output Contract, and Runtime Recipe; starts the Session-owned sdenv Worker; performs real-request validation; and exports a Solver. Header, Query, Body, return-value, and navigation can enter evidence and Contracts, but the high-level tool does not currently complete their independent generation or Solver export. It returns only stage status, evidence levels, strategy, the first blocker, Solver Artifact ID, and next action to the Agent. Source, Cookie values, and full diagnostics stay in private Session Artifacts.

mode: "auto" selects the semantic runtime by default. mode: "algorithm" has no automatic algorithm engine yet and returns an explicit program blocker: algorithm-recovery-engine-not-implemented. The Agent can then use the existing Hook, Debugger, and Code Mode workflow to recover the local logic that affects the target output, or wait for a future algorithm engine. DeepSpider does not present this unimplemented escalation as automatic completion.

Eight-stage reverse-engineering workflow

intake → evidence → locate → recover → runtime → extraction → validation → handoff

The Coordinator makes at most three semantic attempts and stops on success. It does not modify the Runtime Recipe or handle a blocker between attempts. If all three fail, it returns the first blocker and next action; the Agent may then collect evidence or manually change the Recipe before starting a new recovery run. environment is a missing browser semantic, resource is a dependency or network-response problem, program is behavior the current engine cannot execute, and validation means an output was generated but the request rejected it.

DSH Agent and Dialog

• Sessions run multiple tasks with isolated browsers, Workers, and files.
• Goals and Todo hold the task objective and current execution items.
• Code Mode uses run_code and the generated TypeScript SDK to call DeepSpider tools.
• Cordis dynamic tools inspect and invoke runtime capabilities within current Agent permissions.
• Web Search finds public references; the Browser Oracle remains the source of page facts.
• Dialog is an optional in-browser panel that shows browser evidence, Artifact Graph, Node generation, and request validation status.

When the output kind is ambiguous, login interaction is required, or algorithm recovery needs approval, DeepSpider uses the native DSH single-choice, multiple-choice, and custom-answer protocol. browser_dialog opens only when the current Session owns a browser. Answers return to the same Session; there is no second dialog state machine.

Usage

fetch does not launch a browser or enter the Agent workflow. The MCP stdio external adapter gives other clients the same central tool catalog. Use DSH Web for the complete multi-Session workflow.

Tool catalog

Catalog size is derived from code and is not a documentation contract.

Session artifacts and Solver

~/.deepspider/sessions/<sha256(agent.id)>/
├── evidence/
│   ├── sites/            # Requests, responses, scripts, and site indexes
│   └── artifacts/        # Artifact Graph, Contract, Recipe, Run, Validation, Solver
├── runs/                 # sdenv Worker requests, results, and diagnostics
├── solvers/              # Standalone Solvers
├── screenshots/
└── browser-data/

Every successful automated Cookie recovery creates four files under solvers/:

solver.mjs
contract.json
recipe.json
package.json

Install and run them with npm. Installation builds the sdenv native module:

npm install
node solver.mjs

The Solver creates a fresh Cookie Jar. It imports no Patchright code and reads neither Session browser-data/ nor captured Cookies. It prints a compact validation result and closes sdenv and CycleTLS before exit.

Architecture

DSH Web Host Plane
├── Sessions, models, Goals, Todo, Cordis, and event routing
└── Spider Agent Plane
    ├── Code Mode + DeepSpider Catalog
    └── Session-owned DeepSpider Runtime
        ├── Patchright Chromium + CDP + Dialog
        ├── Browser Oracle + SessionArtifactStore
        ├── Session Artifact Graph + RecoveryCoordinator
        └── sdenv Worker + CycleTLS Validator + Solver

MCP stdio adapter
└── the same DeepSpider Catalog

The Host Plane owns application services and multiple Sessions. An Agent Plane performs one Session's reverse-engineering work. RuntimeManager enforces state boundaries. When a Session is disposed or the Host receives an exit signal, DeepSpider aborts the active operation and closes Workers before Dialog, CDP, Patchright Chromium, and the Store.

Development and release checks

Source development uses Node.js >=24.15.0 and pnpm 11.22.0.

git clone https://github.com/ma-pony/deepspider.git
cd deepspider
pnpm install

pnpm test
pnpm lint
pnpm test:integration
pnpm smoke:pack

Integration tests require an environment that permits Patchright Chromium child processes. DeepSpider does not load a project-root .env automatically. Set DEEPSPIDER_HEADLESS=true explicitly for headless operation.

Security and authorization

DSH stores model-provider settings and credentials; DeepSpider bundles no accounts. Cordis, browser debugging, script execution, and network access are privileged capabilities for trusted tasks. Analyze only targets that you own or are authorized to assess, and comply with target terms and applicable law.

License

MIT