DeepSeek Harness Web Scraping: MCP, Skills, and CLI

TL;DR

• dsh mounts MCP servers as single plugin entries, and their tools appear to the agent as mcp__<serverName>__<tool_name>.
• dsh reads the Anthropic skill format and scans ~/.agents/skills with no configuration, so skills built for other agents port over unchanged.
• Any CLI already on your machine works through the agent's built-in shell tool, the lowest-friction path if you already have a working scraper.
• Built-in web search and fetch handle open pages fine; blocked, client-rendered, or geo-varying targets need a dedicated scraping tool like Decodo instead.

What DeepSeek Harness is and why its plugin model matters for data work

DeepSeek Harness runs on a micro-kernel built on the Cordis framework, where model adapters, tool registries, sandboxes, session handlers, and even the interface all load as independent plugins rather than fixed parts of the runtime. Composition is declarative: you add or swap a capability in config, and the harness source itself stays untouched. Every message, tool call, token count, and subagent dispatch gets written to an append-only event log. This matters later, like when a scrape returns nothing, and you need to see what the agent actually received rather than what it reported. dsh launched on 13 August 2026 under the MIT license, and it's still a developer preview, so expect breaking changes between releases.

The architecture is why mounting an external data source here is genuinely easy: a scraping tool is just another plugin entry, not a fork or a patch.

The runtime ships four modes, but only one of them matters for scraping. Standard mode is the full agent, with file editing, shell access, web search, skills, planning, and subagents, and it's where scraping tools belong. Code mode exposes tools through the Code Mode SDK, so the model composes multi-step operations as a single TypeScript program, worth knowing about for the batch scraping example later in this guide. Minimal mode strips the agent down to persistent bash and str_replace_editor, with no web layer at all. Creator mode adds preset authoring and runtime inspection, and it's the mode to test a new plugin or skill in before promoting it to Standard.

For the broader context of pairing an agent with external data, see what AI scraping is. If you're weighing dsh against other agent CLIs before committing to one, our roundup of AI coding tools covers the wider field.

Why dsh agents fail on real websites

Four failure modes account for most of it, and each one leaves a distinct trace if you know where to look:

• Empty JavaScript shells. The fetch tool returns the page's raw HTML skeleton, before any script has run, so a page that's genuinely full of products comes back looking empty. The agent reports there's nothing there. Confirm this one by diffing the raw response against what DevTools shows after rendering; if the two don't match, this is your cause.
• Anti-bot challenges. The tool gets back a challenge page or a 403, and the agent treats that response like real content, summarizing the challenge HTML as if it answered the prompt. Nothing in the response tells the model it failed.
• Geo-restricted or personalized results. Pricing, stock, and rankings shift with the datacenter IP the agent happens to be running from. This one is worse than the first two because nothing is missing. The data just isn't correct.
• Rate limiting mid-task. The first several pages of a crawl succeed, then the rest start returning 429, and the agent quietly finishes with a partial dataset that looks complete.

Why the agent makes it worse

An agent has no way to tell that a page came back wrong. A blocked response is still a successful HTTP transaction, so the loop keeps running and the model reasons over garbage as if it were data. That gets worse under subagent delegation: a subagent reads a challenge page, summarizes it, and hands that summary up. The parent agent never sees the raw response, so it has nothing to audit against.

The countermeasure is the same for all four. Read the event log and check the raw tool response, not the model's summary of it. This is where dsh's trajectory logging earns its place, since it's the only place in the loop where you can see what the tool actually returned before the model started interpreting it.

For the detection landscape behind challenge pages and geo-blocking, see our anti-bot systems guide. If empty JavaScript shells are the failure you're hitting, the headless browser guide covers the rendering side of the problem.

Three ways to connect a scraping tool to dsh

Pick one of these based on what you're actually trying to reach, rather than working through all three in sequence.

Decision rules

• Open documentation, blogs, and public APIs. Built-in tools are enough. Don't over-engineer this.
• Broad research across many domains with no blocking problem. A community search plugin is the cheapest path.
• Structured extraction from sites that block, render client-side, or vary by region. Use an MCP server, since retry, rendering, and IP logic sit outside the agent loop, where the model can't misreason them.
• You want the agent to know when to escalate. Add a skill. A mounted tool the agent never reaches for is dead weight.
• You already have a working CLI or scraper. Expose it through the shell tool or your own MCP server rather than rebuilding it.

These combine in practice. The realistic production setup pairs an MCP server for the tool itself with a skill that tells the agent when to use it, rather than picking one option and stopping there.

Option 1: Mount a scraping MCP server

Finding the config

This assumes dsh is already running on your machine. If it isn't yet, installation is one command: 

npx @deepseek-ai/dsh web

Node.js 22.19 or newer is the only real prerequisite. The first time you run it, dsh creates a folder at ~/.dsh and sets up a default profile inside it called web.

That profile lives at ~/.dsh/profiles/web/, and inside it is a file called cordis.patch.yml. This is the file you'll edit. It's where every plugin you want dsh to load gets listed, MCP servers included, so every config block in this section goes into that one file.

Open it in any text editor. If it's empty or doesn't exist yet, that's normal; you're about to add its first entry.

Open a terminal and run:

nano ~/.dsh/profiles/web/cordis.patch.yml

The ~ is shorthand for your home folder. nano is a plain text editor already installed on Mac and Linux, so there's nothing extra to install just for this. 

The reason this needs to be a terminal command rather than browsing for the file normally: .dsh is a hidden folder, since it starts with a dot, so it won't show up in a regular file browser by default.

One thing worth knowing before you start: changes to this file don't apply instantly the way a settings change does. Plugin mounts load when dsh starts up, so after you save your edit, you'll need to relaunch dsh before the new tool actually shows up.

How dsh models an MCP server

If you've configured MCP servers in other agent CLIs before, forget that mental model for a second. In dsh, an MCP server isn't a separate JSON file or its own config block. It's a plugin entry, just like a tool or a sandbox, sitting in your dsh profile config alongside everything else the agent can use.

Inside cordis.patch.yml, an MCP server isn't its own special format; it's just another entry in the same list as any other plugin. Each entry needs four things:

• id. A label you choose for this entry, so you can reference it elsewhere in your config.
• name. Always the same value for MCP servers: @deepseek-ai/dsh-mcp-client. This tells dsh which plugin type is loading.
• config.serverName. The namespace this server's tools will live under. Letters, numbers, underscores, and hyphens only, up to 32 characters.
• config.transport. How dsh talks to the server. There are two options, covered below.

Once mounted, every tool the server exposes shows up to the agent as mcp__<serverName>__<tool_name>. So if you set serverName to decodo, a tool like scrape_as_markdown appears to the agent as mcp__decodo__scrape_as_markdown. If you've used MCP with other agent CLIs, this naming pattern will look familiar.

One distinction matters before you write anything into cordis.patch.yml: a bare id / name / config entry is treated as an override, meaning "find the existing row with this id and replace its config." Since a fresh mount like this one has no existing row anywhere in the profile yet, that shape fails with an "entry not found" error instead of creating anything. To add a new row rather than modify one that doesn't exist, wrap it in an insert list instead, as seen in the examples below.

Hosted endpoint over streamable HTTP

Use this when you want to call a server that's already running somewhere else, rather than starting one yourself. This is the simplest starting point if you're setting this up for the first time, since there's nothing to install or run locally.

Here's the tested config for Decodo's MCP server:

- insert:
  - id: mcp-decodo
    name: '@deepseek-ai/dsh-mcp-client'
    config:
      serverName: decodo
      transport: streamable-http
      url: https://mcp.decodo.com/mcp
      headers:
        Authorization: !!js '`Basic ${process.env.SCRAPER_API_TOKEN}`'

Walking through each field:

• id: mcp-decodo. Just a name for this entry in your config.
• name. The fixed MCP client plugin value from above.
• serverName: decodo. This becomes the mcp__decodo__ prefix on every tool.
• transport: streamable-http. Tells dsh to connect over HTTP rather than launching a local process.
• url. The hosted endpoint address.
• headers.Authorization. How your credentials get sent with every request.

On that last line: dsh has no built-in OAuth flow, so authentication happens through a static header instead. Notice the config doesn't contain an actual token. It reads the token from an environment variable, SCRAPER_API_TOKEN, at runtime. Set that variable in your shell or your deployment environment; never paste the token directly into this file, and keep this config file out of version control if it ever does end up holding a real secret by mistake.

Local server over stdio

Use this instead if you want the server running as a local process alongside the agent, rather than calling a hosted endpoint. This is also the option to reach for if you need to pin a specific server version rather than depend on whatever the hosted endpoint is running.

Same plugin, same id and name, just a different transport:

- insert:
  - id: mcp-decodo
    name: '@deepseek-ai/dsh-mcp-client'
    config:
      serverName: decodo
      transport: stdio
      command: npx
      args: ['-y', '@decodo/mcp-server']
      env:
        SCRAPER_API_TOKEN: !!js process.env.SCRAPER_API_TOKEN

Instead of a url, stdio transport gives dsh a command to run. Here, that command is npx -y @decodo/mcp-server, which downloads and starts the server locally. The env block passes your token through as an environment variable to that local process – the same credential, just handed over a different way since there's no HTTP request to attach a header to.

The config keys that matter for scraping

These four settings aren't unique to Decodo's server; they apply to any MCP server you mount in dsh, but they matter more for scraping than for most other tool types:

• toolCallTimeoutMs. Defaults to 60000 (one minute). A page sitting behind a rendering challenge can take longer than that to resolve, and when it does, you'll see a timeout error rather than anything mentioning a block. That's worth knowing before you spend time debugging the wrong problem. Raise this value for heavier targets.
• failOnStartupError. Defaults to false. That means a typo in your url, or any other startup mistake, gives you an agent that quietly has no scraping tool at all, with no error telling you why. Set this to true while you're first setting things up, so mistakes surface immediately instead of hiding.
• reconnect.enabled, initialDelayMs, maxDelayMs, and maxAttempts. These control what happens if a hosted server drops the connection partway through a long crawl. Worth tuning if you're running jobs that take a while.
• cwd. Only relevant for stdio transport. Sets the working directory the local server runs from, useful if that server needs to run against a specific project folder.

Verifying the mount

Before you trust this setup with a real task, confirm it's actually working.

1. Run npx @deepseek-ai/dsh --profile web --dump-config and check that your MCP entry appears in the resolved plugin tree.
2. Start a session and list the available tools. You should see tools prefixed with mcp__decodo__.
3. Run one scrape against a page where you already know what the output should look like. Then check the raw tool response in the event log, not the agent's summary of it, to confirm the data actually came back correctly.

If something's not working, it's almost always one of three things: a handshake timeout because the remote server was cold and slow to respond, an empty tool list caused by incorrect YAML indentation somewhere in the config, or a naming collision because two mounted servers share the same serverName. That last one is worth watching for specifically, since how dsh resolves the collision isn't documented anywhere. Keep every serverName value in your config distinct, and you avoid the question entirely.

Decodo's MCP server is the worked example throughout this section. For MCP fundamentals beyond what's specific to scraping, setting up an MCP server from scratch covers that ground separately.

Option 2: Install a scraping skill

The quotable fact first: skills you've already written for another agent CLI work here unchanged. No conversion, no rewriting, just a copy.

Where skills live

dsh reads the same skill format Anthropic uses, and it scans for skills in three places automatically, with nothing to configure.

• ~/.agents/skills. Global, available to every project.
• ~/.dsh/skills. Also global, a dsh-specific location.
• <project>/.agents/skills. Scoped to just the project you're currently in.

The official dsh repo ships its own .agents/skills folder, which is a strong signal that this is the intended way to extend the agent, not a side door.

Practically, this means a skill you wrote for a different agent CLI is a copy-and-paste away from working in dsh.

Installing the Decodo scraping skill

A skill is just a folder, and it lives in Decodo's agent skills repo. Get it onto your machine first, then copy it into place.

git clone https://github.com/Decodo/agent-skills.git
cd agent-skills
mkdir -p ~/.agents/skills
cp -r skills/decodo-web-scraping ~/.agents/skills/

That last line copies the whole decodo-web-scraping folder into the global skills directory. dsh reads the skill catalog once, when a session starts, so restart dsh the same way as before (Ctrl+C, then relaunch), then open a new session. The skill should show up under the Skills group in the command palette.

Why a scraping skill earns its place

Mounting the MCP server from Option 1 only gets you halfway. Without a skill telling it otherwise, the agent still defaults to the cheapest tool available, which is plain fetch, and plain fetch is exactly what returns the blocked page in the first place.

A skill fixes this by giving the agent an explicit rule for when to escalate. Something like: use plain fetch for static docs and text pages, but switch to the scraping tool when the response comes back under a certain size, contains a challenge marker, or the target is a site you already know is dynamic.

Here's what that looks like as an actual skill file, using the same quotes.toscrape.com target from earlier in this series, so you can see the shape of a minimal one:

---
name: collect-quotes
description: Use when scraping quotes.toscrape.com for quote text, author, and tags. Triggers on requests to collect quotes, extract quote data, or paginate through the site.
---

Scrape quotes.toscrape.com and return structured data.

Target: https://quotes.toscrape.com

Fields to extract per quote:
- text (the quote itself)
- author
- tags (list)

Pagination: follow the "Next" link at the bottom of each page until it's no longer present.

Output: one JSON object per quote, newline-delimited (JSON Lines).

Use the mcp__decodo__ scraping tool for this target instead of the built-in fetch tool. The tag pages here render some content client-side, and plain fetch can return incomplete tag lists.

Confirming the skill is picked up

Before relying on it, check that the agent actually sees it. Start a new session and ask something that should trigger the skill, like "collect the quotes from quotes.toscrape.com." If the agent reaches for the mcp__decodo__ tool instead of plain fetch on its own, the skill is working. If it still reaches for plain fetch, double-check that the folder landed in the right place and that its SKILL.md frontmatter actually describes the trigger clearly enough for the agent to match it.

If the agent still isn't reaching for it, the most common cause isn't a misplaced file; it's forgetting the restart. Skills installed mid-session don't appear until the next one starts.

Test new skills in Creator mode first, since that's the mode built for runtime inspection, and only move a skill into your regular Standard profile once you've confirmed it behaves the way you expect.

Decodo's agent skills repo is at github.com/Decodo/agent-skills. For the output format this skill is producing, scraping a website to Markdown covers why that's usually the format agents handle best.

Option 3: Call a scraping CLI from the shell tool

This is the shortest of the three options. dsh Standard mode already ships a shell tool, so anything installed on your machine and reachable on PATH is something the agent can already run.

How it works

If a CLI is installed and callable from your terminal, the agent can call it too, the same way it runs any other shell command. There's no separate setup step specific to dsh here, since this isn't a plugin or a skill; it's just the agent using a tool it already has.

Concretely, set your credential as an environment variable in the same terminal dsh runs in:

export DECODO_AUTH_TOKEN="your-token-here"

Once that's set, the agent can call the CLI directly, either the installed command or through npx:

decodo scrape https://quotes.toscrape.com
# or, without a global install
npx -y @decodo/cli scrape https://quotes.toscrape.com

You don't need to relaunch dsh for this one. Environment variables set in the terminal before you start dsh are just part of the process environment the shell tool inherits, not something dsh has to load or scan.

The trade-off

An MCP tool comes with a schema, telling the agent exactly what arguments exist and what they do. A shell command doesn't. The agent has to infer the CLI's usage from whatever it can see, like a --help output or general familiarity with the tool, which means less reliable argument use than Option 1 gives you. That makes this a good fit for testing something quickly or running a one-off job, and a weaker fit for a pipeline you're going to run repeatedly and trust.

If you do end up keeping this as your long-term setup rather than just testing with it, pair it with a skill. The skill can supply exactly the usage guidance a shell command has no schema to carry on its own, closing the gap Option 2 already solves for the MCP approach.

The Decodo CLI itself lives atgithub.com/Decodo/cli.

Handling blocked pages, rendering, and rate limits

JavaScript-rendered targets

Before adding any rendering layer, confirm that's actually the problem. Check the raw response your fetch tool returns against what DevTools shows after the page finishes loading. If they don't match, that's client-side rendering, and the fix belongs here. If they do match, the failure is something else, most likely a block, and adding a headless browser won't touch it.

Once confirmed, there are two paths. In-house, you run a headless browser behind a thin HTTP wrapper and mount that wrapper as a local stdio MCP server, the same shape covered in Option 1. This works, but you now own the browser's lifecycle: crashes, memory growth, and keeping it alive across long sessions become your problem, not the agent's. The managed alternative is a scraping API that renders server-side, so the dsh tool call itself stays a plain request that comes back with parsed data, no browser process for you to babysit on your own machine.

IP reputation and geo-accuracy

Anything running inside a container or CI runner defaults to a datacenter IP, and consumer-facing sites score those accordingly, often serving a challenge page before your scraper does anything else.

Residential IPs matter here for a reason beyond just getting through: correctness. Pricing, stock levels, and rankings are location-dependent, so the wrong IP doesn't return an error; it returns data that looks completely plausible and is simply wrong. That's the failure an agent has no way to catch on its own, since nothing about a wrong-but-valid response looks broken.

For anything behind a login or a multi-step flow, use sticky sessions too, so the IP doesn't rotate out from under you mid-flow and invalidate the session it's in.

Rate limits and crawl pacing

Cap in-flight requests at the tool layer itself rather than trusting the agent to pace things on its own. Left unconstrained, the model will parallelize aggressively if the tool lets it, which is exactly what triggers rate limiting in the first place.

When a 429 comes back, surface it to the agent as an explicit error, never as ordinary content. A silent partial dataset, where the crawl looks complete but quietly isn't, is the worst outcome an agentic scrape can produce, precisely because nothing about it looks wrong.

Worth watching from a cost angle too: every retried page gets re-read by the model, not just re-fetched. Keep an eye on the event log's token metrics so a retry loop shows up as a cost spike before it shows up as a bad dataset.

When you need to scale

At some point, maintaining a browser pool, a proxy rotation layer, and challenge handling just to keep one agent supplied with data starts to look like rebuilding a scraping API around that agent. The build-versus-buy line here isn't about capability; it's about where you want that ongoing maintenance to sit: on your own infrastructure, or behind a single endpoint someone else keeps running.

Decodo's Web Scraping API is that single-endpoint option, handling rendering, proxies, and challenge pages together rather than as three separate things you're wiring up yourself. And where the agent specifically needs consumer-grade IPs for geo-accurate results rather than just access, Decodo's residential proxies are the layer that addresses that correctness problem directly.

The operational side of running this at real volume, beyond what a single agent needs, is really its own topic, and mismatched IP location on a proxied request is one of the more common ways an otherwise-working setup starts returning quietly wrong data.

Worked example: a dsh agent that builds a product dataset

This ties Options 1 and 2 together into one real run, using books.toscrape.com as the target, the same site used earlier. It has 1,000 books spread across 50 pages, 20 per page, which gives you a known total to check your final dataset against.

This assumes dsh is already installed and running, picking up right where the earlier sections left off.

Step 1: Define the goal in the prompt

Open a new session and state the task plainly, including the output shape you want. Something like this works:

Scrape books.toscrape.com and collect the title, price, rating, and 
availability for every book across all paginated listing pages. 
Output the result as JSON Lines, one book per line.

For now, you don't need to input this anywhere; just jot it down in a note. Being explicit about the fields and the output format here matters. It's what the skill in Step 3 will reference, and what makes the final output checkable in Step 6.

Step 2: Add the plugin entry and verify the tool list

If you haven't already mounted the Decodo MCP server from Option 1, do that now: add the insert-wrapped block to ~/.dsh/profiles/web/cordis.patch.yml, save it, and relaunch dsh.

- insert:
  - id: mcp-decodo
    name: '@deepseek-ai/dsh-mcp-client'
    config:
      serverName: decodo
      transport: streamable-http
      url: https://mcp.decodo.com/mcp
      headers:
        Authorization: !!js '`Basic ${process.env.SCRAPER_API_TOKEN}`'

Once dsh is back up, confirm it's mounted before doing anything else. Run npx @deepseek-ai/dsh --profile web --dump-config in a terminal and check the mcp-decodo entry appears, then start a session and list available tools to confirm you see ones prefixed mcp__decodo__. If either check fails, stop here and revisit Option 1 rather than continuing, since nothing past this point works without the tool actually being there.

Step 3: Drop in the skill

Mounting the tool only means the agent can use it, not that it will. Without a skill, the agent still defaults to plain fetch, and books.toscrape.com's tag pages render some content client-side, so plain fetch on its own can miss data.

Create a skill file for this specific target:

mkdir -p ~/.agents/skills/collect-books
nano ~/.agents/skills/collect-books/SKILL.md

Enter this inside the file:

---
name: collect-books
description: Use when scraping books.toscrape.com for book title, price, rating, and availability across the full paginated catalog. Triggers on requests to collect books, extract book data, or build a books dataset.
---

Scrape books.toscrape.com and return structured data.

Target: https://books.toscrape.com/catalogue/page-1.html

Fields to extract per book:
- title
- price
- rating
- availability

Pagination: follow the "next" link at the bottom of each page until it's 
no longer present. There are 50 pages, 20 books each, 1000 books total.

Output: one JSON object per book, newline-delimited (JSON Lines).

Use the mcp__decodo__ scraping tool for this target instead of the 
built-in fetch tool.

Save the file.

Then, same as Option 2, restart dsh and start a fresh session, since the skill catalog only refreshes at session start, not mid-session.

Step 4: Run in Standard mode and watch the trajectory

Give the agent the prompt from Step 1 in this new session (Standard mode is the default, so nothing extra to switch on here).

The first page should succeed cleanly: the agent calls the mcp__decodo__ tool, gets back structured data for the first 20 books, and extracts the four fields. From there, it should discover the pagination pattern on its own, the "next" link the skill told it to follow, and start iterating through the remaining pages automatically.

At this point, the agent has everything it needs: the mounted tool, the skill telling it when to reach for that tool over plain fetch, and a clearly defined target to work through the full catalog on its own and return the completed dataset as JSON Lines.

Limitations to expect in the developer preview

• Breaking changes are expected. The maintainers say so directly. Pin your dsh version in any pipeline that matters, and re-verify your config after every upgrade.
• No OAuth flow in the harness core. Static tokens and headers work fine for authentication. Servers needing dynamic client registration require a community plugin, such as dsh-oauth-mcp-client.
• Tool name collision behavior across servers is undocumented. Keep every serverName value distinct rather than relying on how dsh resolves conflicts.
• No documented GitHub-native workflow or hosted background agents, unlike established commercial agent CLIs. Local and self-hosted only, for now.
• The plugin ecosystem is only weeks old. Extension contracts are unstable, and most plugins have a single maintainer. Read the source of anything you mount with credentials.

Final thoughts

DeepSeek Harness's plugin-first design makes it unusually easy to extend with data tools; mounting a scraping server is one config entry, not a fork. Skills porting across unchanged means work you've already done for other agents carries over rather than starting from scratch. But the real cost in this piece sits somewhere else: the gap between an agent that fetched a page and one that has correct data, a gap that stays invisible unless you read raw tool responses instead of trusting what the agent reports. Verify everything here against the live repo before relying on it. This is a preview, and the config in this article, like any article's, has a shelf life.