---
name: mrscraper-onboarding
description: |
  MrScraper helps AI agents and apps search the web, scrape pages as HTML,
  Markdown, or structured JSON, capture rendered-page screenshots, extract
  website branding, and run browser workflows that click, type, scroll, run
  JavaScript, and follow links. Use these capabilities for research, pricing
  intelligence, and competitor comparisons. Use this skill to connect
  MrScraper through the official connector or plugin, MCP server, or CLI
  and skills, obtain credentials, or set up x402 access. Choose the path
  that matches the work the agent needs to do.
---

# MrScraper

MrScraper helps agents discover sources through web search, scrape their
pages as HTML or Markdown, and extract structured JSON with a prompt.
Capture rendered-page screenshots for visual checks, or extract a website's
colors, typography, logos, and social links to understand its branding.
When a page needs interaction first, run a browser workflow over HTTP.

Use these outputs to research markets, collect product and pricing data,
compare competitor prices, build brand inventories, or prepare content for
retrieval pipelines. Retrieve data during the current session, integrate
the same capabilities into an app, or combine the evidence into a finished
report or dataset. Check account usage before scaling a workflow.

## Install

Choose the connection that fits the current environment. Check for an
existing working connection before installing or authenticating again.

1. **Native plugin or connector.** If MrScraper is available in the active
   platform, connect it there and verify its tools in this session. Follow
   Path G for platform setup.
2. **MCP.** Use `https://mcp.mrscraper.com/mcp` with Streamable HTTP and
   browser OAuth. For a local server, use `@mrscraper/mcp` over stdio.
3. **CLI and skills.** Use the setup below when terminal commands are
   available. A CLI login and an MCP authorization are separate credentials.
4. **Application code or HTTP.** Path B covers SDKs; Path E covers direct
   requests; Path H covers browser workflows. Path F covers paid x402
   access without a MrScraper API key.

Keep setup within the requested platforms and coding harnesses. If MCP is
unavailable or awaiting authorization, proceed with CLI setup when the
environment supports it and identify the MCP step still outstanding. When
terminal commands cannot run, complete the supported connector or MCP
connection without attempting to install the CLI or its skills.

Report each connector, MCP connection, and CLI environment independently.
An unfinished setup on one interface should not hold up the others. Run
the verification checks you can perform yourself, and reuse checks that
already succeeded for that interface in the same environment during this
setup. Involve the user when authorization is required or a platform
control is beyond the agent's access. Distinguish package installation,
credential configuration, and verified API access in the readiness report.

### CLI

Requires Node.js 20 or newer. For agent-controlled installation, install the
CLI and available skills first, then authenticate separately:

```bash
npx -y @mrscraper/cli@latest init --all --yes --skip-auth
mrscraper login
```

`--all` targets detected supported harnesses. Use `--agent <id>` to target
one, `--skip-install` to reuse an installed CLI, and `--skip-skills` when
skills are unnecessary. Inspect `mrscraper init --help` for the full options.

This installs:

- The seven service commands listed in Path A.
- The `mrscraper`, `mrscraper-fetch`, and `mrscraper-serp` CLI skills.
- CLI authentication when you complete the separate login step.

Check installation and access from the environment that will run the work:

```bash
mrscraper --version
mrscraper auth status --json
mrscraper status --json
mkdir -p .mrscraper
mrscraper scrape-markdown https://example.com --max-retries 1 \
  > .mrscraper/install-check.json
```

`auth status` only checks local credential configuration. The service
`status` command checks API access and returns the raw account body inside
the response envelope. Treat account responses as private.

CLI setup does not register an MCP server or connect another platform.
See the [CLI guide](https://docs.mrscraper.com/docs/getting-started/cli).

## Choose Your Path

- **Connect MrScraper inside the current chat or agent platform:** Path G.
- **Retrieve web data during this session:** Path A.
- **Call MrScraper from an application, script, or agent loop:** Path B.
- **Produce a report, dataset, or other finished artifact:** Path C.
- **Create or authorize an account, or obtain an API key:** Path D.
- **Use HTTP without installing the CLI or SDK:** Path E.
- **Use paid services without a MrScraper API key:** Path F.
- **Click, fill forms, scroll, run JavaScript, or follow links on a page:**
  Path H.

Combine paths when the task requires it. Infer the intended outcome from
the request; ask only for missing information that blocks the next step.

---

## Path A: Live Web Tools

Use the available MCP connection for work inside the current session.
Use the CLI when that is the requested interface or the MCP connection
does not provide the needed service. Inspect the actual tool list and
input schemas before calling tools; client prefixes can differ.

### Seven services

| CLI command       | SDK / MCP name   | Standard description                                                                                                            |
| ----------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `status`          | `status`         | Return the raw account status API response without summarizing, redacting body fields, or adding calculated fields.             |
| `scrape-html`     | `scrapeHtml`     | Retrieve the raw HTML of a page. Browser rendering is optional.                                                                 |
| `scrape-markdown` | `scrapeMarkdown` | Retrieve a page as Markdown. Including HTML also enables browser rendering and returns both outputs as JSON.                    |
| `get-screenshot`  | `getScreenshot`  | Capture a base64 screenshot of the full page or top viewport. Browser rendering is always enabled.                              |
| `scrape-brand`    | `scrapeBrand`    | Extract a website's colors, typography, logos, and social links as JSON.                                                        |
| `scrape-json`     | `scrapeJson`     | Extract structured JSON from a page using a natural-language prompt. Append Json Schema: and a JSON schema to shape the output. |
| `web-search`      | `webSearch`      | Search the web and return ranked results with titles, URLs, descriptions, and sources.                                          |

Python uses snake_case, such as `scrape_json`; LangChain tools add the
`mrscraper_` prefix, such as `mrscraper_scrape_json`.

```bash
mrscraper status
mrscraper scrape-html https://example.com --max-retries 1
mrscraper scrape-markdown https://example.com --warm-start --max-retries 1
mrscraper get-screenshot https://example.com --screenshot top --max-retries 1
mrscraper scrape-brand https://example.com
mrscraper scrape-json https://example.com --max-retries 1 \
  --prompt 'Extract the page title. Json Schema: {"type":"object","properties":{"title":{"type":"string"}},"required":["title"]}'
mrscraper web-search "site:example.com documentation" --country us --num-results 10
```

Choose the smallest useful output:

1. Start with `webSearch` when you need to find source URLs.
2. Retrieve selected URLs as Markdown for reading or HTML for local parsing.
3. Request a screenshot when visual evidence matters, or brand data when
   the task needs colors, typography, logos, or social links.
4. Use `scrapeJson` for prompt-directed extraction with an explicit schema.
5. Validate the returned fields before scaling to more URLs.

HTML and Markdown support `warmStart` (CLI `--warm-start`, Python
`warm_start`), default `false`. Enabling it can improve scraping reliability
and adds latency. Markdown with `html=true` returns both representations
and enables browser rendering. Screenshots always use browser rendering.

Page timeout defaults to 300 seconds; geolocation and proxy country default
to `us`. `maxRetries` is omitted by default, which means no retry limit;
set a finite limit for bounded automated runs. See the
[API Reference](https://docs.mrscraper.com/docs/api/overview) for each
service's accepted fields and defaults.

The JSON service uses the General Agent for one known page. It does not
click, fill forms, paginate, or follow links. For multi-page work, discover
the URLs and request each page explicitly. The
[General Agent guide](https://docs.mrscraper.com/docs/features/ai-scraper/general)
explains its scope. When a page needs clicks, form input, scrolling,
injected JavaScript, or followed links, send a workflow over HTTP as shown
in Path H.

For focused CLI guidance, load the installed `mrscraper-fetch` skill for
known URLs or `mrscraper-serp` for search. Their maintained sources are in
the [CLI skills repository](https://github.com/mrscraper-com/cli/tree/main/skills).

### Verify results and recover

CLI and SDK results contain `status_code`, `data`, and `headers`. MCP places
that envelope in `structuredContent` and also supplies JSON text in
`content`. LangChain returns a JSON string; decode it before inspecting it.

Check HTTP status and the service body. JSON extraction and search expose
`success`; a brand job has its own `status` and `error`. HTML and Markdown
can be plain text. In extraction metadata the usage key is `token_usage`;
search uses `tokenUsage`. Preserve response key names and types.

The account details in a wrapped status response are at `data.data`.
Return that body unchanged when raw status is requested; do not calculate
or substitute a quota summary.

Correct invalid inputs before retrying. For a transient network or server
failure, retry a bounded number of times with a delay and retain the
attempt outcomes. Authentication, missing scopes, and exhausted quota
require fixing the corresponding access issue.

---

## Path B: Integrate MrScraper Into an App

Use this path when the resulting code will keep calling MrScraper after
the current session. Inspect an existing project's API and secret-handling
patterns before adding a client. For a new project, choose the language,
install its SDK, configure credentials, and verify one representative call.

Provide the key through the project's secret store or an untracked local
environment file:

```dotenv
MRSCRAPER_API_KEY=YOUR_TOKEN
```

Choose the SDK for the application:

```bash
# Python
python -m pip install --upgrade mrscraper-sdk

# JavaScript or TypeScript
npm install @mrscraper/sdk

# LangChain
python -m pip install --upgrade langchain-mrscraper
```

Check that the installed release exports the seven-service interface in
Path A before using these examples. If a needed method is absent, use a
compatible release or source checkout and verify it before proceeding.

Python:

```python
import asyncio
from mrscraper import MrScraper

async def main():
    result = await MrScraper().scrape_markdown(
        "https://example.com", warm_start=True, max_retries=1
    )
    print(result["data"])

asyncio.run(main())
```

JavaScript:

```js
import { scrapeMarkdown } from "@mrscraper/sdk";

const result = await scrapeMarkdown("https://example.com", {
  warmStart: true,
  maxRetries: 1,
});
console.log(result.data);
```

Route the application's requested output through the seven services in
Path A. Keep the same parameter meanings while using the SDK's native
naming and types. Run one authorized smoke test and check both transport
success and the expected output structure before expanding the integration.

SDK references: [Python](https://github.com/mrscraper-com/sdk-python),
[Node](https://github.com/mrscraper-com/sdk-node), and
[LangChain](https://github.com/mrscraper-com/langchain-mrscraper).
If credentials are missing, complete Path D or select Path F for an
application explicitly using x402 payments.

---

## Path C: Repeatable Deliverables

Use this path for a finished artifact: a research brief, comparison, brand
inventory, source-backed dataset, content audit, or knowledge base.

1. Identify the deliverable, required fields, and source boundaries from
   the request. Resolve only the details that would change the work.
2. Use search to discover sources, then retrieve the relevant pages.
3. Keep source URLs and retrieval dates alongside the evidence. Use JSON
   extraction or local parsing for repeated fields and validate a sample.
4. Produce the requested artifact with citations or a source column, as
   appropriate. Record missing data and failed requests explicitly.
5. Save the inputs, schema, and relevant service options so the workflow
   can be repeated.

Coordinate concurrency with account limits. Repeated monitoring requires
an explicitly configured scheduler or application; these seven services
do not create monitoring jobs. If the request becomes an application
integration, continue with Path B.

---

## Get Credentials

Choose the credential route that matches the interface:

- **Dashboard or CLI:** sign in at
  [MrScraper](https://app.mrscraper.com/auth/login), create an account when
  needed, and obtain a key from the
  [API keys page](https://app.mrscraper.com/api-tokens), or complete CLI
  browser login. Continue with Path D.
- **AgentMail:** use an authorized AgentMail inbox as the email identity
  for MrScraper account signup and verification. Follow the actual account
  form and its verification messages, then obtain the MrScraper API key.
  AgentMail inbox access and a MrScraper API key are separate credentials.
- **Hosted MCP or a native connector:** complete its OAuth connection;
  API keys are the fallback for clients without OAuth.
- **x402:** use a compatible funded wallet or payment-provider balance for
  supported paid routes. Path F does not require a MrScraper account key.

An existing key can serve API and SDK calls through `MRSCRAPER_API_KEY` or
`MRSCRAPER_API_TOKEN`. Check credential precedence for the selected client
when both are configured. Never put credentials in public links, saved
reports, or committed files.

If a request returns `401`, repair authentication for the interface that
made it. For an MCP scope error, reconnect and grant the required scope.
For x402 `402`, inspect the payment challenge; an API-key login does not
complete a wallet payment.

---

## Path D: Account Authorization or API Key

### Step 1: Check existing access

For CLI work, run `mrscraper auth status --json`. For an existing MCP
connection, inspect its tools and make an account-status call. Reuse a
verified credential instead of starting another sign-in.

### Step 2: Sign in through the selected interface

For CLI browser authorization:

```bash
mrscraper login
```

Keep the command running while browser sign-in and approval complete.
`--no-open` prints the login URL and keeps the local callback listener
active. Use it when the browser must be opened separately.

For dashboard/API access, open
`https://app.mrscraper.com/auth/login`, complete the account flow, and open
`https://app.mrscraper.com/api-tokens` to obtain a key.

### Step 3: Complete email verification when required

With AgentMail, use the inbox the agent is authorized to control for this
account. Read the verification message associated with this signup or
login, and complete the link or code requested by the MrScraper form.
If mailbox access is unavailable, leave that verification step pending.
Use the account's displayed sign-in flow; no separate AgentMail OAuth
endpoint is assumed here.

### Step 4: Store the credential and verify access

CLI browser login saves its dedicated key in `~/.mrscraper/auth.json`.
For an SDK, CI process, or direct HTTP client, supply the key through its
environment or secret storage. Authenticate MCP through its connection;
do not pass keys in tool arguments.

```bash
mrscraper auth status --json
mrscraper status --json
```

The second command must succeed before calling CLI setup verified. A
configured credential alone does not confirm that it is valid.

---

## Path E: Use MrScraper Without Installing Anything

Use direct HTTP for session work or application code when a CLI or SDK is
unnecessary. Configure `MRSCRAPER_API_KEY` using Path D first.

Scraping and search use `https://api.mrscraper.com`. Send the key in the
`x-api-token` header, never as a query parameter. Status uses the separate
account endpoint shown below.

### Available services

| Service         | HTTP request selection                                                                        |
| --------------- | --------------------------------------------------------------------------------------------- |
| Status          | `GET https://api.app.mrscraper.com/api/v1/subscription-accounts`                              |
| Scrape HTML     | `GET /` with `url`, `html=true`, `markdown=false`, `screenshot=false`                         |
| Scrape Markdown | `GET /` with `url`, `html=false`, `markdown=true`, `screenshot=false`                         |
| Get screenshot  | `GET /` with `url`, `browserRendering=true`, `screenshot=full`, `html=true`, `markdown=false` |
| Scrape brand    | `GET /` with `url`, `action=brand-scraping`, `bundle=none`                                    |
| Scrape JSON     | `GET /` with `url`, `agent=general`, `prompt`, `html=false`, `screenshot=false`               |
| Web search      | `GET /` with `query`, `country=us`, `numResults=10`                                           |
| Workflow        | `POST /` with a JSON body holding `url` and a `workflow` array; see Path H                     |

Retrieve Markdown:

```bash
curl --get 'https://api.mrscraper.com' \
  -H "x-api-token: $MRSCRAPER_API_KEY" \
  --data-urlencode 'url=https://example.com' \
  --data-urlencode 'html=false' \
  --data-urlencode 'markdown=true' \
  --data-urlencode 'screenshot=false' \
  --data-urlencode 'warmStart=true' \
  --data-urlencode 'maxRetries=1'
```

Search:

```bash
curl --get 'https://api.mrscraper.com' \
  -H "x-api-token: $MRSCRAPER_API_KEY" \
  --data-urlencode 'query=site:example.com documentation' \
  --data-urlencode 'country=us' \
  --data-urlencode 'numResults=10'
```

The direct response is the API body. CLI and SDK `status_code`/`data`/`headers`
wrappers are added by those clients. Preserve actual response names and
types rather than constructing a wrapper around an assumed JSON shape.
For long JSON-extraction requests, the SDKs switch to POST when the encoded
URL exceeds 8,000 characters, moving `url`, `prompt`, and `agent` to the JSON
body while retaining other controls in the query.

### Documentation and references

- [API Reference](https://docs.mrscraper.com/docs/api/overview)
- [General Agent](https://docs.mrscraper.com/docs/features/ai-scraper/general)
- [Workflow](https://docs.mrscraper.com/docs/features/workflow)
- [CLI](https://docs.mrscraper.com/docs/getting-started/cli)
- [MCP setup](https://docs.mrscraper.com/docs/getting-started/mcp-server)
- [x402 integrations](https://docs.mrscraper.com/docs/integrations/x402)

---

## Path F: x402 Access Without a MrScraper API Key

x402 provides paid access without a MrScraper API key. Select a marketplace
and inspect its current contract, supported inputs, and payment requirements
before making a paid request. A funded wallet or provider balance is still
required; the payment route is not a free-access tier.

| Provider          | Discovery and contract                                                                                                                                                      | Service routes                                                                     |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Inflowpay         | [MrScraper listing](https://directory.inflowpay.ai/services/3ce219dc-ec6b-4e74-9d71-281d3645166a/view/) · [OpenAPI](https://docs.mrscraper.com/x402/inflowpay/openapi.json) | Six `GET /v1/inflowpay/<service>` routes                                           |
| Agentic Market    | [MrScraper listing](https://agentic.market/services/api-mrscraper-com) · [OpenAPI](https://docs.mrscraper.com/x402/agentic-market/openapi.json)                             | Six `GET /v1/cdp/<service>` routes                                                 |
| Circle for Agents | [MrScraper listing](https://agents.circle.com/services/mrscraper) · [Integration guide](https://docs.mrscraper.com/docs/integrations/x402/circle)                           | `POST /circle/api/mrscraper/scraperAPI` and `POST /circle/api/mrscraper/webSearch` |

All routes above use `https://api.mrscraper.com`. For the six GET routes,
`<service>` is `scrape-html`, `scrape-markdown`, `get-screenshot`,
`scrape-brand`, `scrape-json`, or `web-search`. Account `status` belongs to
the authenticated account interface and is not included in these paid
marketplace routes.

Circle's `scraperAPI` combines page outputs such as HTML, Markdown, and
screenshots in one request. Its input schema differs from the six GET
routes. Select the contract for the actual route being called rather than
translating CLI flags into a payment-provider request blindly.

For a paid call:

1. Use the chosen provider's compatible x402 client and the wallet or
   balance already authorized for the task.
2. Submit a valid request to the selected route. An unpaid request can
   return HTTP `402` with payment requirements.
3. Read the current amount, network, asset, and recipient from the live
   challenge. Keep payment within the user's authorized budget.
4. Let the payment client sign and resubmit the same request using the
   required payment headers. Keep wallet keys and signing material private.
5. Inspect both payment outcome and service result. A paid request can
   still have an extraction or upstream error. Keep receipts when the
   workflow needs accounting, and bound retries that could create another
   payment.

Use the live challenge for pricing and payment details. Consult the
provider's contract for any asynchronous job or polling flow. If a payment
setup or spending authorization is missing, report that step as pending.

---

## Path G: Native MCP Connections

Use this path to connect MrScraper in the active assistant or coding
platform. The CLI skill pack alone does not establish that connection.

1. Inspect existing connections and the tools actually available in the
   current session. Reuse a working MrScraper connection.
2. Use the platform's supported MrScraper plugin or connector when present.
   For a custom MCP connection, enter the hosted URL exactly as
   `https://mcp.mrscraper.com/mcp` and choose Streamable HTTP.
3. Complete the platform's browser OAuth flow. The seven-service server
   uses `account:read` for status and `scrape:write` for scraping/search.
4. Reload or reconnect when the platform requires it, then inspect the
   tool list and call status. Verify a representative retrieval or search
   when that access is part of the requested setup.
5. Report which connection was tested and any step still pending. Mark
   scraping/search ready only after its own successful call.

A typical hosted configuration is:

```json
{
  "mcpServers": {
    "mrscraper": {
      "url": "https://mcp.mrscraper.com/mcp"
    }
  }
}
```

When OAuth is unavailable, use the client's secret mechanism to provide
`Authorization: Bearer YOUR_TOKEN`. For local stdio, launch
`npx -y @mrscraper/mcp@latest` with `MRSCRAPER_API_KEY` in its environment.
Use the client's configuration format; the
[MCP repository](https://github.com/mrscraper-com/mrscraper-mcp) contains the
seven-service package configuration and examples.

Some existing hosted connections expose the combined `scraperAPI` tool
alongside `webSearch` and `status`. Inspect that connection's advertised
schema and use the available outputs. To use separate service tools,
use a package release or source checkout that exposes the seven names in
Path A. Installing `@latest` alone does not establish that all seven tools
are deployed on the hosted server.

---

## Path H: Browser Workflows over HTTP

A workflow runs browser steps on the target page in order, then returns
what its extraction steps collected. Use it when a page needs interaction
first (a search form, a "load more" button, a cookie banner), when data
sits in a script variable or a network response, or when prompt-directed
JSON extraction is not precise enough. Workflows are an HTTP feature; the
seven services in Path A do not expose them. Configure `MRSCRAPER_API_KEY`
using Path D first.

Send a `POST` to `https://api.mrscraper.com` with a JSON body holding the
target `url` and a `workflow` array. Each step is
`{"type": "<step type>", "data": {...}}`:

```bash
curl -X POST 'https://api.mrscraper.com/?proxyCountry=us&super=true' \
  -H "x-api-token: $MRSCRAPER_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://books.toscrape.com",
    "workflow": [
      {
        "type": "follow_urls",
        "data": { "name": "book_pages", "selector": "article.product_pod h3 a", "concurrency": 3 }
      },
      {
        "type": "extract",
        "data": { "name": "title", "selector": ".product_main h1", "selector_type": "css", "extraction_type": "text", "extraction_quantity": "first" }
      },
      {
        "type": "extract",
        "data": { "name": "price", "selector": ".product_main .price_color", "selector_type": "css", "extraction_type": "text", "extraction_quantity": "first" }
      }
    ]
  }'
```

The response holds the extracted values under `data`, keyed by step
`name`. `follow_urls` runs every later step on each followed page.

### Step types

| `type`           | What it does                                          | `data` fields                                                                                                                                                         |
| ---------------- | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `extract`        | Reads elements from the page.                         | `name`, `selector`, `selector_type` (`css`, `xpath`, `regex`), `extraction_type` (`text`, `html`, `outer_html`, `attr`), `attribute` (with `attr`), `extraction_quantity` (`first` or `all`) |
| `click`          | Clicks an element.                                    | `selector`                                                                                                                                                            |
| `input`          | Types into a field.                                   | `selector`, `text`                                                                                                                                                    |
| `delay`          | Waits a fixed time.                                   | `duration` (ms)                                                                                                                                                       |
| `wait`           | Waits for a selector or text, up to a timeout.        | `selector`, `text`, `timeout` (ms)                                                                                                                                    |
| `scroll`         | Scrolls to the end, to a text, or to an element.      | `type`, `text`, `selector`, `selector_count`, `load_more`, `load_more_selector`                                                                                       |
| `script`         | Runs JavaScript in the page and returns its value.    | `name`, `code` (an IIFE; may be `async`), `timeout`                                                                                                                   |
| `listen_network` | Captures matching network responses.                  | `name`, `url_patterns`, `excluded_url_patterns`, `timeout`, `listener_type` (`response`)                                                                              |
| `follow_urls`    | Opens each matched link and runs the remaining steps. | `name`, `selector`, `concurrency`                                                                                                                                     |

Guidance:

- Prefer `wait` over `delay`; it continues as soon as the content appears.
- Test a `script` step in a browser console first and return plain JSON,
  for example `(() => window.__DATA__)()`.
- Set `"homePage": true` in the body to visit the site's home page before
  the target URL when a site blocks direct visits to deep pages.
- `super=true` routes the run through a real device for complex sites;
  `proxyCountry` picks the country to load the page from.
- A workflow saved in the dashboard can be copied as JSON from its
  **⋮ → Manual Scraper API Access** menu.
- Workflows bill by runtime and bandwidth; the `token_usage` response
  header reports the tokens used, and `x-status-code` reports the target
  page's own HTTP status. Bound retries with `maxRetries` and `tokenCap`.

See the [Workflow guide](https://docs.mrscraper.com/docs/features/workflow)
for step details and examples.
