# SuperScraper

> The web context API for AI agents. Search, scrape, crawl, and extract pages
> into markdown or typed JSON — with a confidence score and a source trail on
> every structured field, so an agent knows what to trust.

## Product

SuperScraper converts any URL into clean markdown or structured JSON without
requiring you to manage browsers or proxies. A fetch cascade (plain fetch →
Playwright stealth → Spider Cloud → Browserbase) escalates only as far as a
given site requires, and structured results carry `_confidence`,
`_extraction_method`, and `_provenance` so a caller can decide what to trust.

## Base URL

https://api.superscraper.dev

## Authentication

All /v1/* endpoints require an API key (keys are formatted `ss_live_…`):

  Authorization: Bearer ss_live_<your_key>

An agent can mint its own free-tier key without a human (no dashboard, no human
sign-in — this route is mounted PUBLIC, before the auth guard):

  POST /v1/keys/provision
  { "label": "research-agent" }
  → { "apiKey": "ss_live_...", "tier": "free" }

Keys are also issued from the dashboard at https://app.superscraper.dev.

## Core endpoints

POST /v1/scrape      — a URL → markdown / HTML / links / JSON / screenshot
POST /v1/search       — web search, optionally scraping each result
POST /v1/crawl        — async, recursive site crawl (BullMQ)
GET  /v1/jobs/:id     — poll an async crawl job
POST /v1/map          — discover every URL on a domain
POST /v1/extract      — free JSON-LD/Open Graph cascade, or schema-based LLM extract
POST /v1/batch        — up to 100 URLs per request
POST /v1/parse        — a PDF or DOCX (upload or URL) → markdown
POST /v1/screenshot   — full-page or viewport capture

## Response envelope

Structured results include:
  _confidence         — 0–1 float; how sure the extraction is
  _extraction_method  — "json-ld" | "opengraph" | "regex" | "llm"
  _provenance         — the fetch tier + source that produced the field

## For AI agents

- Read this file, then a page's own /llms.txt (if present) for site-specific grounding.
- Use POST /v1/scrape to read a URL as clean markdown before reasoning over it.
- Use POST /v1/extract with a JSON schema for typed structured data from any page.
- Prefer the hosted MCP server for tool-calling clients — see /mcp for the config
  and the full tool catalog (scrape, extract, crawl, batch_scrape, search,
  google_maps_search, yelp_search, screenshot, transform, unblock, list_tools).
- All error states return { error } and a non-200 status; nothing throws unhandled.

## More

- Playground (no signup): https://superscraper.dev/playground
- MCP server + tool catalog: https://superscraper.dev/mcp
- Anti-bot benchmark (dated, reproducible): https://superscraper.dev/benchmarks
- Full API reference and docs llms.txt: https://docs.superscraper.dev/llms.txt