High-throughput HTTP scrapes when the markup is already in the first response
Website Crawler
Pull static HTML and API-backed pages over BrowserSolver private routing. Control redirects, keep parsers simple, and escalate to Rendering or WebUnlocker only when the cheap lane fails. One API key across every scrape mode.
- Fastest credit lane for pages that do not need JavaScript
- Follow or stop redirect chains when mapping soft blocks
- Same credential as Rendering, Screenshot, and WebUnlocker
- HTML ready for XPath, CSS selectors, or LLM chunkers
1×
credit lane
HTTP
static scrape
302
redirect control
Bulk
job ready
From URL list to parser-ready HTML
- Step 1
Queue URLs
Your worker owns the crawl loop, retries, and storage.
- Step 2
Fetch with Crawler
BrowserSolver returns page bodies over private egress.
- Step 3
Escalate on failure
Empty DOM or blocks? Switch to Rendering or WebUnlocker on the same key.
The cheap lane in a scraping stack
Most pipelines mix cheap static scrapes with heavier browser modes. BrowserSolver keeps the HTTP lane fast so you spend credits where JavaScript or vendor defenses actually show up.
- Follow or stop at redirects when mapping chains or catching soft blocks
- HTML ready for XPath, CSS selectors, or LLM chunkers without browser overhead
- Rate limits that scale with plan tier for steady bulk extraction
- One key shared with Rendering and WebUnlocker for clean escalation
Output shaped for parsers and warehouses
Responses arrive as HTML you can feed straight into extractors. No browser overhead when the markup is already in the first response.
- Stable schema documented in OpenAPI with sample payloads
- Switch to Website Rendering when the target hides data behind client-side JS
- Combine with Website Technology or AntiBot when you need stack context first
- Markdown endpoint twins for agents and IDE copilots
Start with HTTP, escalate when needed
Use crawler mode for bulk extraction, then move up to render or unlock only on URLs that demand it.
Redirect control
Follow chains or stop at the first hop.
Parser-ready HTML
Ship page bodies straight into your extractors.
Static HTML
Best fit when content is in the first response.
Throughput
Parallel fetches from your job runner.
Fastest scrape lane
Best fit when content is in the first response.
Your pipeline
You schedule URLs; we return page bodies.
Cheap lane
Best fit when content ships in the first response
Redirects
Inspect or follow hop chains per call
One key
Share credits with render and unlock modes
Parser-ready
Raw HTML for your extractors, not ours
Frequently asked questions
Credits, scrape types, and billing before your first production job.
When should I use Crawler instead of Rendering?
Use Crawler when the data is in the initial HTML or JSON response. Switch to Website Rendering when the page hydrates fields with client-side JavaScript.
Does Crawler bypass Cloudflare?
No. Crawler is the HTTP lane. For vendor-hardened allowlisted domains, use Website WebUnlocker after AntiBot confirms the stack.
Who owns the crawl loop?
You do. BrowserSolver fetches each URL you send. Scheduling, dedupe, and warehouses stay in your stack.
How does billing work?
Successful Crawler calls use the lowest credit cost in the scrape ladder. Failed attempts follow the published billing rules in docs.
Pick your next move
Wire up an integration, review plans, or read the API reference before your first job.
Integrations
Trigger BrowserSolver from n8n, Zapier, Make, or any stack that speaks HTTP.
Scrape your first blocked URL
Fetch, render, screenshot, or unlock from one dashboard. Proxies, browsers, and anti-bot lanes stay on our side.
- 500 signup credits
- No card to start
- All scraping modes