Crawl a website
Point it at a starting URL and it follows links outward across the whole site, discovering pages you didn't already have a list of, while reading and obeying robots.txt the way any well-behaved crawler should. It's the endpoint for when the question isn't "what's on this one page" but "what does this entire site actually contain."
When you need the whole map, not one pin on it
Single-page scraping assumes you already know which URL holds the answer; crawling is for when you don't — auditing every page of a site before a migration, discovering every product URL a competitor has live, or building a site map from scratch because the sitemap.xml is stale or missing. It trades the speed of a single request for coverage across an entire domain.
How the crawl actually proceeds
A call to POST /web/crawl with a start URL and crawl parameters — depth, page limit, URL patterns to include or exclude — returns a task_id right away, then the crawler works outward link by link, fetching robots.txt first and skipping any path it disallows, exactly as a legitimate crawler is expected to. As pages are discovered and processed, they accumulate into a single structured result delivered to your signed webhook or a signed link once the crawl finishes or hits its configured limit.
robots.txt: a 1994 handshake that still governs the web
The Robots Exclusion Protocol dates to 1994, drafted informally by webmasters who wanted a machine-readable way to tell automated visitors which paths were off-limits, and it has never needed a formal standards body to remain the accepted norm. Respecting it isn't optional here — it's read before the first page beyond the start URL is ever fetched, and disallowed paths are skipped rather than silently crawled, which is both the ethical baseline and, often, the difference between a crawl that stays welcome on a site and one that gets blocked.
What the result actually contains
You get back a structured list of every page the crawl visited, each with its URL, status code and extracted content, plus the link graph the crawler used to discover them — enough to reconstruct the shape of the site, not just its text. Because a page that fails to load is retried automatically up to three times before being marked failed rather than silently dropped, and because pricing is per page actually crawled, a handful of slow or broken pages won't quietly inflate your bill or leave gaps you don't know about.
Where crawling fits into automation
Crawl results feed naturally into content audits, broken-link sweeps, migration checklists and SEO structure reviews — anywhere the deliverable is a full inventory rather than a single answer. Set a depth and page limit that matches the job: a shallow, narrow crawl for a quick section check, a deep one with generous limits when the goal is a genuinely complete map of a site before a redesign or acquisition.
What you can do with it
Pre-migration content audit
Crawl a site before a platform migration to get a complete inventory of every live URL, so nothing gets silently dropped in the move.
Competitor site mapping
Discover every product or category page a competitor has published, even ones not linked from their main navigation.
Rebuilding a missing or stale sitemap
Crawl a site whose sitemap.xml is outdated or absent to reconstruct an accurate, current list of its pages.
Site-wide broken link and status audit
Walk an entire domain and collect the status code of every page to find 404s and redirect chains before they hurt rankings.
FAQ
Does the crawler respect robots.txt?
Yes, robots.txt is fetched and checked before any path beyond the start URL is crawled, and disallowed paths are skipped rather than visited.
How much does web.crawl cost?
$0.040 per request plus $0.001 per page actually crawled, so cost scales directly with how much of the site you cover.
Can I limit how deep or how wide the crawl goes?
Yes, depth and page-count limits, along with URL include and exclude patterns, are all configurable per request.
Is there a free trial for the website crawler api?
There's no free tier — free tiers get abused and slow everyone down. Access runs on a prepaid ForHosting KIT balance: top up from $10.00 (it never expires) and each request is charged at its published price, so a call with no balance returns HTTP 402. No subscription, no tokens, no invented credits, and a failed task is never charged.
What happens to pages that fail to load during a crawl?
They're retried automatically up to three times; if they still fail, they're marked as failed in the result rather than silently omitted.
How do I get the crawl results?
As one structured result delivered to your signed webhook when the crawl finishes, or via a signed link that stays valid for 24 hours.
Can I crawl only a section of a site instead of the whole domain?
Yes, URL patterns let you constrain the crawl to a specific path, subdomain or category rather than the entire site.
How is crawling different from single-page scraping?
Scraping endpoints extract data from URLs you already know; crawling discovers those URLs for you by following links across the site, which is the point when you need coverage instead of a single lookup.
For developers — API access
Everything on this page is available programmatically. This section is for teams who want to wire it into their own systems; everyone else can just use the tool above.
API endpoint
Prefer to automate it? One authenticated POST creates the task; the result comes back by webhook or a signed link. The same capability also runs here on the web, and soon from our app, email and Telegram.
Call it from your stack
curl -X POST https://api.kit.forhosting.com/web/crawl \
-H "Authorization: Bearer $KIT_KEY" \
-H "Content-Type: application/json" \
-d '{"input":"…"}'const res = await fetch("https://api.kit.forhosting.com/web/crawl", {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.KIT_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
"input": "…"
})
});
const { task_id } = await res.json();import os, requests
res = requests.post(
"https://api.kit.forhosting.com/web/crawl",
headers={"Authorization": f"Bearer {os.environ['KIT_KEY']}"},
json={
"input": "…"
},
)
task_id = res.json()["task_id"]<?php
$res = file_get_contents("https://api.kit.forhosting.com/web/crawl", false, stream_context_create([
"http" => [
"method" => "POST",
"header" => "Authorization: Bearer " . getenv("KIT_KEY") . "\r\nContent-Type: application/json",
"content" => '{"input":"…"}',
],
]));
$task = json_decode($res, true);body := bytes.NewBufferString(`{"input":"…"}`)
req, _ := http.NewRequest("POST", "https://api.kit.forhosting.com/web/crawl", body)
req.Header.Set("Authorization", "Bearer "+os.Getenv("KIT_KEY"))
req.Header.Set("Content-Type", "application/json")
res, _ := http.DefaultClient.Do(req)Example request
{
"input": "…"
}Example response
{
"task_id": "tsk_a1b2c3d4e5f6a1b2c3d4e5f6",
"type": "web.crawl",
"status": "queued",
"_links": {
"result": "/tasks/tsk_…/result"
}
}The API is asynchronous: the call returns a task_id immediately and the result arrives by webhook. Polling is capped at 1 req/s per task.
Pricing
Published price — no tokens, no invented credits. A failed task is never charged.
Limits
timeout_sec | 30 |
max_crawl_pages | 25 |
Errors
| HTTP | Code | Meaning |
|---|---|---|
401 | unauthorized | Missing or invalid API key. |
402 | insufficient_balance | Your balance doesn't cover the task price. |
404 | unknown_type | That task type doesn't exist. |
429 | rate_limited | Too many requests. Use the webhook instead of polling. |
413 | input_too_large | The file exceeds the size limit. |
422 | task_failed | The task failed after 3 retries. You are never charged for it. |