Analyze robots.txt
One misplaced slash in a robots.txt disallow line can wall off an entire section of a site from search engines, and it's the kind of mistake that no one notices until traffic to that section quietly flatlines. This endpoint parses a robots.txt file rule by rule and tests it against real URLs so you know precisely what's allowed and what isn't.
Run it online
Run this on our servers with your account. Free tools run in your browser; this one bills your KIT balance per the price above.
A tiny file with outsized consequences
Robots.txt predates almost everything else in modern SEO — the Robots Exclusion Protocol was proposed in 1994, and its syntax has stayed deliberately minimal ever since: a handful of directives, no conditionals, no comments that crawlers are required to respect. That minimalism is also the danger. A single Disallow: / left over from a staging environment, a trailing slash that turns a narrow rule into a broad one, or a rule ordered incorrectly against a more specific Allow can silently deindex sections of a live site.
What the analysis covers
The task fetches the robots.txt file, parses every user-agent block and directive, and reports the effective rule set per crawler — since different user-agent groups can have different permissions, a file can behave one way for general crawlers and another for specific ones. Given a list of sample URLs alongside the file, it also runs each path against the parsed rules and reports, for each one, whether it's allowed or blocked and by which specific directive, so there's no ambiguity about which line is responsible.
The overlaps people miss
Robots.txt only controls crawling, not indexing — a blocked URL can still appear in search results without a snippet if other pages link to it, a distinction that trips up teams trying to use Disallow as a substitute for a noindex tag. It also frequently conflicts with the sitemap directive at the bottom of the same file: a sitemap listing URLs that the rules above it block is a common, self-defeating pattern the analysis flags directly.
Where mistakes come from
Most robots.txt incidents trace back to a deploy pipeline that promotes a staging robots.txt to production, or a CMS plugin that regenerates the file and reorders directives in a way that changes which rule wins for a given path. Because the file sits at a fixed, predictable URL and gets fetched by every major crawler on its own schedule, an error there can persist and compound for days before anyone checks it manually.
Verifying it automatically
Given the file's outsized blast radius, checking it after every deploy is one of the cheapest safeguards available: submit the robots.txt URL with a set of representative paths, get a task_id back immediately, and receive the per-path verdict on your webhook once processing finishes. At $0.002 per request, it costs less to verify than to explain a week of lost crawl coverage.
What you can do with it
Pre-deploy safety check
Run the analyzer against a staging robots.txt before promoting it to production to confirm it doesn't carry over a blanket Disallow.
Crawler-specific rule audit
Verify that a rule intended only for a specific bot's user-agent group isn't accidentally applying to general crawlers too.
Sitemap conflict detection
Catch a sitemap directive listing URL paths that a Disallow rule higher in the same file actually blocks.
New section rollout check
Before launching a new site section, confirm the intended URL paths resolve as allowed rather than inheriting a broad, unintended block.
FAQ
What does a robots.txt analyzer API actually do?
It parses every user-agent block and directive in a robots.txt file and, given sample URLs, tests each one against the parsed rules to report whether it's allowed or blocked and by which line.
Does blocking a URL in robots.txt remove it from search results?
Not necessarily. Robots.txt controls crawling, not indexing, so a blocked URL can still appear in results without a snippet if other pages link to it; a noindex tag is the correct tool to prevent indexing.
Can it check rules for a specific crawler user-agent?
Yes, the report is broken down per user-agent group, since the same robots.txt file can grant different permissions to different crawlers.
Is there a free trial for this endpoint?
There's no free tier — free tiers get abused and slow everyone down. Access runs on a prepaid ForHosting KIT balance: top up from $10.00 (it never expires) and each request is charged at its published price, so a call with no balance returns HTTP 402. No subscription, no tokens, no invented credits, and a failed task is never charged.
How much does one analysis cost?
$0.002 per request, only charged on a successful parse; failed fetches retry up to three times at no charge before returning an error.
How are results delivered?
By signed webhook, recommended for automated checks, or a signed link valid for 24 hours.
Can it detect a conflict between robots.txt and the sitemap it references?
Yes, it flags cases where the sitemap directive lists URLs that a Disallow rule higher in the same file blocks.
Can I test multiple URL paths against one robots.txt in a single request?
Yes, submit a list of paths alongside the robots.txt URL and the report covers each one individually.
For developers — API access
Everything on this page is available programmatically. This section is for teams who want to wire it into their own systems; everyone else can just use the tool above.
API endpoint
Prefer to automate it? One authenticated POST creates the task; the result comes back by webhook or a signed link. The same capability also runs here on the web, and soon from our app, email and Telegram.
Call it from your stack
curl -X POST https://api.kit.forhosting.com/seo/robots-analyze \
-H "Authorization: Bearer $KIT_KEY" \
-H "Content-Type: application/json" \
-d '{"domain":"ejemplo.com"}'const res = await fetch("https://api.kit.forhosting.com/seo/robots-analyze", {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.KIT_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
"domain": "ejemplo.com"
})
});
const { task_id } = await res.json();import os, requests
res = requests.post(
"https://api.kit.forhosting.com/seo/robots-analyze",
headers={"Authorization": f"Bearer {os.environ['KIT_KEY']}"},
json={
"domain": "ejemplo.com"
},
)
task_id = res.json()["task_id"]<?php
$res = file_get_contents("https://api.kit.forhosting.com/seo/robots-analyze", false, stream_context_create([
"http" => [
"method" => "POST",
"header" => "Authorization: Bearer " . getenv("KIT_KEY") . "\r\nContent-Type: application/json",
"content" => '{"domain":"ejemplo.com"}',
],
]));
$task = json_decode($res, true);body := bytes.NewBufferString(`{"domain":"ejemplo.com"}`)
req, _ := http.NewRequest("POST", "https://api.kit.forhosting.com/seo/robots-analyze", body)
req.Header.Set("Authorization", "Bearer "+os.Getenv("KIT_KEY"))
req.Header.Set("Content-Type", "application/json")
res, _ := http.DefaultClient.Do(req)Example request
{
"domain": "ejemplo.com"
}Example response
{
"task_id": "tsk_a1b2c3d4e5f6a1b2c3d4e5f6",
"type": "seo.robots_analyze",
"status": "queued",
"_links": {
"result": "/tasks/tsk_…/result"
}
}The API is asynchronous: the call returns a task_id immediately and the result arrives by webhook. Polling is capped at 1 req/s per task.
Pricing
Published price — no tokens, no invented credits. A failed task is never charged.
Errors
| HTTP | Code | Meaning |
|---|---|---|
401 | unauthorized | Missing or invalid API key. |
402 | insufficient_balance | Your balance doesn't cover the task price. |
404 | unknown_type | That task type doesn't exist. |
429 | rate_limited | Too many requests. Use the webhook instead of polling. |