ForHosting KIT · Semantic Search & RAG

Index a website

A website is content scattered across hundreds of URLs, and no visitor is going to click through all of them to find one answer. This endpoint crawls a site starting from a given address and turns its pages into an index you can query by meaning instead of by sitemap. Give it a starting URL and let it do the walking.

● StablePer request + per page$0.040

Run — $0.040 + $0.0025/page How it works & API

Use it from WebAPIEmailApp soonTelegram soon

Run it online

Run this on our servers with your account. Free tools run in your browser; this one bills your KIT balance per the price above.

The gap between a sitemap and an answer

A site can have excellent content and still be hard to use if the only way to find something is a rigid category menu or a search box that matches literal words. Someone looking for 'how do I change my billing cycle' should find that page even if it's titled 'subscription management,' and that's a mismatch a sitemap alone can't fix. search.index_url crawls the actual pages and indexes what they say, not just where they live in a folder structure.

What the crawl actually does

POST /search/index-url with a starting address, get back a task_id, and receive confirmation by webhook or a signed 24-hour link once the crawler has followed links from that page, extracted the readable content of each one, split it into fragments, embedded it, and added it to your searchable index. You don't hand-list every page — the crawl discovers them the way a visitor following links would.

Crawling is an old problem with new stakes

Automated web crawling is as old as search engines themselves, going back to the first web crawlers of the early 1990s that existed purely to discover what pages existed at all. What's changed since then isn't the basic mechanics of following links — it's what happens after the page is fetched: instead of just indexing keywords, the content now gets converted into vectors so it can be matched by meaning rather than exact phrasing.

What stays out of scope

This indexes readable page content, not content locked behind a login, generated purely by client-side interaction after complex user input, or blocked by the site's own crawling rules. It's built for public or accessible content — documentation, marketing pages, knowledge bases, blogs — not as a way around access controls a site has deliberately put in place.

Where it plugs into a real workflow

Companies index their own documentation site so a support assistant can answer from the current published content instead of a stale copy someone maintains separately. Marketing teams index a competitor's public blog to search across months of posts for how a topic has been covered over time. Priced per request plus per page, indexing a ten-page marketing site and a two-hundred-page documentation site both cost exactly what the crawl actually covers.

What you can do with it

Documentation-powered support assistant

A company indexes its own docs site so a chat assistant answers from the live, published pages instead of a manually maintained copy that drifts out of date.

Competitive content research

A marketing team indexes a competitor's public blog to search across its history for how a specific topic or feature has been discussed over time.

Internal portal search upgrade

A company indexes its intranet's public-facing pages to replace a keyword search box that only matches exact page titles.

Site content audit

A content team indexes their own site to search for pages covering a topic that may need consolidating or updating after a rebrand.

FAQ

How do I index a website for search with the API?

Send a starting URL to POST /search/index-url, store the returned task_id, and get confirmation the site is indexed by webhook or a signed link valid for 24 hours.

Is the website indexing API free?

No, there's no free tier or trial; it costs $0.040 per request plus $0.0025 per page, and a failed task is never charged.

How many pages will it crawl from one starting URL?

It follows links outward from the starting page up to the current crawl limits; check the endpoint reference for the exact page and depth caps.

Can it index pages behind a login?

No, it indexes publicly accessible pages; content requiring authentication or blocked by the site's crawling rules is out of scope.

How is this different from search.index_text or search.index_document?

search.index_text and search.index_document take content you already have; search.index_url discovers and fetches the content itself by crawling from a URL.

Will it re-index a site if the content changes?

Each call is a fresh crawl as of that moment; to keep an index current, submit a new crawl on whatever schedule matches how often the site changes.

Can I index multiple websites in one job?

Submit one starting URL per crawl task, and run multiple async tasks in parallel for multiple sites, collecting each result by webhook as it completes.

Is the crawled content stored permanently?

No, crawled content and results are deleted after the retention window and are never used for training.

For developers — API access

Everything on this page is available programmatically. This section is for teams who want to wire it into their own systems; everyone else can just use the tool above.

API endpoint

POSThttps://api.kit.forhosting.com/search/index-url

Prefer to automate it? One authenticated POST creates the task; the result comes back by webhook or a signed link. The same capability also runs here on the web, and soon from our app, email and Telegram.

Call it from your stack

curl -X POST https://api.kit.forhosting.com/search/index-url \
  -H "Authorization: Bearer $KIT_KEY" \
  -H "Content-Type: application/json" \
  -d &#39;{"input":"…"}'

const res = await fetch("https://api.kit.forhosting.com/search/index-url", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.KIT_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    "input": "…"
  })
});
const { task_id } = await res.json();

import os, requests

res = requests.post(
    "https://api.kit.forhosting.com/search/index-url",
    headers={"Authorization": f"Bearer {os.environ[&#39;KIT_KEY']}"},
    json={
    "input": "…"
},
)
task_id = res.json()["task_id"]

<?php
$res = file_get_contents("https://api.kit.forhosting.com/search/index-url", false, stream_context_create([
  "http" => [
    "method"  => "POST",
    "header"  => "Authorization: Bearer " . getenv("KIT_KEY") . "\r\nContent-Type: application/json",
    "content" => &#39;{"input":"…"}',
  ],
]));
$task = json_decode($res, true);

body := bytes.NewBufferString(`{"input":"…"}`)
req, _ := http.NewRequest("POST", "https://api.kit.forhosting.com/search/index-url", body)
req.Header.Set("Authorization", "Bearer "+os.Getenv("KIT_KEY"))
req.Header.Set("Content-Type", "application/json")
res, _ := http.DefaultClient.Do(req)

Example request

{
  "input": "…"
}

Example response

{
  "task_id": "tsk_a1b2c3d4e5f6a1b2c3d4e5f6",
  "type": "search.index_url",
  "status": "queued",
  "_links": {
    "result": "/tasks/tsk_…/result"
  }
}

The API is asynchronous: the call returns a task_id immediately and the result arrives by webhook. Polling is capped at 1 req/s per task.

Pricing

Per request$0.040

Per page$0.0025

Published price — no tokens, no invented credits. A failed task is never charged.

Limits

`max_chunks`	10000
`max_tokens`	20000

Errors

HTTP	Code	Meaning
`401`	`unauthorized`	Missing or invalid API key.
`402`	`insufficient_balance`	Your balance doesn't cover the task price.
`404`	`unknown_type`	That task type doesn't exist.
`429`	`rate_limited`	Too many requests. Use the webhook instead of polling.
`413`	`input_too_large`	The file exceeds the size limit.
`422`	`task_failed`	The task failed after 3 retries. You are never charged for it.

Read the full KIT documentation →