Fuzzy dedupe
Two rows can describe the same customer, product or address without a single character in common between how they were typed. This endpoint compares records by meaning and shape rather than exact string equality, flagging "Jon Smith, 4 Oak St" and "Jonathan Smith, 4 Oak Street" as the same person a human reviewer would merge on sight.
Run it online
Run this on our servers with your account. Free tools run in your browser; this one bills your KIT balance per the price above.
Why exact matching quietly fails
A plain SQL DISTINCT or a hash-based dedup only catches rows that are byte-for-byte identical, which sounds strict but is actually the weaker guarantee — it misses every duplicate introduced by human variation. Someone typed "St." instead of "Street," a form auto-capitalized a name differently on a second visit, a lead was re-entered with a middle initial the first time lacked. None of these are errors exactly; they are just the normal noise of data entered by different people at different times, and exact matching treats every one of them as a distinct, unrelated record.
How the comparison actually works
Call POST /data/dedupe-semantic with your rows and the task scores pairs of records for similarity across the fields you designate, using techniques that tolerate typos, reordered words, abbreviations and partial matches rather than demanding identical text. Instead of a binary yes-or-no, each candidate pair gets a similarity signal, and rows crossing your chosen threshold are grouped as duplicates in the response, with the rest of the file passed through untouched.
Choosing what counts as a duplicate
There is no universal answer to how similar is similar enough — a CRM merging leads might want a loose threshold that catches nicknames and typos, while a compliance list checking for sanctioned entities wants a tight one that avoids false positives. You control which fields feed the comparison and how aggressively they are weighted, so a name-and-email match can be treated differently from a name-and-birthdate match, and the task adapts to what "duplicate" actually means for your dataset rather than applying one fixed rule.
The gap fuzzy matching fills
Deterministic dedup tools have existed since the earliest relational databases, built for a world where data entry followed strict forms. Real data has never fully lived up to that assumption, and the gap between clean-in-theory and messy-in-practice is exactly where duplicate customer records, repeated support tickets and redundant leads accumulate silently for years. Semantic comparison closes that gap by approximating the judgment a careful human would make, at a scale no team could review row by row.
Fitting into a data pipeline
Because the task runs asynchronously and returns a task_id immediately, it slots into nightly CRM hygiene jobs, pre-import validation before a merge, or a one-off cleanup of an inherited spreadsheet without blocking whatever process kicks it off. Results arrive by signed webhook or a 24-hour signed link, ready to drive an automatic merge, a review queue, or a simple count of how much duplication your dataset is actually carrying.
What you can do with it
CRM lead consolidation
Catch leads entered twice under slightly different spellings before they generate two outreach sequences to the same person.
Customer record cleanup after a merger
Reconcile two companies' customer lists where the same account appears with different formatting, abbreviations or typos.
Support ticket dedup
Identify tickets describing the same issue in different words so agents stop duplicating work on a single incident.
Mailing list hygiene
Reduce duplicate sends to the same household or person before a campaign goes out, cutting cost and avoiding a spammy repeat.
FAQ
How is semantic deduplication different from exact matching?
Exact matching only catches identical strings; this fuzzy deduplication api compares meaning and structure, so typos, abbreviations and reordered fields are still recognized as duplicates.
Can I choose which columns are compared?
Yes, you specify which fields feed the similarity comparison and how they are weighted, so name-and-email matching can be tuned separately from address matching.
Can I control how strict the matching is?
Yes, you set a similarity threshold; a looser one catches more near-duplicates but risks false positives, a stricter one is safer but may miss some.
Is there a free tier to test dedupe-semantic?
There's no free tier — free tiers get abused and slow everyone down. Access runs on a prepaid ForHosting KIT balance: top up from $10.00 (it never expires) and each request is charged at its published price, so a call with no balance returns HTTP 402. No subscription, no tokens, no invented credits, and a failed task is never charged.
What happens to rows that are not duplicates?
They pass through the response unchanged; only rows crossing your similarity threshold are grouped and flagged as candidate duplicates.
How large a file can I submit for dedup?
Submit files as separate async requests, each with its own task_id, so very large datasets can be split and processed independently.
How do I get the deduplication results?
Through a signed webhook, recommended for automated workflows, or a signed link valid for 24 hours if you prefer to retrieve it manually.
What does the semantic dedup API cost?
$0.002 per request plus $0.002 per 1,000 rows processed, charged only when the task completes successfully.
For developers — API access
Everything on this page is available programmatically. This section is for teams who want to wire it into their own systems; everyone else can just use the tool above.
API endpoint
Prefer to automate it? One authenticated POST creates the task; the result comes back by webhook or a signed link. The same capability also runs here on the web, and soon from our app, email and Telegram.
Call it from your stack
curl -X POST https://api.kit.forhosting.com/data/dedupe-semantic \
-H "Authorization: Bearer $KIT_KEY" \
-H "Content-Type: application/json" \
-d '{"text":"…"}'const res = await fetch("https://api.kit.forhosting.com/data/dedupe-semantic", {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.KIT_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
"text": "…"
})
});
const { task_id } = await res.json();import os, requests
res = requests.post(
"https://api.kit.forhosting.com/data/dedupe-semantic",
headers={"Authorization": f"Bearer {os.environ['KIT_KEY']}"},
json={
"text": "…"
},
)
task_id = res.json()["task_id"]<?php
$res = file_get_contents("https://api.kit.forhosting.com/data/dedupe-semantic", false, stream_context_create([
"http" => [
"method" => "POST",
"header" => "Authorization: Bearer " . getenv("KIT_KEY") . "\r\nContent-Type: application/json",
"content" => '{"text":"…"}',
],
]));
$task = json_decode($res, true);body := bytes.NewBufferString(`{"text":"…"}`)
req, _ := http.NewRequest("POST", "https://api.kit.forhosting.com/data/dedupe-semantic", body)
req.Header.Set("Authorization", "Bearer "+os.Getenv("KIT_KEY"))
req.Header.Set("Content-Type", "application/json")
res, _ := http.DefaultClient.Do(req)Example request
{
"text": "…"
}Example response
{
"task_id": "tsk_a1b2c3d4e5f6a1b2c3d4e5f6",
"type": "data.dedupe_semantic",
"status": "queued",
"_links": {
"result": "/tasks/tsk_…/result"
}
}The API is asynchronous: the call returns a task_id immediately and the result arrives by webhook. Polling is capped at 1 req/s per task.
Pricing
Published price — no tokens, no invented credits. A failed task is never charged.
Limits
max_mb | 25 |
Errors
| HTTP | Code | Meaning |
|---|---|---|
401 | unauthorized | Missing or invalid API key. |
402 | insufficient_balance | Your balance doesn't cover the task price. |
404 | unknown_type | That task type doesn't exist. |
429 | rate_limited | Too many requests. Use the webhook instead of polling. |