ForHosting KIT · Data & Files

Remove duplicate rows

Merge two contact exports, re-import a file twice by accident, or aggregate feeds from several sources, and duplicate rows sneak in almost every time. This CSV deduplication API removes exact-match duplicates based on whichever columns you specify, whether that means an entire row or just an email address, keeping the first occurrence and discarding the rest.

● StableFree · in your browser

Run — free How it works & API

Use it from WebAPIEmailApp soonTelegram soon

Where duplicate rows actually come from

Duplicates rarely appear because someone typed the same row twice on purpose; they show up from re-running an import that wasn't idempotent, combining an old export with a new one that overlaps in date range, merging lead lists from two marketing tools that both captured the same signup, or a scheduled sync that fired twice after a retry. Left unremoved, duplicates inflate row counts, skew any aggregation or count-based report, and — worst of all in a contact list — mean the same person gets emailed twice.

How column-based matching works

You call POST /data/dedupe-csv with your file and a list of the columns that define a match, then get a task_id back immediately since the job runs asynchronously. If you specify every column, only rows that are identical across the entire record are removed; if you specify just an email or an order ID column, any row sharing that value with an earlier row is dropped even if other fields differ, which matters when the same customer appears twice with a slightly updated address. Matching is exact, not fuzzy, so 'Acme Inc.' and 'ACME INC' are treated as distinct unless you normalize case beforehand, typically with a clean CSV pass first.

Exact match versus fuzzy matching, and why exact comes first

Deduplication has always split into two families: exact matching, where equality is unambiguous and the result is deterministic, and fuzzy matching, where near-identical records are judged similar by some scoring heuristic that can disagree with a human reviewer. Exact, column-based dedupe is the right default for most pipelines because it never produces a false merge — two rows are only combined when they are provably the same on the columns that matter — which is why it belongs as the first pass before anyone reaches for the more judgment-heavy fuzzy techniques.

Where deduplication fits an automated pipeline

Run this step after merging files from multiple sources, after cleaning a CSV so whitespace and casing differences don't hide real duplicates, and before loading data into any system where duplicate rows cause downstream harm, such as a CRM, a billing system, or an email sending list. It composes naturally as the second step of a two-stage pipeline: clean first to normalize the data, then dedupe on the columns that define uniqueness for your use case.

Delivery, pricing, and status

The deduplicated CSV is delivered by a signed webhook, recommended for import pipelines, or a signed link valid for 24 hours; source and output are deleted after the retention window and never used for training. Pricing is a flat $0.002 per request regardless of row count, and a failed task — such as specifying a column name that doesn't exist in the file — is never charged after the standard three retries. This endpoint is live now.

What you can do with it

Merge contact lists without double-emailing anyone

Combine exports from two marketing tools and dedupe on the email column, keeping one record per address before the list reaches your sending platform.

Clean up an accidental double import

Remove exact-match duplicate rows created when the same file, or an overlapping export, got imported twice into the same dataset.

Deduplicate order records by order ID

Drop repeated rows sharing the same order ID after aggregating feeds from multiple sales channels, keeping the first record for each order.

Prepare a clean list before a CRM import

Dedupe a prospect list on company name and domain before loading it into a CRM, avoiding duplicate account records that fragment sales history.

FAQ

How do I remove duplicates from a CSV with an API?

Send the file to POST /data/dedupe-csv along with the columns that define a match; a task_id returns right away, and the deduplicated CSV arrives at your signed webhook or a signed link once processing finishes.

Is the CSV deduplication API free?

The tool above runs free in your browser. The API is paid — each call draws from your prepaid ForHosting KIT balance: top up from $10.00 (it never expires), pay each request's published price, and a call with no balance returns HTTP 402. No subscription, no tokens, and a failed task is never charged.

Can I dedupe on just one column instead of the whole row?

Yes. Specify any subset of columns, such as just an email or order ID, and rows sharing that value with an earlier row are removed even if other fields differ.

Does it do fuzzy matching for near-duplicate rows?

No, matching is exact on the columns you choose; near-duplicates that differ in spacing or case are only caught if you clean the CSV first to normalize those differences.

Which duplicate row is kept?

The first occurrence in the file is kept and later matching rows are removed, so the order of your input determines which version of a duplicate survives.

Can I deduplicate large CSV files?

Yes, within the size limits published for the endpoint; pricing stays a flat fee per request regardless of how many rows the file contains.

Can I dedupe CSVs in bulk?

Yes. Each request is an independent asynchronous task, so you can submit many files in parallel and collect each deduplicated result by webhook as it completes.

Should I clean a CSV before deduplicating it?

It's a good practice: cleaning trims whitespace and normalizes encoding first, so exact-match dedupe on a text field like email or company name catches duplicates that inconsistent formatting would otherwise hide.

For developers — API access

Everything on this page is available programmatically. This section is for teams who want to wire it into their own systems; everyone else can just use the tool above.

API endpoint

POSThttps://api.kit.forhosting.com/data/dedupe-csv

Prefer to automate it? One authenticated POST creates the task; the result comes back by webhook or a signed link. The same capability also runs here on the web, and soon from our app, email and Telegram.

Call it from your stack

curl -X POST https://api.kit.forhosting.com/data/dedupe-csv \
  -H "Authorization: Bearer $KIT_KEY" \
  -H "Content-Type: application/json" \
  -d &#39;{"text":"…"}'

const res = await fetch("https://api.kit.forhosting.com/data/dedupe-csv", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.KIT_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    "text": "…"
  })
});
const { task_id } = await res.json();

import os, requests

res = requests.post(
    "https://api.kit.forhosting.com/data/dedupe-csv",
    headers={"Authorization": f"Bearer {os.environ[&#39;KIT_KEY']}"},
    json={
    "text": "…"
},
)
task_id = res.json()["task_id"]

<?php
$res = file_get_contents("https://api.kit.forhosting.com/data/dedupe-csv", false, stream_context_create([
  "http" => [
    "method"  => "POST",
    "header"  => "Authorization: Bearer " . getenv("KIT_KEY") . "\r\nContent-Type: application/json",
    "content" => &#39;{"text":"…"}',
  ],
]));
$task = json_decode($res, true);

body := bytes.NewBufferString(`{"text":"…"}`)
req, _ := http.NewRequest("POST", "https://api.kit.forhosting.com/data/dedupe-csv", body)
req.Header.Set("Authorization", "Bearer "+os.Getenv("KIT_KEY"))
req.Header.Set("Content-Type", "application/json")
res, _ := http.DefaultClient.Do(req)

Example request

{
  "text": "…"
}

Example response

{
  "task_id": "tsk_a1b2c3d4e5f6a1b2c3d4e5f6",
  "type": "data.dedupe_csv",
  "status": "queued",
  "_links": {
    "result": "/tasks/tsk_…/result"
  }
}

The API is asynchronous: the call returns a task_id immediately and the result arrives by webhook. Polling is capped at 1 req/s per task.

Pricing

Per request$0.002

Published price — no tokens, no invented credits. A failed task is never charged.

Limits

max_mb 25

Errors

HTTP	Code	Meaning
`401`	`unauthorized`	Missing or invalid API key.
`402`	`insufficient_balance`	Your balance doesn't cover the task price.
`404`	`unknown_type`	That task type doesn't exist.
`429`	`rate_limited`	Too many requests. Use the webhook instead of polling.

Read the full KIT documentation →

Run — free