Remove Unicode combining marks from text

Accented letters such as é, ñ, and ü become their base forms, while punctuation, spacing, symbols, letter case, and unrelated scripts remain intact. The result is useful when a search index, identifier, filename, or URL slug needs an accent-insensitive representation without applying broader transliteration or language-specific spelling rules. Processing is deterministic, local to the request, and returns a simple text value that can be copied or passed directly into another normalization step.

What the transformation does

Unicode can represent many accented letters in more than one way. A character such as é may be stored as one precomposed code point, or as a base letter e followed by a combining acute accent. Visual comparison alone cannot reveal which representation is present. This capability first applies canonical Unicode decomposition, commonly called NFD, so characters with canonical decompositions are expressed as base characters followed by their marks. It then removes characters in every Unicode mark category. The result turns examples such as café, piñata, and Ångström into cafe, pinata, and Angstrom. It does not lowercase text, collapse whitespace, remove punctuation, or replace symbols. It also does not promise full ASCII transliteration: letters without a canonical decomposition, such as ł, remain unchanged. That narrow behavior is intentional. It gives callers a predictable normalization primitive that can be combined with separate lowercase, punctuation, or slug formatting rules without hiding extra transformations inside one operation.

Building stable search keys and slugs

Accent-insensitive search often works best when the original value and a normalized key are stored separately. Keep the source spelling for display, then pass it through this capability to produce a comparison field. A query can receive the same treatment before matching, allowing a person who types cafe to find café without losing the correctly accented label shown in results. Slug pipelines can use the output as an early stage: remove marks first, then lowercase, replace spaces, and apply the project’s own punctuation policy. Ordering matters because decomposition exposes marks that would otherwise remain attached to precomposed letters. The operation is deterministic, so identical Unicode input always produces identical output and can safely be used in cache keys or repeatable data preparation. However, normalization is not a substitute for locale-aware collation. Languages disagree about whether accented letters are variants or distinct letters, so applications should retain the original text and choose accent-insensitive matching only where that behavior fits the product and its users.

Boundaries, scripts, and verification

The removal rule covers combining marks across Unicode rather than only the familiar accent range used by Western European languages. That makes it technically consistent, but it also means the effect can be substantial in writing systems where marks carry essential information. Test representative content from every language your application accepts before applying the result to search, routing, or identity fields. Emoji variation selectors and other characters classified as marks may also be removed, so the output should be treated as a normalized key rather than a display-perfect copy. Canonical decomposition is deliberately different from compatibility decomposition: decorative or compatibility forms are not broadly flattened, and the tool avoids pretending to be a general transliterator. For verification, compare both precomposed and decomposed spellings of the same sample and confirm that their results match. Also include punctuation, non-Latin text, emoji, and letters without decompositions in your tests. Those cases reveal whether this focused operation is sufficient or whether your pipeline needs additional, explicitly chosen normalization stages.