How to Deduplicate a Product Catalog Without Merging the Wrong Parts

Two records in your ERP read 6205-2RS and 6205-2Z. One character apart. Same bore, same outer diameter, same width, same load rating. Different seal — one is contact rubber, the other is a metal shield. Merge them and you have told every downstream system that they are interchangeable, which they are not in a washdown environment.

This is the decision at the centre of every catalog deduplication project, and it is the reason “just run a matching algorithm over it” fails in industrial distribution. Claro’s matching layer exists specifically to make this decision reviewable: every candidate pair carries a confidence score and the evidence behind it, high-confidence matches write back to your ERP and PIM automatically, and the ambiguous middle goes to a person who can see exactly why it was flagged. Deduplication is not a cleanup task you finish. It is a judgment you have to be able to audit.

The two errors do not cost the same

Most deduplication tooling optimises for a single number: how many duplicates it found. That framing hides the actual risk, because the two ways of being wrong have wildly different consequences.

A missed merge leaves two records for one product. You carry the same bearing twice, so your stock position looks thinner than it is, your purchasing team buys from the second-best contract, and your reporting understates category concentration. The cost is margin leakage. It is real, it is recurring, and it is recoverable the moment you catch it.

A false merge collapses two genuinely different products into one identity. Now a requisition for the shielded variant can be filled with the sealed one. The failure shows up weeks later, in a plant, as unplanned downtime — and nobody traces it back to a catalog decision made eleven months earlier by an automated process with no record of its reasoning.

In weak-standard catalogs where part numbers differ by punctuation and attributes arrive as free text, false merges are not a hypothetical. They are the default outcome of aggressive automatic matching.

So the correct target is not “maximum duplicates found.” It is the highest recall you can achieve at a false-merge rate your operation can tolerate — and that tolerance is different for MRO spares than it is for packaging consumables.

Why part numbers alone don’t resolve identity

Teams reach for the manufacturer part number first, reasonably. It fails in three predictable ways.

Formatting variance. 6205-2RSH, 6205 2RSH, 62052RSH, and NN.6205.2RSH are one part across four supplier files. Normalising punctuation and casing is table stakes and gets you a surprising distance — but it also creates false candidates, because stripping separators makes genuinely different part numbers look closer than they are.

Supersession. The manufacturer issued a new number; your ERP still holds the old one; the regional operating company holds a third. All three are live, all three are correct, and none of them match as strings. This needs an alias graph, not a string comparison.

Missing identifiers. A supplier feed with no GTIN and a description like BEARING BALL 25MM SEALED has to be matched on attributes or not at all. This is where the technique has to shift from deterministic to probabilistic matching — and where confidence scoring stops being optional.

The three-band model

The workable structure is not a binary merge/don’t-merge. It is three bands, with an explicit threshold between each.

Band 1 — auto-merge. Deterministic agreement on a strong identifier, or probabilistic agreement above a high threshold with no contradicting attribute. Example: same GTIN, or same normalised MPN plus same manufacturer plus no conflicting dimensional attribute. These write back without human intervention. Every merge still keeps its evidence, because it may need to be reversed.

Band 2 — review queue. Strong similarity with at least one unexplained difference: matching dimensions but different seal codes, matching descriptions but a 4% price gap, matching MPN with different manufacturers listed. This is where the value is, and it is the band every product demo skips, because a queue of 400 pairs awaiting review is a worse slide than “fully autonomous.” A good queue shows the reviewer the two records side by side, the specific fields that agree, the specific fields that conflict, and the source document each value came from — so the decision takes fifteen seconds, not fifteen minutes.

Band 3 — no match. Below threshold, left alone. Re-evaluated when new attributes arrive, because a record that was unmatchable in March may be resolvable in June once its datasheet has been parsed.

The thresholds are not universal constants. Set them per category. A catalog of fasteners tolerates more automation than a catalog of hydraulic components where a wrong pressure rating is a safety event.

Deduplicate before you enrich

The most common sequencing mistake is enriching first because it produces a visible win faster.

If you fill missing attributes on a fragmented catalog, you pay to enrich the same product three times, you get three subtly different attribute sets for one physical item, and you have made the matching problem harder — because now the records disagree on values that were previously just absent. Absence is easy to reconcile. Contradiction is not.

Resolve identity first. Enrich the canonical record once. Then every enrichment you buy applies to every supplier alias underneath it. This trusted identity layer is also a prerequisite for machine-readable product data that AI agents can act on.

Before and after, on one pair

Before — two records, both live in the ERP:

After — resolved, with evidence:

The €1.75 gap was invisible while the records were separate. That is the recurring, quiet cost of duplicates — not the storage, the purchasing.

A deduplication sequence that holds

1. Pick one category, not the whole catalog. Highest supplier overlap, highest spend. A working loop in one category beats a stalled migration across all of them.
2. Normalise before you match. Units, casing, punctuation, and separators. Skipping this inflates both misses and false candidates.
3. Build the alias graph. Supersessions, regional numbers, and internal legacy codes are aliases of one identity, not duplicates to be deleted.
4. Set thresholds per category, in writing. If nobody can state the threshold, nobody can defend a merge later.
5. Staff the review queue before you need it. Twenty minutes a day beats a 2,000-pair backlog nobody opens.
6. Keep every merge reversible. Store the constituent records and the evidence. Some merges will be wrong; the question is whether you can find and undo them.
7. Re-run continuously. New supplier files create new duplicates every month. A one-off deduplication project is a catalog that starts degrading the day it ends.

None of this replaces your ERP or PIM. It decides which records they should have been holding in the first place, and writes the answer back.

FAQ

What's the difference between deduplication and entity resolution? +

Deduplication removes redundant records. Entity resolution decides which records refer to the same real-world product and links them under one identity, keeping every supplier alias mapped rather than deleted. In multi-supplier catalogs you want resolution — you still need to receive files that use the old part numbers.

Should product matching be fully automatic? +

Only above a confidence threshold you have set deliberately. High-confidence matches on strong identifiers can write back without review. Ambiguous pairs should go to a person, because a false merge in industrial supply can put the wrong part into a maintenance job.

How do you match products with no GTIN? +

On normalised part numbers plus attributes — dimensions, material, rating, manufacturer — extracted from descriptions and datasheets. This is probabilistic, so it produces scores rather than verdicts, which is precisely why the review band exists.

Should we deduplicate or enrich first? +

Deduplicate first. Enriching a fragmented catalog means paying to enrich the same product several times and creating attribute contradictions that make later matching harder.

Is this a one-time project? +

No. Every new supplier file and every supersession creates new duplicates. Deduplication has to run as a continuous loop against incoming files, or the catalog degrades back to its starting state within a few quarters.

See the duplicates in your own catalog

Claro resolves supplier records into one trusted product identity, scores every match with the evidence behind it, routes the ambiguous ones to review, and writes the resolved record back into your ERP and PIM — continuously, as new files arrive.

Book a 30-minute catalog review →

Bring one problem category and a couple of supplier files. We’ll show you the duplicate pairs, the confidence bands, and which ones you’d want a human to look at.

Your catalog should get smarter every day.