The Product Matching Benchmark: How AI Handles Messy Supplier Catalogs (Methodology)

Most product matching claims cannot be checked, because the data, the labels and the pass criteria are not shown. Demos tend to use clean examples (why product matching demos fail in production covers the pattern), and a single accuracy number hides the errors that hurt most. This page describes how we intend to measure, so the measurement can be challenged before it exists. It complements the buyer-side view in how to evaluate product matching software and the internal practice in the product matching regression test set playbook.

Status and scope

Question the benchmark answers

Given two or more messy supplier catalogs, how reliably does a system decide whether records refer to the same product, a variant, a replacement, or an unrelated item, and how much of that work can it complete without human review at a precision level the buyer chooses in advance? It does not measure enrichment quality, classification, price or business outcomes.

Dataset design

• Source mix: supplier catalogs from distribution domains with identity-hard data (for example fasteners, electrical components, industrial supplies). Each source is recorded with its origin, permission status and collection date. Where real data cannot be shared, a synthetic set is built and labelled as synthetic; synthetic and real results are never pooled.
• Unit of evaluation: the record pair, drawn from a candidate set, plus the cluster of records per true product for cluster-level metrics.
• Candidate generation: a blocking step shared by all systems, so recall differences come from matching, not from which pairs each system was allowed to see. Blocking recall is reported on its own.
• Relationship types: same product, variant of, replacement for, different product, and cannot decide. The distinction follows variant vs duplicate vs replacement.

Sampling

Stratified sampling over the slices below, with the strata and their sizes fixed before labelling. Hard cases are deliberately over-represented, so absolute rates on the benchmark are not estimates of production rates; only a separately drawn, unweighted random sample can estimate those, and it is reported separately if collected. The sampling frame, seed and exclusions are recorded.

Label standard and adjudication

• A written labelling guide defines each relationship type, which attributes are identity-defining per category, and what counts as sufficient evidence. Evidence sources allowed: the supplier records themselves, linked datasheets and drawings, and manufacturer pages cited by URL.
• Every pair is labelled independently by at least two trained labellers who have not seen any system output.
• Disagreements are adjudicated by a third senior reviewer; adjudicated outcomes become ground truth, and the original disagreement is retained.
• Inter-annotator agreement is reported (with the statistic named) before and after the labelling guide is frozen.
• cannot decide is a valid label. Pairs labelled this way are excluded from precision/recall and reported as a count, so uncertainty is not silently converted into errors.
• Labellers and adjudicators are domain-qualified for the category; their qualifications and number are disclosed.

Baselines

All baselines are defined before the first run and run on identical candidate sets.

Configuration effort and any vendor assistance are recorded, because a system that needs a week of tuning per catalog is a different product from one that does not.

Contamination precautions

• Split by source and by product family, not by pair, so near-identical records do not appear on both sides of the split.
• The test split is frozen before any system is configured and is never used for tuning thresholds or prompts. Thresholds are set on a separate calibration split.
• Records and labels are not published in a form that could end up in training data of the systems or models under test before the run completes; after release, the test set is versioned and a hold-out refresh is planned.
• Systems under test are asked, and where possible verified, not to have been trained on the labelled records.
• Any use of the test split by the benchmark authors for development is logged, and affected slices are reported as contaminated.

Metrics and denominators

Every metric is reported with its numerator, denominator and an interval method named in advance.

Cluster-level metrics are computed on the transitive closure of decisions, because one wrong link can merge two large groups. No composite single score is published; results are reported per slice.

Failure taxonomy

Each error is assigned one primary class by the adjudicators.

Evaluation slices

Results are reported separately for each slice; no pooled headline.

Documented failure cases we will test

These pages describe failure modes the dataset is built to include: pack-size conflicts, language and abbreviation variants, contradictory datasheet values, and matching with no shared GTIN.

Limitations

• Hard-case over-sampling means benchmark rates do not predict production rates.
• Domains covered will be a small fraction of all catalog types; results will not generalize to categories absent from the data.
• Ground truth is produced by human judgment and can be wrong; residual label noise will be estimated and reported.
• Vendor configuration effort cannot be fully equalized.
• Models and systems change; results are tied to named versions and dates.
• The benchmark is run by a vendor that sells matching software. Independent replication, and the disclosure of how each system was configured, are therefore part of the release requirements.

Pre-release sign-offs

Before any result is published, each of these is recorded:

• Data owners confirm permission to use and, where applicable, publish each source.
• The labelling lead signs off the final guide, agreement statistics and adjudication log.
• An independent reviewer outside the team checks that the test split was never used for tuning.
• Each system under test has had its configuration reviewed by its operator, and has been offered a right of reply on factual errors.
• Legal and privacy review of any released records.
• The metrics and slices published match this protocol, or every deviation is listed.

What happens next

Results will be published only after the sign-offs above are complete, with the data card, configurations and deviations from this protocol listed. Until then this page is the full public content of the benchmark.

Related

FAQ

Has the product matching benchmark been run? +

No. This page publishes the protocol before any run, so that the dataset design, labels, metrics and pass criteria are fixed in advance. No benchmark has been run and no performance numbers are published here.

Why publish a methodology before results? +

Declaring the protocol first prevents the metrics, slices and thresholds from being chosen after the results are known. It also lets buyers and other vendors challenge the design, and run the same protocol on their own data.

Why report false-merge rate separately from precision? +

Precision pools every kind of error. A false merge combines two different products into one record, which can corrupt pricing, stock and orders, so it is reported on its own with its denominator and severity, not averaged into a single score.

What is automation coverage at a predeclared precision threshold? +

It is the share of candidate pairs a system resolves without human review while staying at or above a precision level fixed in advance, for example the fraction of pairs auto-approved when the system is tuned to meet the threshold on a held-out calibration set. It is reported with the threshold and the denominator, never alone.