Build a Product Matching Regression Test Set

A product matching system can report 93% accuracy and still merge a 12-pack with a single unit, confuse two voltage variants, or link a merely compatible substitute to the wrong canonical SKU. An aggregate score hides the individual failures that create incorrect prices, inventory, compliance claims, and purchase orders.

A regression test set makes quality concrete. It is a versioned collection of product pairs or candidate sets with reviewed expected outcomes. Every time a person accepts a difficult match, rejects a tempting one, separates a false merge, or identifies a supplier alias, that exception becomes a future test case. Whenever the model, prompt, threshold, retrieval strategy, normalization logic, or business rule changes, replay all of those cases. This is AI matching regression testing as an operating discipline, not a one-off model assessment.

Continuous product matching needs continuous evaluation.

That is a stronger production guarantee than a one-time claim that a model is “93% accurate.” The percentage describes one measurement on one sample. A maintained benchmark tells you whether the exact failure modes your business has already paid to discover will return in the next release.

What belongs in the regression set

Do not build the benchmark from easy exact-GTIN matches alone. Preserve the decisions that expose the boundary between “same product,” “different product,” and “valid substitute.” At minimum, capture these case types:

Include ordinary cases too. A suite made entirely of edge cases can detect regressions but cannot estimate production performance. Maintain a representative evaluation sample for overall measurement and a growing “never break again” suite for known exceptions. Report them separately so a flood of easy cases cannot conceal a critical regression.

Build the product matching benchmark

1. 1
   Write the match policy before labeling cases

   Define the unit of identity for your business. Is a different pack quantity always a different SKU? Can private-label and manufacturer-branded records ever be the same product? Is a compatible replacement an identity match, a substitute relationship, or no relationship? Turn these questions into a short decision guide with required attributes by category. Reviewers cannot create reliable ground truth if “match” changes meaning from one queue to another. Keep exact identity and substitution as separate labels even when both are useful downstream.

2. 2
   Capture the complete evidence for each decision

   Store both raw source records, their normalized forms, the candidate set shown to the reviewer, and any authoritative evidence used to decide. Assign a stable case ID and record the expected outcome, reason code, reviewer, review date, supplier, category, and pipeline version that produced the exception. Preserve values for identity-critical fields such as manufacturer, MPN, GTIN, dimensions, voltage, material, and pack quantity. A label without its evidence cannot be audited when product data changes.

3. 3
   Convert review outcomes into controlled labels

   Use a small label taxonomy: MATCH, NO_MATCH, and, if the workflow needs it, SUBSTITUTE. Add reason codes such as PACK_QUANTITY_CONFLICT, VARIANT_CONFLICT, SUPPLIER_ALIAS, IDENTIFIER_CONFLICT, or INSUFFICIENT_EVIDENCE. Require a second review for high-risk cases and disagreements. Do not automatically treat every click in a review queue as truth; remove accidental decisions, duplicate pairs, and cases whose source evidence is no longer available.

4. 4
   Separate evaluation data from development data

   Freeze a held-out benchmark that prompt authors, rule writers, and model tuning jobs do not optimize against. Keep new production exceptions in a quarantine set until they are reviewed, then promote them on a schedule. Version each release of the dataset and record additions, removals, and corrected labels. If cases from one product family appear on both sides of a split, near-duplicate descriptions can leak the answer and make performance look better than it is; split by product family, supplier, or time where appropriate.

5. 5
   Measure the decisions the system actually makes

   Evaluate candidate retrieval and final classification separately. A perfect scorer cannot recover the correct SKU if retrieval never supplied it. For identity decisions, track precision, recall, and F1, along with false-merge and false-split rates. For a thresholded workflow, also measure auto-match precision, review rate, rejection rate, and coverage. Weight or hard-gate costly errors: one wrong merge involving pack quantity may matter more than many unresolved pairs sent to review.

6. 6
   Slice results until aggregate scores become actionable

   Break metrics down by supplier, category, language, identifier coverage, confidence band, and reason code. Compare products with and without GTINs, clean and noisy MPNs, new and established suppliers, and common and long-tail categories. A release can improve overall F1 while getting worse on electrical variants or multilingual descriptions. Slices reveal which part of the catalog is paying for an apparent global improvement.

7. 7
   Set release gates and replay on every change

   Save the current production pipeline’s predictions as the baseline. For each proposed change, run the same immutable cases and generate a diff: fixed cases, newly broken cases, score movements, and slice-level metric changes. Set non-negotiable gates—for example, zero new false merges in critical cases and no drop beyond an agreed tolerance in auto-match precision. Require an explicit approval for any trade-off rather than letting a higher average score silently authorize new failures.

8. 8
   Feed every new exception back into the suite

   Instrument production so reviewers can flag an incorrect link, missed link, variant confusion, wrong quantity, alias, or rejected substitute with the supporting evidence. Triage those events, correct the canonical relationship, and add the reviewed case to the next benchmark version. This closes the loop: production finds the edge case once; the regression suite ensures every later pipeline must handle it correctly.

Use a test-case record that can survive pipeline changes

The benchmark should describe the business expectation, not the current model’s implementation. A practical record can look like this:

{
  "caseId": "match-004281",
  "leftRecordId": "supplier-a-9917",
  "rightRecordId": "sku-18422",
  "expectedLabel": "NO_MATCH",
  "reasonCode": "PACK_QUANTITY_CONFLICT",
  "critical": true,
  "sourceContext": {
    "supplier": "supplier-a",
    "category": "air-filters",
    "leftPackQuantity": 24,
    "rightPackQuantity": 1
  },
  "review": {
    "status": "adjudicated",
    "reviewedAt": "2026-09-08"
  }
}

Keep predictions in a separate run artifact keyed by caseId. That lets the same expected outcome test a deterministic rules engine, an embedding pipeline, a prompted model, or a hybrid system without rewriting the ground truth. Store the pipeline version, configuration hash, threshold, retrieval version, and timestamp with every run so a result can be reproduced.

Define release gates around business risk

A useful scorecard answers more than “Did F1 increase?” Include:

• Critical-case pass rate: all cases involving safety, compliance, pack quantity, or irreversible auto-merges must pass.
• Match precision: of the pairs the system declares identical, how many truly are identical?
• Match recall: of the true identical pairs, how many did the system find?
• Retrieval recall at K: how often did candidate generation include the correct record?
• False-merge and false-split rates: how often did the system combine distinct products or separate identical ones?
• Automation coverage: what share can be decided automatically while meeting the precision gate?
• Review burden: what share enters the gray zone, and how did the change affect queue volume?
• Slice deltas: which suppliers, categories, and exception types improved or regressed?

Make every change produce an explainable diff

When a new run disagrees with the baseline, review the changed cases rather than only the aggregate. Classify each delta as an intended fix, an acceptable trade-off, a new regression, or a ground-truth issue. The release report should name the change under test and answer:

1. Which previously failing cases are now correct?
2. Which previously correct cases are now wrong?
3. Which confidence scores crossed an auto-match or rejection threshold?
4. Which supplier, category, or failure-mode slices moved materially?
5. Does the change increase manual review even if accuracy remains flat?
6. Has any critical case failed?

This process applies to every layer of the system. Re-run after changing a model or prompt, but also after updating candidate retrieval, normalization, synonym dictionaries, supplier aliases, confidence calibration, match thresholds, or deterministic rules. Seemingly harmless preprocessing changes can alter more decisions than a model upgrade.

How Claro closes the evaluation loop

Claro treats human review as durable evaluation data rather than disposable queue work. Accepted and rejected relationships retain their source evidence, confidence, and decision history. Known false merges, variant conflicts, quantity mismatches, supplier aliases, and rejected substitutes can be promoted into a versioned product matching test dataset and replayed before a changed matching pipeline is released.

The result is continuous evaluation alongside continuous product matching: quality is demonstrated against your real catalog and your real exceptions, not inferred from a generic benchmark or compressed into a single accuracy claim. As suppliers and assortments change, the test set grows with them—and the system is required to remember what your team has already learned.

Related resources

FAQ