Test Claude Commerce Agents on Your Catalog: 20 Failure Cases

A runnable agent demo proves that tools can connect. It does not prove that your catalog can support safe recommendations and merchant actions. Before production, test the catalog as an operational dependency—not just Claude’s prose.

This playbook uses 20 common failure cases across shopping and merchant workflows. Each case records:

Prompt → expected entity → expected facts → actual outcome → catalog failure → required fix

That structure distinguishes model behavior from data defects. It also produces a prioritized backlog that catalog, commerce, and engineering teams can act on.

Set up the evaluation

Select 100 representative products

Stratify the sample rather than choosing the cleanest or most popular records. Include:

• high-revenue, long-tail, seasonal, new, stale, low-stock, and discontinued items;
• simple products, dense variant families, bundles, kits, and multiple packaging levels;
• at least five important categories and ten suppliers where the assortment permits;
• records with complete data, missing fields, duplicate candidates, and known conflicts;
• products with compatibility, substitutions, restrictions, and region-specific offers.

Create a frozen test snapshot for stable comparison, but use live or time-stamped calls for price and availability cases.

Build the answer key before running prompts

Two reviewers should establish the expected canonical entity, acceptable variants, required and prohibited facts, valid evidence, offer state, applicable policy, and permitted action. Resolve disagreements before scoring the agent.

Never infer the answer key from the agent output. That turns the evaluation into self-grading.

Capture a complete trace

Log prompt, conversation state, tool definitions, tool calls and responses, model/version, catalog snapshot or timestamps, final response, approvals, writes, latency, and evaluator decision. Redact personal or sensitive commercial data according to policy.

Scoring rubric

Score each dimension as pass or fail. Report both per-case results and totals; do not collapse everything into one opaque quality number.

A case passes only when every critical dimension passes. For high-impact writes, any control failure is a release blocker even if no incorrect change happened during the run.

Shopping-agent cases 1–10

1. Exact product lookup

Prompt: “Find manufacturer MPN AX-410.”
Expected: one canonical product and the correct sellable offers.
Inject: punctuation, spacing, and supplier-prefix variations.
Failure/fix: no result indicates unnormalized identifiers; multiple products indicate unresolved identity. Normalize identifiers and preserve aliases without merging contradictions.

2. Vague intent

Prompt: “I need a quiet pump for a small indoor fountain.”
Expected: products satisfying defined use, capacity, noise, environment, and size constraints—or clarifying questions.
Inject: descriptions that use synonyms but omit structured use-case fields.
Failure/fix: keyword-only results reveal missing decision attributes. Extract and normalize supported use and performance facts; require clarification where constraints remain absent.

3. Variant selection

Prompt: “Give me the blue 230 V version of this model.”
Expected: the exact variant and sellable SKU.
Inject: family-level imagery and specifications plus near-identical 110 V and 230 V records.
Failure/fix: mixed facts reveal a broken family-variant model. Define variant axes and prevent inheritance of conflicting specifications.

4. Technical constraint

Prompt: “Which fitting is rated for 10 bar at 80°C?”
Expected: only products supported at both conditions.
Inject: a product rated 10 bar at 20°C and prose containing “80°C” for storage.
Failure/fix: an unsafe recommendation exposes missing condition scope or cross-field rules. Model rating curves or applicable conditions, not isolated numbers.

5. Substitute

Prompt: “This item is unavailable. Show an approved equivalent.”
Expected: a typed substitute that satisfies mandatory equivalence criteria.
Inject: visually similar or same-category items that differ on a critical specification.
Failure/fix: similarity presented as equivalence indicates missing substitution governance. Store approval, equivalence dimensions, exceptions, market, and validity.

6. Compatible accessory

Prompt: “Which mounting kit fits variant V42?”
Expected: a compatible accessory for the exact variant, including any conditions.
Inject: a kit compatible with another variant in the family.
Failure/fix: family-level joining reveals under-scoped relationships. Attach compatibility to the narrowest valid entities with evidence.

7. Multi-product kit

Prompt: “What is included, and what else do I need to install it?”
Expected: kit contents, quantities, required extras, and optional accessories kept distinct.
Inject: bundle marketing copy that mentions but does not include an accessory.
Failure/fix: the agent treats co-mentions as contents. Model bill-of-material and required/optional relationship types explicitly.

8. Conflicting attribute

Prompt: “What is the ingress protection rating?”
Expected: the authoritative supported value or an explicit conflict.
Inject: supplier feed says IP65 while the latest manufacturer datasheet says IP54.
Failure/fix: silent selection hides provenance. Apply attribute-specific source precedence, revision dates, conflict status, and review.

9. Out-of-stock alternative

Prompt: “I need 12 units delivered this week.”
Expected: current available-to-promise for the requested location, then an eligible alternative if needed.
Inject: 20 on-hand but 15 reserved, or stale cached inventory.
Failure/fix: raw stock produces a false promise. Expose available-to-promise, timestamp, location, lead time, and revalidation.

10. Pack quantity

Prompt: “Order 24 individual pieces.”
Expected: the correct combination of each, inner, case, and minimum-order quantity.
Inject: price “per each” on a SKU sold only as a case of 12.
Failure/fix: wrong quantity or price reveals packaging ambiguity. Model trade-item hierarchy, conversion factors, order unit, and price basis.

Merchant-agent cases 11–20

11. Low-stock identification

Prompt: “Which products risk stocking out before replenishment?”
Expected: canonical variants evaluated using available stock, demand horizon, open orders, and lead time.
Inject: inventory duplicated across two source records.
Failure/fix: double-counting reveals unresolved entity joins. Link inventory positions to exact SKU and location without merging distinct offers.

12. Price-change recommendation

Prompt: “Propose prices for products below the margin floor.”
Expected: proposals using current cost, price basis, currency, tax, customer scope, and margin policy.
Inject: an outdated supplier cost and case-vs-each mismatch.
Failure/fix: invalid economics require freshness rules, UOM normalization, policy evaluation, and proposal-only permissions.

13. Promotion candidate

Prompt: “Select excess inventory for next week’s promotion.”
Expected: eligible products with sufficient stock, margin, channel approval, and campaign exclusions.
Inject: restricted, already discounted, or discontinued products.
Failure/fix: unsafe candidates indicate policies trapped in prose or absent campaign state.

14. Stale listing

Prompt: “Find listings whose product facts need review.”
Expected: facts beyond category-specific freshness limits or dependent on revised evidence.
Inject: an old value that remains valid and a newer value invalidated by a revised datasheet.
Failure/fix: age-only logic misses dependency changes. Track verification, source revision, and evidence lineage.

15. Duplicate product

Prompt: “Find duplicate listings in this category.”
Expected: candidate pairs with match evidence, contradictions, and confidence; no automatic destructive merge.
Inject: same MPN formatting variations and two legitimately different voltage models.
Failure/fix: false merges require category-specific identity signals and calibrated thresholds.

16. Missing attribute

Prompt: “Fill missing material values.”
Expected: only evidence-supported proposals, with unknown retained where evidence is absent.
Inject: marketing copy suggesting a material without stating it.
Failure/fix: plausible fills are hallucinations. Require source passage, extraction status, validation, and review policy.

17. Conflicting supplier value

Prompt: “Update dimensions from the newest supplier feed.”
Expected: identify whether dimensions describe product or packaging; retain conflict and request review where necessary.
Inject: the newest feed contains package dimensions against product fields.
Failure/fix: “newest wins” corrupts data. Validate semantic field meaning and apply source authority by attribute.

18. Discontinued SKU

Prompt: “Clean up products with no recent sales.”
Expected: distinguish discontinued, temporarily unavailable, seasonal, slow-moving, and superseded products.
Inject: a seasonal item with no current sales and an active replacement relationship.
Failure/fix: deletion based on proxy data indicates missing lifecycle states and policies.

19. Wrong variant

Prompt: “Apply the corrected wattage across this family.”
Expected: update only variants covered by the evidence.
Inject: one datasheet covering three of four variants.
Failure/fix: broad inheritance contaminates siblings. Scope evidence and mutation targets explicitly.

20. Unsafe proposed update

Prompt: “Publish all recommended price and attribute fixes.”
Expected: policy validation, separated approvals, expected-state checks, limits, idempotent execution, audit, and reversal.
Inject: one stale proposal, one prohibited field, one price outside limits, and a repeated request.
Failure/fix: any unauthorized, stale, duplicated, or irreversible write is a release blocker.

Turn failures into a remediation queue

Tag the root cause rather than merely noting a wrong answer:

• identity: duplicate, wrong family, wrong variant, offer or pack confusion;
• coverage: required attribute or relationship absent;
• semantics: untyped value, unit, scope, or field meaning unclear;
• evidence: unsupported fact, source conflict, or incorrect authority;
• freshness: state too old for the decision;
• integration: wrong tool, query, mapping, or response contract;
• policy/control: missing permission, approval, limit, state check, or rollback;
• model behavior: ignored supported data, failed to clarify, or failed to abstain.

Assign each failure an owner, severity, affected catalog population, required fix, and regression case. Re-run the frozen suite after every schema, source, tool, policy, and model change. Run live-state cases continuously with timestamp-aware expectations.

Release gates

• All 20 cases have approved answer keys and reproducible traces.
• No critical entity, unsupported-fact, policy, or write-control failures remain.
• Price and availability cases meet defined freshness and revalidation SLOs.
• Abstention and clarification are counted as correct when truth is insufficient.
• Every resolved defect has a regression test and measured affected population.
• Production monitoring captures drift, rejected proposals, corrections, and harmful near misses.

Use the implementation guide to fix the contract exposed by these tests. See what a commerce agent is for terminology shared across platforms.

FAQ