A larger catalog is not always a mess; sometimes it is the same product speaking several data dialects.
A blue mug may arrive once as “Stoneware Mug 12 oz,” again as “12oz Navy Coffee Cup,” and a third time with only a supplier SKU. Treating every variation as separate inflates counts; treating similar names as identical can merge two genuinely different items, such as the 12-ounce and 16-ounce versions.
The useful goal is one traceable product record per sellable item, with each feed row kept as evidence of where its details came from. A match should be strong enough to explain later: shared GTIN, a verified manufacturer part number, or an agreed combination of brand, variant attributes, and SKU. Names and photos can support that decision, but rarely deserve to make it alone. When the evidence is weak or conflicts, leaving records separate is usually safer than creating a false merge that distorts stock, price, or options.
- Keep source-feed IDs on the matched record so corrections can be traced back.
Define what counts as the same product
A match should mean the same purchasable variant: the same brand, model or manufacturer part number, variant attributes, and selling unit. A blue 500 ml bottle is not automatically the same item as the red 500 ml bottle; a two-pack is not the same as one bottle.
Separate the common look-alikes
| Listing relationship | Match as one product? |
|---|---|
| Same SKU/GTIN and variant | Yes |
| Parent product and one size or colour child | No |
| Single item and multipack | No |
| Current model and revised model | No, unless identifiers confirm equivalence |
| Bundle and one component inside it | No |
Product families are useful for browsing, but they are not merge targets. Keep a parent record for a T-shirt range separate from each size-and-colour variant. Likewise, treat optional accessories, seller-created kits, and regional packaging changes as distinct until reliable identifiers say otherwise.
Give every incoming row one status before linking it:
- Exact match — safe to connect to an existing variant.
- Possible match — similar evidence, but a key attribute is missing or conflicts.
- No match — represents a new product or a clearly different variant.
- Needs review — identifiers are absent, duplicated, or ambiguous.
This simple gate prevents a convenient family-level resemblance from becoming a permanent duplicate or, worse, an incorrect merge.
Keep one product record, not one rewritten feed
An approved match should create a link, not overwrite a merchant row. Keep every imported row as a recoverable source record, including its original values and feed reference. That makes it possible to trace a strange price, repair a bad match, or reprocess a feed later.
At the center sits one canonical product: the shared description of the item that matched rows point to. It holds stable identity information, such as:
- brand, manufacturer, model, GTIN, and MPN
- a normalized product title
- variant-defining attributes, such as colour, size, capacity, or pack count
- the chosen category and other reusable product details
Each merchant row remains an offer attached to that product. Offer fields are seller-specific and change often: merchant name, source SKU, product URL, price, currency, stock status, shipping cost, promotion dates, and the time the feed was captured.
For example, three stores may sell the same 1-litre black bottle. They can point to one canonical bottle record while retaining three separate prices, links, and availability states. If one store actually lists a two-pack, it should point to a different canonical variant—or remain unmatched until confirmed.
Normalize comparison fields without erasing differences
A feed title such as ACME Trail Bottle 750 ml – FREE SHIPPING is poor match evidence as written. Its useful identity is the brand, model, and size; the delivery promise is not.
Store the original value for display and provenance, then create a separate comparison value. Typical low-risk cleanup includes:
- lowercasing, trimming repeated spaces, and standardizing Unicode characters;
- converting unit forms such as
0.75 L,750ml, and750 mlto one format; - removing clearly promotional phrases:
free shipping,sale,new arrival, or retailer campaign names; - mapping known brand or color aliases, such as
Hewlett-PackardtoHP, only when the mapping is maintained deliberately.
Preserve identity-bearing details
Normalization must not discard variant evidence. 750 ml and 1 L, blue and navy, or single pack and 2-pack may be different sellable units. Keep size, color, pack count, condition, and edition in dedicated normalized fields rather than treating them as title clutter.
Exact matching becomes stronger when normalized brand, GTIN, model, and variant fields agree. Fuzzy title comparison can then be limited to records with compatible structured fields, reducing false merges caused by similar marketing language.
Use the strongest evidence first
-
Accept only validated GTIN matches
An exact GTIN is the best starting point when its length and check digit are valid and it identifies only one canonical item. If the same GTIN appears against conflicting brands or variants, send it to review rather than merging it.
-
Try stable manufacturer identifiers next
Exact manufacturer part numbers or supplier SKUs can be strong evidence when the brand and product type also agree. Treat retailer-created SKUs as source-specific unless their meaning is known.
-
Require agreement on the locked variant fields
For matching records when GTINs are missing, compare brand, model or base title, and the attributes that define the selling unit. A colour conflict, size conflict, pack-count conflict, volume conflict, or incompatible format is a hard no-match—even when names look nearly identical.
-
Use controlled combinations, not loose similarity
Match only when a predefined combination agrees, such as brand + model + colour + 500 ml + 2-pack. Missing data is not agreement; it merely lowers confidence. Fuzzy title similarity can suggest candidates, but should never create the match.
-
Keep ambiguous pairs out of the automatic merge
Label close candidates for review and store the fields that supported or blocked the decision. This makes later corrections possible without silently creating duplicates.
The costliest mistakes happen when two records describe products from the same family but not the same thing that can be bought. A parent listing for a shoe style, for example, should organize child listings for size and color; it should not replace them. Each child remains its own sellable variant with its own SKU, barcode, price, and availability.
Treat selling units as distinct
Keep separate canonical records when any of these changes:
- Pack count: one shampoo bottle is not a three-pack, even when the barcode-like text looks similar.
- Bundle contents: a camera body differs from a kit containing a lens, bag, or memory card.
- Condition: new, refurbished, used, and open-box offers need separate identities unless the catalog deliberately models condition as an offer attribute.
- Revision or edition: a 2024 model, updated formula, regional edition, or revised packaging may carry meaningful compatibility differences.
Related records can still be useful together. Store a parent_id, bundle_of, or successor_to relationship rather than forcing a shared product ID. This lets browsing group a family while checkout and inventory retain the exact item. Rules that prevent mistaken variant merges should block automatic matching whenever pack size, condition, or included components conflict.
When uncertain, preserve both records and mark the relationship for review. A duplicate is inconvenient; a false merge can misstate what is being sold.
Route ambiguity to a small review queue
-
Send only close calls to review
Keep exact identifier matches automatic, but queue pairs with conflicting attributes, missing variant details, or several plausible candidates.
-
Show the evidence beside each pair
Include source titles, brand, normalized size, pack count, identifiers, images or URLs, and the fields that disagree. A reviewer should be able to decide without opening multiple exports.
-
Use three clear outcomes
Approve the match, reject it, or mark it as needing more data. Rejections should remain visible so the same pair is not repeatedly proposed.
-
Capture the reason, not just the decision
A short reason such as “different pack count” or “GTIN reused across colors” makes later rule changes safer. This is especially useful for products with externally supplied variations.
-
Sample automatic approvals regularly
Review a small random set from each high-confidence rule. A mistaken normalization or reused identifier often appears in the sample before it creates a large duplicate problem.
A queue can begin as a spreadsheet tab; consistency matters more than elaborate tooling.
An automatic match rule deserves occasional spot checks, even when its score is near-perfect. If a sample exposes one false merge, pause that rule, identify the shared misleading field, and re-run the affected records rather than simply correcting the sampled pair.
Make matching survive the next refresh
A refresh should update an existing link before it looks for a new match. Store a stable source key for every offer—ideally merchant ID plus variant ID, or a durable URL-derived fallback—and map it to the canonical record. Titles, prices, and image URLs can then change without creating another product.
Keep an explicit decision log for every accepted match and review outcome. At minimum, retain:
- source key and canonical product ID
- decision status and confidence
- matching evidence and rule version
- reviewer or automated process
- first-seen, last-seen, and decision dates
Incoming rows should carry change flags such as new, updated, unchanged, and missing. Only new or materially changed rows need fresh comparison; prior approved links can be revalidated against identifier and variant checks. This is often when feed-matching software is more practical than spreadsheets, especially once logs and refresh history grow.
Send broken identifiers, conflicting variant details, and reused merchant IDs to a quarantine queue rather than guessing. For items absent from a feed, mark the offer inactive after a defined grace period; do not delete its history. A discontinued canonical product can remain archived with its old links intact, while newly supplied offers are prevented from attaching to it unless deliberately reinstated.
Scale Only What the Evidence Supports
- A rule is ready to expand only after sampled matches remain correct across new feed refreshes.
- A separate record is safer than an irreversible merge when product identity is unclear.
A merge should happen only when the available evidence establishes the same purchasable item—not merely a similar title, image, or brand. If an identifier is missing, a variant detail conflicts, or the selling unit cannot be confirmed, the records should remain separate and be marked for review.
Before applying rules across a catalog, test them on one brand or category known to contain many duplicates. Check both the merges and the records left apart. Once that small trial stays accurate through a refresh, the same rule can be expanded with much less risk.
Choose a duplicate-heavy area with familiar products, such as one shoe brand or a single skincare line. Run the proposed rule, inspect a sample of merged pairs, and look specifically for size, color, pack-count, and condition mistakes. A rule that works in one clean sample still needs a refresh test before broad use.
