A larger catalog is not always a mess; sometimes it is the same product speaking several data dialects.
A blue mug may arrive once as “Stoneware Mug 12 oz,” again as “12oz Navy Coffee Cup,” and a third time with only a supplier SKU. Treating every variation as separate inflates counts; treating similar names as identical can merge two genuinely different items, such as the 12-ounce and 16-ounce versions.
The useful goal is one traceable product record per sellable item, with each feed row kept as evidence of where its details came from. A match should be strong enough to explain later: shared GTIN, a verified manufacturer part number, or an agreed combination of brand, variant attributes, and SKU. Names and photos can support that decision, but rarely deserve to make it alone. When the evidence is weak or conflicts, leaving records separate is usually safer than creating a false merge that distorts stock, price, or options.
- Keep source-feed IDs on the matched record so corrections can be traced back.
Define what counts as the same product
A match should mean the same purchasable variant: the same brand, model or manufacturer part number, variant attributes, and selling unit. A blue 500 ml bottle is not automatically the same item as the red 500 ml bottle; a two-pack is not the same as one bottle.
Separate the common look-alikes
| Listing relationship | Match as one product? |
|---|---|
| Same SKU/GTIN and variant | Yes |
| Parent product and one size or colour child | No |
| Single item and multipack | No |
| Current model and revised model | No, unless identifiers confirm equivalence |
| Bundle and one component inside it | No |
Product families are useful for browsing, but they are not merge targets. Keep a parent record for a T-shirt range separate from each size-and-colour variant. Likewise, treat optional accessories, seller-created kits, and regional packaging changes as distinct until reliable identifiers say otherwise.
Give every incoming row one status before linking it:
- Exact match — safe to connect to an existing variant.
- Possible match — similar evidence, but a key attribute is missing or conflicts.
- No match — represents a new product or a clearly different variant.
- Needs review — identifiers are absent, duplicated, or ambiguous.
This simple gate prevents a convenient family-level resemblance from becoming a permanent duplicate or, worse, an incorrect merge.
Keep one product record, not one rewritten feed
An approved match should create a link, not overwrite a merchant row. Keep every imported row as a recoverable source record, including its original values and feed reference. That makes it possible to trace a strange price, repair a bad match, or reprocess a feed later.
At the center sits one canonical product: the shared description of the item that matched rows point to. It holds stable identity information, such as:
- brand, manufacturer, model, GTIN, and MPN
- a normalized product title
- variant-defining attributes, such as colour, size, capacity, or pack count
- the chosen category and other reusable product details
Each merchant row remains an offer attached to that product. Offer fields are seller-specific and change often: merchant name, source SKU, product URL, price, currency, stock status, shipping cost, promotion dates, and the time the feed was captured.
For example, three stores may sell the same 1-litre black bottle. They can point to one canonical bottle record while retaining three separate prices, links, and availability states. If one store actually lists a two-pack, it should point to a different canonical variant—or remain unmatched until confirmed.
Normalize comparison fields without erasing differences
A feed title such as ACME Trail Bottle 750 ml – FREE SHIPPING is poor match evidence as written. Its useful identity is the brand, model, and size; the delivery promise is not.
Store the original value for display and provenance, then create a separate comparison value. Typical low-risk cleanup includes:
- lowercasing, trimming repeated spaces, and standardizing Unicode characters;
- converting unit forms such as
0.75 L,750ml, and750 mlto one format; - removing clearly promotional phrases:
free shipping,sale,new arrival, or retailer campaign names; - mapping known brand or color aliases, such as
Hewlett-PackardtoHP, only when the mapping is maintained deliberately.
Preserve identity-bearing details
Normalization must not discard variant evidence. 750 ml and 1 L, blue and navy, or single pack and 2-pack may be different sellable units. Keep size, color, pack count, condition, and edition in dedicated normalized fields rather than treating them as title clutter.
Exact matching becomes stronger when normalized brand, GTIN, model, and variant fields agree. Fuzzy title comparison can then be limited to records with compatible structured fields, reducing false merges caused by similar marketing language.
Use the strongest evidence first
-
Accept only validated GTIN matches
An exact GTIN is the best starting point when its length and check digit are valid and it identifies only one canonical item. If the same GTIN appears against conflicting brands or variants, send it to review rather than merging it.
-
Try stable manufacturer identifiers next
Exact manufacturer part numbers or supplier SKUs can be strong evidence when the brand and product type also agree. Treat retailer-created SKUs as source-specific unless their meaning is known.
-
Require agreement on the locked variant fields
For matching records when GTINs are missing, compare brand, model or base title, and the attributes that define the selling unit. A colour conflict, size conflict, pack-count conflict, volume conflict, or incompatible format is a hard no-match—even when names look nearly identical.
-
Use controlled combinations, not loose similarity
Match only when a predefined combination agrees, such as brand + model + colour + 500 ml + 2-pack. Missing data is not agreement; it merely lowers confidence. Fuzzy title similarity can suggest candidates, but should never create the match.
-
Keep ambiguous pairs out of the automatic merge
Label close candidates for review and store the fields that supported or blocked the decision. This makes later corrections possible without silently creating duplicates.
The costliest mistakes happen when two records describe products from the same family but not the same thing that can be bought. A parent listing for a shoe style, for example, should organize child listings for size and color; it should not replace them. Each child remains its own sellable variant with its own SKU, barcode, price, and availability.
Treat selling units as distinct
Keep separate canonical records when any of these changes:
- Pack count: one shampoo bottle is not a three-pack, even when the barcode-like text looks similar.
- Bundle contents: a camera body differs from a kit containing a lens, bag, or memory card.
- Condition: new, refurbished, used, and open-box offers need separate identities unless the catalog deliberately models condition as an offer attribute.
- Revision or edition: a 2024 model, updated formula, regional edition, or revised packaging may carry meaningful compatibility differences.
Related records can still be useful together. Store a parent_id, bundle_of, or successor_to relationship rather than forcing a shared product ID. This lets browsing group a family while checkout and inventory retain the exact item. Rules that prevent mistaken variant merges should block automatic matching whenever pack size, condition, or included components conflict.
When uncertain, preserve both records and mark the relationship for review. A duplicate is inconvenient; a false merge can misstate what is being sold.
Route ambiguity to a small review queue
-
Send only close calls to review
Keep exact identifier matches automatic, but queue pairs with conflicting attributes, missing variant details, or several plausible candidates.
-
Show the evidence beside each pair
Include source titles, brand, normalized size, pack count, identifiers, images or URLs, and the fields that disagree. A reviewer should be able to decide without opening multiple exports.
-
Use three clear outcomes
Approve the match, reject it, or mark it as needing more data. Rejections should remain visible so the same pair is not repeatedly proposed.
-
Capture the reason, not just the decision
A short reason such as “different pack count” or “GTIN reused across colors” makes later rule changes safer. This is especially useful for products with externally supplied variations.
-
Sample automatic approvals regularly
Review a small random set from each high-confidence rule. A mistaken normalization or reused identifier often appears in the sample before it creates a large duplicate problem.
A queue can begin as a spreadsheet tab; consistency matters more than elaborate tooling.
An automatic match rule deserves occasional spot checks, even when its score is near-perfect. If a sample exposes one false merge, pause that rule, identify the shared misleading field, and re-run the affected records rather than simply correcting the sampled pair.
Make matching survive the next refresh
A refresh should update an existing link before it looks for a new match. Store a stable source key for every offer—ideally merchant ID plus variant ID, or a durable URL-derived fallback—and map it to the canonical record. Titles, prices, and image URLs can then change without creating another product.
Keep an explicit decision log for every accepted match and review outcome. At minimum, retain:
- source key and canonical product ID
- decision status and confidence
- matching evidence and rule version
- reviewer or automated process
- first-seen, last-seen, and decision dates
Incoming rows should carry change flags such as new, updated, unchanged, and missing. Only new or materially changed rows need fresh comparison; prior approved links can be revalidated against identifier and variant checks. This is often when feed-matching software is more practical than spreadsheets, especially once logs and refresh history grow.
Send broken identifiers, conflicting variant details, and reused merchant IDs to a quarantine queue rather than guessing. For items absent from a feed, mark the offer inactive after a defined grace period; do not delete its history. A discontinued canonical product can remain archived with its old links intact, while newly supplied offers are prevented from attaching to it unless deliberately reinstated.
Scale Only What the Evidence Supports
- A rule is ready to expand only after sampled matches remain correct across new feed refreshes.
- A separate record is safer than an irreversible merge when product identity is unclear.
A merge should happen only when the available evidence establishes the same purchasable item—not merely a similar title, image, or brand. If an identifier is missing, a variant detail conflicts, or the selling unit cannot be confirmed, the records should remain separate and be marked for review.
Before applying rules across a catalog, test them on one brand or category known to contain many duplicates. Check both the merges and the records left apart. Once that small trial stays accurate through a refresh, the same rule can be expanded with much less risk.
Choose a duplicate-heavy area with familiar products, such as one shoe brand or a single skincare line. Run the proposed rule, inspect a sample of merged pairs, and look specifically for size, color, pack-count, and condition mistakes. A rule that works in one clean sample still needs a refresh test before broad use.
17 responses to “How to Match Products Across Feeds Without Creating Duplicates”
I like the conservative approach, but I’m not convinced a manual review queue stays “small” once you have millions of offers. Ambiguous barcode-less apparel is basically an infinite queue. At some point don’t you need a probability threshold and accept a few bad merges?
A probability threshold can be useful for prioritization, but I would separate “high probability” from permission to merge. For a broad, low-risk use case you may auto-link at a high threshold; for canonical identity, a wrong merge can be harder to unwind than a duplicate. In apparel, variant attributes and source history often provide more value than trying to force title similarity into a definitive answer.
We cap the queue by only reviewing pairs that would materially reduce duplication. The rest remain separate but grouped as possible relatives, which is less glamorous but keeps the team sane.
The line about preserving raw values is huge. Someone on our team normalized “12 pack” and “12-pack” correctly, then later normalized “12 pack + 2 free” to 12 as well. Guess which products got merged 😬
Step 3 didn’t work for me at first because our “stable” source key was actually the merchant’s row number. 🙃 Every export reordered it. Once we switched to their SKU plus merchant ID, refreshes stopped recreating the same products.
For validated identifiers, are you checking checksum only, or also verifying that the identifier is plausible for the claimed brand/category? We receive plenty of syntactically valid GTINs pasted into the wrong rows.
Checksum validation is necessary but not sufficient. We also check identifier length/prefix conventions where applicable, whether it has appeared consistently for that source, and whether it conflicts with strong attributes such as brand, pack size, or variant. A valid-looking identifier that contradicts the sellable item should lower trust or trigger review, not force a merge.
Yep. We found a supplier using one valid GTIN for an entire color range. The checksum passed perfectly; the catalog was still wrong.
This is more disciplined than most matching advice I’ve seen. Especially the random approval samples—people tend to inspect only obvious wins, then declare the rule safe.
We sample recent auto-merges by category and have caught some very weird manufacturer-ID reuse that way.
The distinction between a canonical product and a source offer finally clicked for me. We were overwriting the “main” price every import and then wondering why it kept changing. Keeping merchant price/stock on linked offers fixed that mess.
This worked! We piloted the rules only on a duplicate-heavy electronics category, as suggested, instead of turning them on for everything. The first refresh exposed a source that changed its URL format but not its SKU, so we fixed the key logic before expanding.
The decision log also made it much easier to explain to management why a few apparent duplicates were intentionally left alone.
“Related” is not “same” should be printed above every feed pipeline dashboard. Bundles were our nemesis: a console, a console + game, and a console + two games all kept getting collapsed into one. Great for imaginary bargains, terrible for customers.
I appreciate that this doesn’t pretend fuzzy title matching is magic. A lot of tools sell “AI dedupe” like it can read minds, then merge black shoes with navy shoes because 94% confidence. No thanks.
Could you expand on archival statuses? If a source offer disappears for two feeds and then returns, do you reactivate the old link or create a new one? We have seasonal sellers whose listings vanish for weeks, and treating every return as new is causing lots of false “new product” alerts.
Usually I would retain the old source-offer record and mark it inactive/archived rather than delete it. If the same stable source key returns under the same source, reactivate that link and record the observed change. If the key is missing or has been reused for a materially different item, quarantine it for review instead of automatically reviving it.
Question on “remove promotions” during normalization: do you strip phrases like “new 2024 edition” from titles? In our data, that wording is sometimes marketing fluff and sometimes the only clue that it is a revised item. Feels risky to remove it wholesale.
I would not strip it wholesale. Keep the raw title, extract likely revision signals into a comparison field where possible, and treat an explicit edition/revision conflict as a blocker. Promotional phrases such as “limited-time deal” are different from identity-bearing wording like “2024 edition.”