Data sourcing for ecommerce: How to build reliable product data inputs

A product catalog is only as dependable as the inputs behind it. Supplier files, store APIs, retailer pages, product documents, and marketplace offers each describe part of what you sell. They also carry different rights, identifiers, update schedules, and failure modes.

Data sourcing gives your team a system for choosing those inputs and keeping them trustworthy. This guide focuses on ecommerce and retail product data: how to select sources, document permissions, preserve evidence, validate freshness, and route approved records into one governed catalog.

What data sourcing means for ecommerce#

Data sourcing is the process of identifying, evaluating, acquiring, ingesting, and maintaining data for a defined use. For an ecommerce team, the useful unit is an evidence-backed product, variant, or offer record that can move safely through a catalog and into downstream channels.

Sourcing is broader than extraction. Extraction retrieves values from a page, file, API, or database. Sourcing decides which source should be used, whether your team may use it, which fields it owns, how often it should be refreshed, and what happens when it changes.

It is also different from enrichment and syndication:

Our product data extraction guide goes deeper on retrieval methods. The sourcing decision comes first because an efficient extractor can still produce data your team cannot publish or explain.

For the wider set of product, offer, inventory, and market signals, see our retail data guide. The sourcing model here stays focused on the inputs that become governed product records.

Start with the decision your data must support#

Do not begin with a provider list. Begin with the product decision or workflow that needs an input.

Write down:

• the products, variants, and offers in scope;
• the markets, languages, and channels involved;
• the fields required to identify, compare, buy, or fulfill an item;
• the maximum age each field can have;
• the evidence and permissions your reviewers need;
• the downstream system that will own the approved value.

A category team may need material, fit, dimensions, and care instructions. A pricing workflow may need seller, currency, price, promotion, and observed time. An AI shopping feed needs explicit product identity, variant relationships, availability, and current commercial facts. These are different sourcing problems even when they start with the same URL.

Define the grain before you define the fields:

A black, size-medium shoe is a variant. A marketplace seller's price for that shoe is an offer. Keep those objects separate so one seller's price does not overwrite every seller, and one size's stock does not spread to the whole product family. Google documents the same separation through a unique item ID per product and a shared item_group_id for variants in its item group rules.

Choose the source route that fits the record#

There is no universal best source. Use the source with the right authority and field depth for the job, then add a fallback when one source cannot cover the record.

Treat public availability as an access condition, not a reuse license. The Robots Exclusion Protocol says its rules are not access authorization. A robots.txt file can tell a crawler which paths to avoid, but it does not settle ownership, terms, privacy, attribution, or republishing rights. Check the source's terms and your agreement before collecting or distributing a field.

Catalog is our recommended route when your problem starts with product URLs or domains on the open web and ends with typed product objects. Catalog can return structured product information for identifiers, variants, prices, availability, images, specifications, and freshness signals. It complements a PIM, ERP, supplier feed, or warehouse. It does not grant permission to reuse a retailer's content or replace the system that owns your inventory.

Score every source with the same contract#

A source earns a place in your pipeline when it meets the record's requirements. Score it against the same dimensions each time:

1. Coverage: Does it include the exact retailers, suppliers, categories, markets, and page types in scope?
2. Field and variant depth: Can it represent category-specific attributes and every sellable option, rather than one flattened row?
3. Authority: Which organization owns each value? Is the source an approved system of record, a partner submission, or an observation of another site?
4. Freshness: How old can each value be? Are price and availability updated on a different schedule from dimensions and descriptions?
5. Provenance: Can you retain the source URL or system ID, original value, observation time, transformation, and revision history?
6. Permissions: Does your agreement cover access, storage, internal use, publication, attribution, geography, and retention?
7. Stability: What happens when an endpoint, file layout, page template, identifier, or taxonomy changes?
8. Delivery: Can the data arrive through the API, file, webhook, or connector your catalog can operate?
9. Cost: What is the cost per accepted record after retries, failed rows, mapping, storage, monitoring, and review?

Do not average these into one headline score. Make hard gates for rights, identity, and fields that affect a buying or safety decision. A source with broad coverage cannot compensate for an unclear license. A low request price cannot compensate for stale inventory or unresolved variants.

Record permissions before the first ingest#

Permission belongs in the source registry beside the endpoint, file, or URL pattern. Create one record for each source with:

• source owner and accountable internal team;
• access method and credentials or contract reference;
• allowed purpose and destination scope;
• fields allowed for storage, transformation, and publication;
• geography, language, and retention limits;
• attribution or notice requirements;
• rate limits, robots expectations, and contact path;
• approval date, reviewer, and re-review date;
• deletion or suspension procedure.

Use the narrowest purpose that supports the workflow. If a supplier authorizes product specifications for your storefront, do not assume the same authorization covers a competitor-monitoring dataset or redistribution to a third party. Keep customer or employee information out of a product-data route unless the workflow has a separate, documented purpose and control.

The W3C Data on the Web Best Practices treat license, provenance, quality, access, and update information as metadata that helps people and software assess reuse. Apply that principle at field level where it matters. A permission record without a link to the approved source terms is hard to audit.

Define a schema contract before mapping fields#

A schema contract turns a source submission into a predictable catalog input. It should answer what every field means, which object owns it, how it is typed, and what happens when it is absent or disputed.

At minimum, define:

Keep identifiers in namespaces. A supplier SKU, internal SKU, GTIN, MPN, provider ID, variant ID, and offer ID answer different questions. GS1 describes a GTIN as an identifier for a defined trade item. It does not become a safe replacement for a missing merchant SKU, and you should never invent one to fill a blank.

A practical record carries both source and catalog values:

product_group_id: group_482
variant_id: variant_482-black-m
offer_id: seller-a-482-black-m
sku: source value plus namespace
source_url: source URL or source-system ID
observed_at: when the source value was seen
processed_at: when your pipeline accepted the record
price: amount plus ISO currency
availability: documented enum, including an explicit unknown state
attributes: typed values plus original source values
provenance: source, transformation version, reviewer, and change history

This contract is your internal model. Destination feeds can map it into their own requirements. For example, OpenAI's current product-feed specification requires stable item IDs and explicit price and availability, and models variants with a group ID and variant options. Use the product feed schema as one destination example, not as your canonical model.

Preserve provenance through normalization#

Provenance is the trail behind a value: where it came from, when it was observed, which operation changed its representation, and who approved the result. The W3C provenance model frames provenance as information used to assess data quality, reliability, and trustworthiness.

Store three layers rather than overwriting one value:

• Observed: the source said navy, 12 oz, or $79.99 at a known location and time.
• Normalized: your mapping converted those values to a controlled color, canonical unit, and typed money object.
• Enriched: a reviewer or model added a category, compatibility label, or use case with its method and review state.

When two sources disagree, keep both observations and apply a documented precedence rule. For example, your inventory system may own stock, the brand feed may own material, and a retailer page may provide an observed public offer. Do not silently choose the most recent value if the sources have different authority.

Lineage also makes exceptions actionable. A reviewer should be able to answer which source produced a bad claim, which mapping version changed a unit, and which downstream channels received the value. If you cannot answer those questions, the catalog is difficult to correct safely.

Design freshness around the field, not the file#

A file can arrive on time and still contain an old price. A fast API response can also be a cached observation. Track at least two timestamps for every changing value:

• source_observed_at: when the source value was captured or reported;
• published_at or processed_at: when your catalog accepted and exposed the value.

Then measure output lag, the time between a change at the source and a usable value in the catalog or destination. Set freshness budgets by field:

• price, promotion, and availability usually need the shortest window;
• shipping, returns, and lead times need a cadence that matches the promise being made;
• descriptions, dimensions, and stable specifications can use a slower schedule;
• identifiers, taxonomy, and permissions need controlled change and re-review rather than blind polling.

When a value exceeds its age budget, mark it stale or unknown and route it for refresh. Do not silently copy an older value into a field that looks current. A catalog that distinguishes “not observed” from “out of stock” is safer for shoppers and downstream systems.

Route trusted inputs into one governed catalog#

A durable pipeline has one canonical record and many destination outputs:

Approved sources → raw evidence → identity match → normalize → validate → approve → governed product catalog → channel and AI outputs

Use an ownership map for each field. The catalog should retain the source observation and transformation history, while the approved model becomes the shared input for storefront search, marketplaces, feeds, analytics, and AI shopping surfaces.

A useful control set includes:

1. Identity control: preserve source IDs, namespaces, parent-child relationships, and offer links.
2. Evidence control: retain raw values, source locations, capture times, document versions, and parser or mapping versions.
3. Validation control: check required fields, identifier formats, units, currencies, variant relationships, media, price, and availability before publication.
4. Freshness control: apply field-level age budgets and alert when a source stops responding or a record exceeds its budget.
5. Permission control: block storage or distribution when the source record is expired, out of scope, or missing an approval.
6. Change control: version schemas and mappings, test representative records, and keep a rollback path.

Send destination-specific feeds or APIs from the governed model. A marketplace may need a different taxonomy from your storefront. An AI shopping surface may need explicit variant and offer objects. Those are outputs, not competing catalogs.

Failure modes that expose weak sourcing#

Watch for these patterns in production:

• The source is technically reachable but unauthorized. Access succeeds while the permission record is missing or too broad.
• A source has coverage without field depth. It lists a product but omits the variant, offer, specification, or evidence your decision requires.
• A page or file is treated as a source of truth. The team cannot say which system owns price, inventory, or approved claims.
• Variants are flattened. A parent title carries one price, image, or stock state across every option.
• Provenance is discarded during mapping. Normalized values remain, while the original value and transformation disappear.
• Freshness is measured at the pipeline level. A “daily refresh” hides a price that is two days old or an image that changed an hour ago.
• Schema drift is silent. A supplier renames a column, changes units, or adds a category value, and the pipeline accepts it as a different fact.
• One source is used for every field. A source that is authoritative for descriptions may be weak for inventory or market offers.
• The channel becomes the master. Manual fixes in one feed never reach the governed record and disappear on the next sync.

Fix the ownership, evidence, or contract at the source boundary. Repeated cleanup at the channel edge is a symptom of weak sourcing governance.

Data sourcing checklist#

Before promoting a source into production, confirm:

• The source, owner, purpose, markets, and fields are documented.
• Terms, license, attribution, retention, and redistribution rules are approved.
• Robots rules, API scopes, rate limits, and access controls are recorded.
• Product group, variant, and offer grains are explicit.
• IDs are namespaced and stable, with a rule for missing identifiers.
• Required fields, types, units, currencies, and null meanings are versioned.
• Raw evidence, source timestamps, transformations, and reviewers are retained.
• Freshness budgets exist for price, availability, policy, content, and specifications.
• Conflicting sources have an ownership and precedence rule.
• Validation, exception handling, rollback, and re-review owners are assigned.
• Approved records flow into one governed catalog before channel-specific outputs.

FAQ#

What is the difference between data sourcing and data collection?#

Data collection is the act of gathering values. Data sourcing includes collection plus source selection, permissions, provenance, schema contracts, quality gates, and lifecycle maintenance. Collection produces inputs. Sourcing determines whether those inputs are fit for use.

What are the best data sources for ecommerce product data?#

Start with the source that owns the fact. Use your PIM, ecommerce platform, or ERP for owned product and operational data. Use an authorized supplier or brand feed for specifications and approved content. Use retailer or marketplace APIs for permitted offers and availability. Use public product pages or a managed product-data API when you need open-web coverage and have a documented access and reuse policy.

How often should product data be refreshed?#

Set the cadence by field behavior and business risk. Price, promotions, and availability need faster checks than stable dimensions or descriptions. Track source observation time and catalog publication time separately, then mark values stale when they exceed their allowed age.

Is Catalog a product data source or a PIM?#

Catalog is a product data layer. It can turn open-web product URLs and domains into structured, normalized product objects and keep the resulting data useful for AI commerce and downstream systems. It complements the PIM, ERP, supplier feed, or inventory system that owns your approved business facts. It does not replace permission review or inventory ownership.

When product URLs, supplier files, and APIs are producing records you cannot trace or keep current, request a Catalog product-data audit.