AI data collection starts with source-grounded product data

AI data collection often means building a training dataset. For commerce teams, the more immediate job is different: collect the product facts an AI shopping agent needs to answer a shopper accurately, then keep those facts current.

That requires a source-grounded workflow. It turns pages, feeds, PDFs, product information management (PIM) records, and inventory systems into typed product objects with evidence behind every important field. This guide shows how to build that workflow without treating a large language model (LLM) as your source of truth.

AI data collection has two jobs#

The same phrase describes two separate systems:

The OECD taxonomy of data collection focuses on how developers source data for training and fine-tuning. That work matters when you are developing a model. A shopping agent that needs the price and availability of a particular variant today has a different data problem. It needs a retrievable record tied to a current source.

Treating these jobs as the same leads to bad commerce workflows. A large archive of product descriptions may help a model learn language, while doing nothing to answer whether a size 8 shoe is in stock in Chicago. Product-data collection exists for that operational question.

What source-grounded AI data collection means#

Source-grounded collection gives every material product field an evidence trail. The system should answer three questions without a second research project:

1. Where did this value come from?
2. When was it observed?
3. Why was it accepted over another value?

The NIST AI RMF Playbook calls for documenting data provenance, including sources, transformations, constraints, and metadata. Product workflows need the same discipline. A field such as waterproof, compatible_with, or price can change a recommendation, a return, or a purchase decision.

Keep raw observations separate from normalized values. A source may say “sky blue,” while your catalog uses light_blue. Preserve the original phrase, store the normalized value, and record the rule that connects them. This makes a later correction possible without recollecting the source.

A useful evidence record looks like this:

{
  "field": "dimensions.height",
  "raw_value": "8 3/4 in",
  "normalized_value": 8.75,
  "unit": "in",
  "source": {
    "type": "manufacturer_spec",
    "url": "https://example.com/specs/product-123",
    "observed_at": "2026-10-30T10:00:00Z"
  },
  "state": "verified"
}

The URL is illustrative. The important part is the shape: a field, its observed value, its normalized value, its source, and its review state. A confidence score can help route work. It cannot replace evidence.

A source-grounded collection workflow#

1. Start with the agent job and target schema#

Define the questions the product record must answer before collecting anything. “Make the catalog AI-ready” is too broad. “Return waterproof commuter jackets under $200 that fit a 40-inch chest and can ship to New York” gives you a schema to work from. Our product-data model is a useful reference for the core fields.

The schema usually needs:

• stable identity, including SKU, GTIN (global trade item number), MPN (manufacturer part number), brand, and canonical URL;
• category and taxonomy path;
• category-specific attributes such as material, dimensions, ingredients, fit, or compatibility;
• parent and child variant relationships;
• price, currency, availability, and region;
• shipping, returns, warranty, and other policy fields;
• images, manuals, and other supporting media;
• source, timestamp, transformation, and review metadata.

Use category extensions rather than forcing every product into one flat record. A supplement needs serving size, ingredients, allergens, and warnings. An electronics catalog needs ports, voltage, compatibility, and certifications. An apparel catalog needs size systems, fit, fabric, and care instructions.

2. Register sources and their authority#

List every place a product fact currently lives:

• ERP, PIM, inventory, and pricing systems;
• supplier feeds and manufacturer files;
• product pages and their structured data;
• manuals, spec sheets, and PDFs;
• approved marketplace or retail feeds;
• reviews and customer-service content;
• product images when a visual attribute is explicitly documented.

Then set field-level authority. An inventory service should outrank a marketing page for stock. A manufacturer specification should outrank an inferred material tag. A regional policy service should outrank a generic returns paragraph for shipping and returns.

Register access permissions, terms, and retention rules for each source. A page being publicly reachable does not by itself make every reuse permitted. Collection should stay within the access your business is allowed to use, with a record of that decision.

3. Capture observations before rewriting them#

The collector should store the raw page fragment, feed row, document location, or API response that supports a value. Save the source URL, source identifier, retrieval time, and capture method. Keep the source snapshot or a durable reference when the source permits it.

This step prevents a common failure: an LLM rewrites a paragraph into a polished claim, and the team later cannot tell whether the claim came from a specification, a review, or a guess. Extraction should propose values. The evidence record should make them reviewable.

4. Extract into typed fields#

Convert observations into fields with explicit types and units. “12” is not enough for an agent. It could mean 12 inches, 12 pounds, 12 items, or a 12-volt input.

Store 12 in, 12 lb, 12 ct, and 12 V as different values. Use controlled enumerations for states such as in_stock, out_of_stock, and preorder. Keep a field for the original wording when a downstream team needs to audit the transformation.

An LLM can help identify candidate fields in prose, images, and PDFs. It should return the source span and abstain when the evidence is ambiguous. Do not let a generated summary become the canonical product record.

5. Resolve identity, variants, and relationships#

Collection is incomplete until the system knows which records refer to the same sellable item. Match records with stable identifiers where available, then use brand, manufacturer part number, title, dimensions, and other evidence to resolve the remaining cases.

Keep the product and its offers distinct. A parent product can have child variants for color and size. A bundle has components. An accessory has a relationship to a compatible product. A regional offer can have a different price or availability without becoming a new product.

This structure lets an agent answer “which black version is in stock?” instead of returning a parent product with an unavailable variant. It also prevents duplicate listings from taking up every candidate slot in a recommendation.

6. Validate against sources and business rules#

Run validation after extraction and after every update. Useful checks include:

• required-field and category-field coverage;
• valid units, ranges, and enumerated values;
• duplicate identifiers and broken canonical URLs;
• parent-child variant integrity;
• contradictory claims across sources;
• price, currency, and availability consistency;
• policy coverage by region;
• source age and freshness thresholds.

When two authoritative sources disagree, quarantine the field and route it for review. Averaging a price or picking the newest sentence can create a value that no source supports.

Google's product data specification illustrates why these checks matter. It requires accurate identifiers, price, availability, and variant fields, and warns that mismatches between a data source and a landing page can cause incorrect displays or disapprovals. The same mismatch makes an AI recommendation unreliable even when no ad platform is involved.

7. Review consequential uncertainty#

Use human review where an incorrect field could cause a costly or unsafe outcome. That includes ingredients and warnings, safety claims, compatibility, regulated attributes, high-value prices, and shipping promises.

Route records by evidence state:

This is where source-grounded collection differs from prompt-based enrichment. The system has a controlled way to say “unknown.” Unknown is safer than an attractive invention.

8. Publish and refresh the live record#

Send approved product objects to the destinations that need them: APIs, feeds, search indexes, storefront structured data, marketplaces, and AI-facing surfaces. Retain the same identifiers and provenance across each projection so a correction can be traced downstream. The trusted data sources behind those projections should remain visible to the operating team.

Use risk-based refresh intervals. Price, stock, and delivery promises may need event-driven updates. Specifications can refresh when a manufacturer source changes. Reviews and derived summaries need their own cadence. Retired products need explicit tombstones so an agent does not keep recommending a page that no longer represents a purchasable item.

The minimum product object an agent can use#

An agent needs typed facts and relationships, rather than a longer product description. A minimal object can look like this:

{
  "id": "harborline-shell-navy-m",
  "parent_id": "harborline-shell",
  "sku": "HL-SHELL-NVY-M",
  "brand": "Harborline",
  "title": "Waterproof commuter shell",
  "category_path": ["Apparel", "Outerwear", "Rain jackets"],
  "attributes": {
    "color": "navy",
    "size": "M",
    "material": "recycled nylon",
    "waterproof": true,
    "fit": "regular"
  },
  "offer": {
    "price": 149,
    "currency": "USD",
    "availability": "in_stock",
    "region": "US"
  },
  "evidence": [
    {"field": "material", "source_id": "spec-881", "state": "verified"},
    {"field": "offer", "source_id": "inventory-42", "state": "verified"}
  ]
}

The object separates identity, attributes, and offer state. It also exposes the evidence references that support the fields an agent is likely to use. A generated explanation can sit beside this record as a derived output. It should never overwrite the underlying facts.

Make the data reachable for AI shopping agents#

A trustworthy record still has to be available where agents look. Give each important product a consistent set of machine-readable representations:

1. Product structured data on the page. Keep the visible product, price, availability, and variant information aligned with the markup.
2. Destination feeds. Send the fields required by shopping and marketplace channels in their accepted formats.
3. Typed APIs or files. Offer JSON, GraphQL, XML, or another stable format for builders and internal systems.
4. Stable identity and navigation. Preserve canonical URLs, identifiers, sitemaps, and parent-child links.
5. Commercial and policy context. Expose shipping, returns, warranty, and regional constraints alongside the product facts.

Google's product structured-data guidance recommends using page markup, Merchant Center feeds, or both, and explains that the two sources can help Google understand and verify product information. The broader principle applies to AI shopping surfaces: publish the same fact through the surfaces that need it, then monitor for divergence.

Keep the human storefront and the machine-facing product layer connected. The storefront explains the product to a person. The structured object lets a shopping agent compare it against constraints such as budget, size, compatibility, delivery region, and availability.

Measure collection quality before measuring visibility#

A product can disappear from an AI answer because the agent cannot retrieve it, because the record is incomplete, or because the product is a poor match. Separate those causes with operational metrics:

Track these at product and field level. A catalog with 98% complete records can still fail a high-value category if its compatibility or delivery fields are missing. A single stale price can also matter more than dozens of low-risk description gaps.

Common AI data collection failures#

Starting with prompts instead of a schema. A prompt can extract a field. It cannot decide which fields a category needs, how variants relate, or which source outranks another. Define the record first.

Flattening every product into a paragraph. Prose hides units, scope, and relationships. Keep typed fields and use prose as one representation of the record.

Treating inferred attributes as facts. “Looks suitable for hiking” is a derived tag. It does not prove waterproofing, weight, or safety. Label derivations and keep them separate from specifications.

Publishing one-time exports. A clean CSV becomes stale when prices, inventory, or policies change. Build refresh, failure alerts, and rollback into the workflow.

Discarding the evidence after extraction. Without source references, a review team cannot resolve disputes and an agent cannot provide a defensible answer. Keep the observation ledger.

Using a generic schema across categories. The fields that help choose a laptop are different from the fields that help choose a skincare product. Extend the model by category and destination.

Where Catalog fits#

We built Catalog for the product-data layer behind AI commerce. We take raw catalog information, structure it into live normalized product data, publish that data to AI shopping surfaces, and measure how AI systems see and recommend a brand.

That is the right fit when you need agents to work from current product facts without rebuilding the storefront. A PIM can govern internal product content. A feed manager can format channel exports. A custom pipeline can give an engineering team full control. We connect the source-grounded product object to the parallel storefront and AI surfaces where discovery now happens.

The practical test is simple: can your system show the source, value, variant, offer state, and last update for the product an agent is about to recommend? If it cannot, the next AI feature will inherit the same uncertainty.

FAQ#

Is AI data collection the same as collecting training data?#

No. Training-data collection creates examples for model development. Product-data collection creates current, retrievable product facts for AI inference and commerce workflows. A company may use both, but they need different schemas, refresh policies, and quality checks.

Can an LLM collect product data by itself?#

An LLM can help extract and classify candidate values from text, images, and documents. It should operate inside a workflow that defines the schema, records source evidence, validates types and relationships, and routes uncertainty for review. The LLM output is a proposal, not the source of truth.

What data does an AI shopping agent need?#

At minimum, it needs stable identity, category, relevant attributes, variants, price, currency, availability, images, and a product URL. Good commerce records also include shipping, returns, warranty, compatibility, reviews, and source metadata. The required fields depend on the shopper's question and the product category.

How often should product data be refreshed?#

Refresh based on risk and change rate. Price, stock, and delivery promises usually need event-driven or frequent updates. Product specifications can follow source changes. Derived summaries and review signals need their own cadence. Every published value should have a freshness threshold and a failure state.

If your product facts are scattered across pages, files, feeds, and inconsistent variants, book a Catalog demo.