Every knowledge engine starts in the same place: with data.
Not insights. Not intelligence. Not recommendations. Data.
Raw, unprocessed, often messy, frequently incomplete data. The kind that lives in spreadsheets, vendor portals, order management systems, and email threads. The kind that no one has ever tried to connect before.
This is the Data Layer: and it's where everything begins.
What the Data Layer Is
The Data Layer is the foundation of a knowledge engine. It's the collection of raw signals that, once captured, structured, and connected, will eventually become intelligence.
In the promotional products industry, these signals include supplier catalogs and product data, order history and fulfillment records, customer purchase patterns, industry event calendars, compliance and certification data, and market pricing trends.
Individually, none of these signals is particularly powerful. A supplier catalog tells you what products exist. An order history tells you what was purchased. An event calendar tells you what's coming up.
But together: connected, contextualized, and reasoned over: they can tell you what a customer is likely to need before they know they need it.
The Quality Problem
Here's the uncomfortable truth about most data layers: they're built to capture, not to inform.
Systems are designed to record transactions, not to surface patterns. Fields are added over time without a coherent schema. Data is entered inconsistently by different users. Integrations are built for one-time purposes and never maintained.
The result is a data layer that technically contains a lot of information but practically delivers very little intelligence.
Before you can build a knowledge engine, you have to be honest about the quality of your foundation. That means auditing what you have, identifying where it's incomplete or inconsistent, and making deliberate decisions about what to fix, what to monitor, and what to accept as a known limitation.
The Signal Selection Problem
The second challenge is signal selection. Not all data is equally useful.
Some signals are highly predictive: they reliably indicate something meaningful about customer behavior, market trends, or operational risk. Others are directionally useful: they point in the right direction but require additional context to interpret. And some are simply noise: they're captured because they're easy to capture, not because they drive decisions.
The work of building a good Data Layer is largely the work of distinguishing between these categories. It requires you to start from the decisions you want to support and work backward to the signals that inform those decisions: rather than starting from the data you have and hoping insights emerge.
Why This Foundation Matters
Every layer of a knowledge engine: integration, context, reasoning, decision: depends on the quality of the Data Layer beneath it. A weak foundation doesn't just limit what the system can do. It actively misleads. It produces confident-sounding outputs based on incomplete or inaccurate inputs.
The most dangerous AI systems aren't the ones that are obviously wrong. They're the ones that are subtly wrong in ways that take months to detect.
Build the foundation right. Everything else follows from there.
What this looked like in my work
The hidden data problem in the promotional products industry is exactly what I encountered at iPROMOTEu. Supplier data arrived in dozens of formats, with inconsistent field names, missing values, and no shared taxonomy. The data layer work I did was building the ingestion and normalization pipeline that made that data usable. Without that foundation, the AI Intelligence Platform I built later would have had nothing reliable to reason over.
Read the full case study: PromoStandards Vendor Integration: iPROMOTEu
