Why AI Agents Need Human-Validated Semantic Context
2026 is shaping up to be the year agentic AI moves from pilot to production. According to Gartner, 42% of enterprises expect to deploy AI agents by the end of this year. Analytics agents —systems that let business users ask data questions in plain English and get answers back in seconds—are among the most widely adopted. The promise is compelling: no SQL required, no analyst bottleneck, insights on demand.
But the same Gartner research reveals a gap that the adoption numbers obscure: only 1 in 5 enterprises report that GenAI is delivering significant value to their organization. The gap between deployment and value isn’t a model problem. It’s a context problem.
What is semantic context?
Semantic context is the layer of meaning that sits between raw data and a reliable answer. It includes natural-language descriptions of what tables and columns actually mean, metric definitions grounded in how your organization computes them, and validated query patterns that have answered similar business questions before. Without it, an analytics agent has access to your data but no understanding of what it means, no way to know which table, formula, or join to trust.
As teams move beyond pilots, they run into the same wall. The agent has access to the data. It generates SQL. It returns an answer with full confidence instantly, with no visible sign that anything is wrong. And then a data engineer notices the number is off by 40%.
A business user asked the agent what net revenue was last quarter. The agent answered. It used the gross revenue table. It applied a deprecated metric definition. It queried data that was three weeks stale. Nobody told it any of that was wrong because the context layer feeding it didn’t know either.
Pinterest faced a version of this at a scale most teams haven’t had to reckon with yet: 400,000 ungoverned tables, 500 new ones created every day, institutional knowledge locked in people’s heads, in unanswered Slack threads, and in queries that only two analysts on the team actually understood. Before their Analytics Agent could be trusted, they had to solve a more fundamental problem than SQL generation: they had to make the context behind the data reliable.
That’s the challenge at the center of every serious analytics agent deployment. And nothing makes it more dangerous than the fact that users often don’t know the answer is wrong. An agent that fails with an error message is a broken tool. But an agent that fails with a confident, well-formatted, plausible-looking answer is a liability.
To address this, data platforms and vendors, from cloud providers to specialist context platforms, are investing heavily in auto-generation of semantic context: the ability to automatically extract meaning from the data signals that already exist in an enterprise, without asking teams to author documentation from scratch.
What is auto-generation of semantic context?
Auto-generation of semantic context is the process of automatically building a semantic layer: a structured description of what data means, how metrics are defined, and how tables relate to each other.
It’s the process of mining signals that already exist in the enterprise: historical SQL queries, BI dashboard definitions, dbt models, and unstructured documentation from tools like Confluence and Notion. Instead of asking data teams to document everything manually, auto-generation extracts meaning from how analysts have actually used the data, and turns that into context an agent can retrieve and act on.
Auto-generation solves a real problem: the cold-start problem.
You don’t need to spend months in workshops documenting your entire data estate before agents become useful. Feed the system your query history, connect your BI tools, and a working semantic layer emerges from day one!
But auto-generation captures how data has been used, not how it should be used. Those are different things. A query that 50 analysts ran for two years can still encode the wrong metric definition. Finance calls it “conversion.” Product calls it “conversion.” They calculate it differently. Auto-generation surfaces both and has no way to pick the right one.
Without a human validation layer, agents retrieve the most popular context, not the most correct context, and they answer with equal confidence either way.
This is why human curation isn’t a nice-to-have on top of auto-generation. It’s what makes auto-generated context trustworthy enough to act on.
What is human curation of context?
Human curation of context is the process by which domain experts (the analysts, finance leads, and product managers who actually know what data means) review, validate, and correct AI-proposed context before it reaches an agent. Review context in a structured way: a specific proposed definition is surfaced, the expert approves, edits, or rejects it, and the validated result becomes the canonical context that every agent and tool in the organization retrieves.
How semantic context is auto-generated today
Analytics agents don’t come with knowledge of your business baked in. They know how to generate SQL. They don’t know that your finance team defines “active customer” as someone who made a purchase in the last 90 days, or that the revenue_reporting table is the one to trust, not revenue_transactions. That knowledge has to come from somewhere, and building it manually has always been the bottleneck.
Auto-generation addresses this by mining the signals that already exist in your enterprise. Instead of asking data teams to sit in workshops and document everything from scratch, it works backwards from what analysts have already done: the queries they wrote, the dashboards they built, the metric definitions they encoded in dbt models. The semantic layer emerges from actual usage, not from a documentation sprint.
In practice, auto-generation draws from several sources simultaneously:
- Query history from your data warehouse: the SQL your analysts have run recently, which encodes business intent (which tables they trusted, which joins they used, which filters they applied to get to the right answer).
- BI tool definitions: metric calculations, dashboard logic, and business glossary terms that reflect how your organization measures things
- Semantic models from tools like dbt: explicit definitions of metrics, dimensions, and entities that analytics engineers have already formalized
- Unstructured documentation from Confluence, Notion, and similar sources. Institutional knowledge that exists in writing but isn’t connected to any data asset
From these inputs, a context platform generates structured semantic context: natural-language descriptions of what tables and columns mean, validated query patterns that have answered similar questions before, and metric definitions grounded in how the organization actually computes them.
Pinterest, for example, treated their years of analyst query history as “a library of real solutions written by real analysts.” Rather than requiring their team to document 100,000 tables before an agent could be trusted, they extracted the meaning that was already embedded in how analysts had used that data and made it retrievable. The Analytics Agent that resulted became the most-used internal tool at the company, with 10x the usage of the next.
At DataHub, we prioritize recent, high-quality queries over stale or low-signal ones, and draw on usage frequency, asset quality signals, and governance tier to select the patterns worth surfacing. The goal is not to reflect everything analysts have ever done. It’s to surface the patterns most likely to be correct. That filtering matters. And it’s also where the limit of auto-generation becomes visible.
Why auto-generation alone isn’t enough
Query history is a rich signal but an unfiltered one. It contains deprecated logic, conflicting definitions, and incorrect but popular queries — and auto-generation has no way to tell them apart. The diagram below shows what happens: wrong patterns reach the agent with the same confidence as correct ones. That’s the gap human review exists to close.
Why most “human review” implementations fail
Adding a review step sounds like the fix. In practice, it rarely works. Three reasons for that:
- Context ends up defined in the wrong place. Context gets defined inside every platform that touches data (Snowflake, Databricks, your BI tool, etc.) by whoever happens to have access. The finance expert who knows what “net revenue” actually means never gets asked. Data lives in many places. Context needs to live in one.
- Review becomes authoring. Most review workflows hand an expert a blank box and ask: “Is this right?” That’s not review, that’s writing! Experts don’t know what they’re approving against. There’s no draft to accept or reject, no conflict to resolve. When review feels like work, queues fill up and stay full.
- Nobody knows if the change helped. An expert approves a definition. It goes live. Did the agent get more accurate? Nobody knows. Without a way to test context changes against real questions, improvement is invisible.
The pattern is consistent: auto-generation produces context, review process follows but doesn’t stick, and agents keep getting it wrong. The problem isn’t that human review is a bad idea. It’s that most implementations skip the things that make it actually work: structure, accessibility, and a feedback loop that closes.
What human review actually looks like when it works
So what does human review look like when it actually works? Here’s what to look for in any context platform, and how DataHub approaches each.
Does it generate context automatically without a manual authoring project?
A platform should extract semantic meaning from signals that already exist: query history, BI definitions, dbt models, unstructured docs. Your team shouldn’t have to document the data estate before agents become useful.
Does it notify experts where they work, and propagate approved context everywhere?
Domain experts shouldn’t need to log into an unfamiliar platform to validate context. A Slack notification, a plain-English proposal, one decision to make. That’s what a low-friction review looks like. Once approved, validated context should propagate automatically to every agent (Claude, Snowflake Intelligence, Databricks Genie, custom LangChain agent, etc.) from a single source of truth.
Does it tell you if context changes are actually working?
Approval shouldn’t be the end of the process. Context needs a “spell-check” before it reaches the agent. Evals (a set of benchmark questions) run automatically in the background, catch errors before a business user sees them, and tell you exactly what broke and where. Miro went from 50% to 90% accuracy not from a one-time fix, but from this loop running continuously.
Does validated context reach every agent?
Context defined inside Snowflake only helps Snowflake agents. Context defined inside Databricks only helps Genie. A context platform should serve any agent, from one place, so the finance expert’s validated definition of “net revenue” is the same one Claude, Snowflake, and Databricks all retrieve.
Learn more about the DataHub Context Platform for AI agents.
