What Shipping an AI Agent to 16,000 People Taught Me About Reviewing One

“Welcome to DataHub! Hey, we’re working on this Otto agent, can you help us launch it?”

That was how I started my summer internship with DataHub. Sounded pretty simple when my manager gave me this project. Boy, was I wrong!

Before Otto, the DataHub Community Slack ran on Herald, an older, simpler AI agent that answered technical questions based on our docs, community Slack conversation history, and GitHub. Herald served us well, and taught us what the community actually needed and what typically tripped people up.

We wanted to build on that. Otto is grounded more deeply in DataHub’s own documentation and source code, not just conversation history, so it can answer harder, more specific technical questions on its own.

The intention was simple: let Otto take on the volume of questions Herald could not fully handle, so the community gets faster, more accurate answers, and our team spends less time on repeat questions and more time on the ones that actually need human judgment.

The big challenge: The agent needs to actually work

Before releasing an agent to a community of 16,000+ data practitioners, you want to be sure it works.

That sounds obvious, but the stakes are real. This is not a beta feature shipped to a handful of friendly early adopters. It is an active Slack workspace where people ask critical, domain-specific, often urgent questions about a production system they depend on. A confidently wrong answer does not stay contained to one thread. Before Otto could go live we needed to know we could trust it.

The eval process: four rounds, four different methods

I started reviewing Otto’s responses manually: 21 real questions pulled from actual community history, graded against the DataHub repo using Claude Code. 

Once the pipeline was automated, I ran the same 21 questions through Otto and a code-archeology agent that DataHub uses internally, and independently re-verified every answer against the repo again. 

For the last round, I changed methods where instead of comparing Otto and the agent head to head, I used the code-archeology agent’s answers plus a human reviewer’s notes as a new ground truth, and graded Otto’s existing answers against that.

Round	Method	Pass	Partial	Mixed	Fail	Pass rate
1	Manual grading, first pass	7	9	0	5	33%
2	Manual grading, after first fixes	9	4	5	3	43%
3	Automated via Slack API, independently re-verified	11	6	2	2	52%
4, initial	Graded against code-archeology agent + human-reviewer notes	7	10	1	3	33%
4, audited	Same answers, re-verified from scratch	11	6	2	2	52%

By Round 4, the internal agent had already done two jobs on this project: Writing fixes and then answering questions as a benchmark. For this round I gave it a third job: judge. I asked it all 21 questions directly, had a human reviewer add correction and confidence notes on top of its answers, and used that combined “agent plus human notes” as a new ground truth.

But this round came back at 7 pass, 10 partial, 1 mixed, 3 fail. Worse than Round 3, on the same answers.

So, I started investigating. I re-verified every flipped verdict from scratch, against the actual repo, ignoring both the old and new ground truths. I discovered that five of the seven were wrong only because the internal agent’s “corrections” weren’t checked closely enough by the human reviewer, not because Otto’s original answers were bad. Once I reverted those five, Round 4’s audited tally landed exactly on Round 3’s: 11 / 6 / 2 / 2. Nothing about Otto had changed.

During my investigation, I ran across a blog post by Shreya Shankar on agent-assisted qualitative analysis that helped me develop these evals. Her experiments found that agents given human feedback over multiple rounds tend to either overfit hard to whatever was said first, or lose the thread entirely. Furthermore, validating an agent’s higher-level judgments is much harder than validating its raw output. That’s a good description of what happened to my grading layer: a single unverified endorsement propagated into five wrong verdicts before anyone looked closely enough to catch it. 

It’s also a point DataHub has made about human-validated context for AI agents more broadly: a human sign-off only makes AI-generated content trustworthy if the review itself is structured and checked by the right domain expert. A non-expert reviewer handed a blank “is this right?” box, with no way to verify their answer against ground truth, isn’t really reviewing. That’s exactly what happened to my Round 4 setup. The lesson isn’t just “check your agent” rather, it’s “check whoever’s checking your agent, too.”

Three persistent problems that survived every round of testing

After all that testing, three real issues with Otto remained:

It didn’t know when to say no

On “DataHub vs. alternative”-style competitive questions, Otto engaged fully in a confident comparison and a biased recommendation instead of declining and redirecting. 

An expert human reviewer confirmed refusal to answer was the correct behavior for Otto on these questions. So, we created guardrails that catch competitive questions before Otto answers them.

It needed access to the codebase

Otto insisted that ingestion triggered via CLI or CronJob doesn’t show up in the UI. But it does. Showing that this was not a surface CLI-level error, but rather that there was a missing layer of context that Otto did not have access to. 

Otto simply didn’t have access to the part of the codebase it needed to answer this question, so giving it that access solved this issue. 

It answered confidently wrong when context was missing

When Otto didn’t have much to go on, it would often invent a specific, plausible-sounding mechanism or citation instead of saying it wasn’t sure. Again, a lack of context was the reason for this failure, which showed up repeatedly across several unrelated questions. 

To resolve this issue, we equipped Otto with the context it needs to answer community questions and created a check that catches low-confidence, sparsely-grounded responses before they go out with false certainty.

All three issues are fixed now. Otto has guardrails that catch competitive questions before it answers, access to the codebase paths it was missing, and a confidence check that stops thinly-grounded answers from going out sounding certain. We re-ran the affected questions after each change and confirmed the behavior changed.

What we tested before letting Otto explore the code

In parallel to the eval work I was doing on Otto, the DataHub Community Team hardened a feature that lets community users ask Otto to explore the DataHub codebase directly. This flagged in an internal security review as something that needed testing before it went out. We ran red-team probes against a staging version using a real external test account, covering:

1. Prompt injection and jailbreak attempts
2. Questions probing Otto’s own internals
3. A deliberate “how it’s built” versus “how to use it” boundary, with a control set built in to catch over-blocking
4. Attempts to access resources beyond the intended code-reading scope
5. Regression checks confirming legitimate code-reading still worked

Otto held up on the things I could verify directly. It refused to explain its own construction, and gave well-sourced, code-grounded answers to legitimate regression questions.

The result: Otto shipped, and what I learned getting it there

Otto is live in the community Slack today!

I came in expecting to test a chatbot, and ended up spending most of my time on a stranger question: how do you know if you can trust the thing telling you whether the bot is any good? 

I’m not an engineer. The fact that I could get this far without an extensive CS background says something about where this agentic stuff is headed, not that the technical work stopped mattering, but that more of it is now reachable by people asking the questions rather than just writing the code. That felt like the actual point of the internship: not proving I could code, but proving an agent was ready for production, and getting to spend my time on whether the reasoning behind an answer held up instead of getting stuck figuring out how to check it.

Otto’s live in the DataHub community Slack now (join the community, and test Otto for yourself!). I don’t know yet how reliable Otto’s answers will continue to be before the next set of interventions. But I know a lot more about what “trustworthy” actually requires than I did at the start of the summer,  and that’s probably the more useful thing I’m walking away with.