// project
AI Product Engineer Copilot — A multi-step agent that plans, cites its sources, and grades its own work
How the agent is built end to end, and why the eval/observability layer, not the agent loop itself, is where most of the engineering effort went.
Most agent demos stop at 'it can call a tool.' I wanted to answer a harder question: how do you actually know an agent is getting better, not just different — and how do you prove it didn't just make up that analytics number?
// the problem
PM-facing agent tools are easy to demo and hard to trust. A generated PRD can look polished and still cite a metric nobody measured, skip a step because a node degraded silently, or regress on exactly the case that mattered — with no way to catch any of it before a person reads the whole thing.
// the solution
Design the rigor layer in from day one, not bolted on after: a supervisor + sub-agent graph where every deliverable after the PRD depends on it explicitly instead of guessing in parallel, MCP tool calls that degrade to a visible note instead of a fabricated answer, and an eval harness with a golden set, a bias-checked LLM judge, and a four-tag failure taxonomy that can specifically catch a hallucinated claim.
A supervisor + sub-agent graph, not a flat fan-out
The PRD agent runs first, alone — every other deliverable (user stories, architecture review, experiment design) is a downstream consumer of what it decides, not an independent parallel branch. Those three fan out in parallel once the PRD exists in shared graph state, a roadmap agent joins all of it into a sequenced plan, and an assembler merges the five outputs into one response. A flat five-way fan-out was the first draft, and it broke immediately: an architecture reviewer can't review an architecture nobody proposed yet.
MCP tools grounded in real data, not fixtures
Two MCP servers back the graph: one does embedding-based search over this project's own docs via pgvector in Neon Postgres, the other pulls live GitHub repo stats through the GitHub API, cached with a 1-hour TTL. A failed tool call degrades to a visible '[data unavailable]' note in the affected deliverable instead of crashing the run or letting the model quietly invent a number — the same anti-hallucination discipline whether the failure is a flaky API or a rate limit.
An eval harness whose judge has to prove itself first
A golden-set regression harness runs the real graph against a fixed set of requests and grades each deliverable with an LLM judge on a five-point rubric plus the four-tag failure taxonomy. Before it grades anything real, the judge has to correctly score two known-answer control documents — a fluent PRD that says nothing, and specific user stories tied to real numbers — or the run aborts with 'the instrument is broken' instead of a list of false regressions. It isn't run on every visitor's request, since that would roughly double the model spend on a demo whose whole pitch is that it's cheap — it's a pre-deploy check, not a live gate.
// key takeaways
- —A supervisor-first graph shape (PRD before the fan-out) is what makes the rest of the output coherent — parallelism without dependency ordering just produces five deliverables that disagree with each other
- —Real MCP tool use only proves the pattern if the data is real: two servers backed by pgvector search and live GitHub stats, not fixtures, with graceful degradation instead of fabrication on failure
- —An LLM judge is only as trustworthy as its own bias checks — grading known-good vs. known-bad control documents before trusting the judge on anything real
- —Clarifying questions via LangGraph's interrupt/resume landed as one new node and one conditional edge, not a redesign, because the checkpointed graph state it needed was already being built for other reasons
// tradeoffs
- —The graph fans out in parallel after the PRD, but is still statically wired past that first hop — the supervisor's one triage decision (clarify vs. proceed) is the only place the graph actually branches, not a fully dynamic router.
- —Visitor runs are never graded — the eval harness only runs pre-deploy against a fixed golden set, so a real regression on someone else's request wouldn't be caught automatically.