AI tested the AI, and caught a real design flaw before it shipped

An orchestrator that both coordinates a workflow and decides outcomes is doing two jobs at once — a subtle architecture problem that’s easy to miss reading the code, because nothing about it looks wrong in isolation.

Shelby built evals to systematically test the AI Intake Coordinator’s orchestrator against representative cases rather than trust it by inspection. The evals surfaced exactly that flaw.

“The orchestrator has been fighting me because we’re asking it to be the conversational agent and telling it to stay out of things. Re-architecting so it has one job and stays behind the scenes, with dedicated agents for welcome and end-of-conversation.”

— Shelby Corbitt, Staff AI Solutions Engineer

The orchestrator was re-architected to a single responsibility — sequencing only, with decisions delegated cleanly — and re-verified with the same evals before anything shipped to Production. It’s the same determinism principle TracE first taught the team, caught this time by AI testing the AI.

“The intake breakthrough is huge… we were breaking our own rules and didn’t realize it.”

— Michael Pietrangelo