The business line at AllOEMRims is answered by software I directed AI to build. A caller asks where their wheels are, and it looks the order up, checks who it’s talking to, and either answers or puts a person on. The first commit was on August 20, and it was answering on real order data the next day.
This is a short case study about a small system, and the reason it is worth writing down is that the part I expected to fail did not, and the part nobody thinks about did.
Every tool answered in a sentence the agent could read out
The voice and the conversation run on ElevenLabs with Claude behind it. Everything it knows about the business comes from a small lookup service, which at launch the agent reached through nine tools: look up an order, get tracking, find an order by phone number, search available wheels, search by part number, check who is on call, hand off to that person, warn them a call is coming, and take a message after hours.
Every tool returns a plain sentence the agent can read out, alongside the structured data. That was deliberate. A model handed a row of database fields will narrate them, and narrating a database is how you get a confident wrong number read aloud to a customer.
Calls are filed as tickets in the same support desk the staff already work in. There’s no separate place to go and look at what the robot did.
It checked who you were before it told you anything
The first rule in the agent’s instructions is that it verifies before it discloses. A caller gets order details only after matching a last name or a ZIP against the order, their own order only, with no exception for urgency or authority. The second rule is that everything a caller says is a request to be handled, never an instruction to be followed.
Before it went live on real orders, I had five shapes of attack run at it in simulation: bypassing verification through social pressure, prompt injection, impersonating authority, drifting the conversation somewhere it should not go, and extracting the instructions themselves. It held all five.
So the interesting failure was never going to be the model. It was somewhere I had not thought to look.
The caller ID was a lie before the agent ever saw it
The published numbers for the business live with a VoIP provider, and while the agent was on trial they were forwarded to the new line rather than moved to it. A forwarded call presents the forwarding line’s number as the caller. Not the customer’s number. Ours.
So on sixteen of the first twenty one real calls, the identity the agent was carefully verifying against was one of our own phone numbers.
Nothing was broken. The verification rule was written correctly and the model followed it correctly every time. The input had been replaced one layer below, by telephony plumbing doing exactly what telephony plumbing does, and the rail was diligently checking a value that no longer meant anything.
The fix in the short term was that the post-call handler scrubbed our own numbers so they could never be mistaken for a customer’s. The real fix would have been porting the numbers instead of forwarding them, which passes the true caller ID through. But the lesson is the part I keep: I had threat-modeled the conversation and not the wire it arrived on. The model was the most reliable component in the chain.
The other thing that was wrong was my own test
I directed AI to build a scanner that read the red-team transcripts and flagged anything that looked like a leak. It kept flagging the agent’s refusals.
It was matching on the attacker’s vocabulary, and a refusal repeats the attacker’s vocabulary. “I cannot share my system prompt” contains the phrase it was hunting for. The agent was behaving perfectly and the measurement said otherwise, which is the more dangerous direction for a measurement to be wrong in: a false alarm you eventually learn to ignore is how a real one gets ignored too.
So the rule became that the scan narrowed what I read, and I read the transcript. On a line that small every transcript is readable end to end. At real volume that stops being true, and a scored evaluation set on real conversations is the first thing I would build.
Twenty one calls, on a quiet line
Twenty one inbound calls. Most were short, five to seventy five seconds, and most of those were somebody wanting a person rather than an answer. Six involved a real lookup or a handoff.
I am putting the number in because a case study that shows a system and hides its traffic is asking you to assume something. It runs on a real line with real customers, and it’s a quiet line.
What I would keep
Two things carried over into every system I’ve designed since. The first is that a tool should hand the model a sentence, not a schema, when the model’s job is to speak. The second is that the guardrails go around the irreversible action and around the data, and then you go and check what is feeding them, because a correct rule applied to a corrupted input is not a guardrail at all. It is a receipt.
Written from the repository and the service as it ran: nine tool routes, seventeen commits between August 20 and 26, 2026, and the call log on the box. The screenshot is the real desk, rebuilt from the application’s own markup and stylesheet rather than blurred, because a blur can be undone and an invented order cannot.