What did not work, and what I learned from it

Everyone likes to talk about what works. This post is about what did not, or not yet, since I started handing part of my work to AI. These failures taught me more than the successes did. They set the rules I now apply to every new agent.

A voice agent on hold

At Oliverlist, the voice agent for client calls is on hold. I would rather say it plainly: what works in writing does not always carry over to voice.

In writing, AI prepares and a person can review before anything is sent. A clumsy email gets fixed before it leaves. On the phone, every sentence goes out live. There is no draft, no review, and no second chance at a first impression.

A conversation also runs on things an email does not have: tone, hesitation, interruptions, silence. And when an agent speaks on behalf of a client, that client has to be comfortable with every word it says.

Writing has another advantage: everything is on record. You can review a week of exchanges in an hour and see where the agent goes wrong. Listening back to calls takes far longer.

My takeaway: start with written channels, where a human check is still possible before anything happens. And build an off switch from day one. Pausing an agent is not an admission of failure. It is a feature the agent must have.

Too many alerts kill the alert

At Oliverlist, Sorter classifies every prospect reply every 5 minutes. A reply marked as hot creates a task for the team. The problem: it flagged too many replies that were not hot. Every false alert costs a little attention, and attention wears out. An alert that is often wrong ends up ignored, including on the day it is right.

I reworked the sorting instructions over the past few days, and the measurement is under way. But the lesson goes beyond this case. Cutting false alerts must never mean missing real ones. So count both, false alerts and misses, before and after every change. For misses, there is no shortcut: someone has to review a sample of what the sorting set aside.

Another question helps here: what should the person receiving the alert do, right now? If the answer is "nothing, it is for information", it is not an alert. It is a report, and it belongs somewhere else.

Nightly checks should follow the same rule: only anomalies get escalated. A check that flags everything flags nothing.

Checks nobody read

I also built checks that nobody read. A report nobody opens checks nothing. It only gives the illusion of cover.

The trap is easy to fall into. A check takes minutes to build. You add one for every worry, never remove any, and the report grows until nobody opens it.

A useful check combines three things: a named recipient, a threshold and an expected action. The alert should land where people already work, with the detail one click away, rather than on a dashboard someone has to remember to open. With teams spread across countries and time zones, a message that reaches the right person does far more than a dashboard everyone is supposed to check. That is how my agents now work: they write directly to the people concerned.

A simple test you can run this week: for each automated report, ask who read it last week and what they did with it. If nobody can answer, change the recipient or delete the report. A check without a recipient is a cost, not a safeguard.

Effects still being measured

The layer I have been building since July is only a few months old. Its effects are measured in places, not everywhere. For some agents, I cannot yet say whether they save time, money, or nothing at all. I would rather say so than publish a fragile number.

Building takes days; measuring takes weeks. You need a baseline, enough volume, and a way to separate the agent's effect from everything else moving at the same time: seasonality, campaigns, changes in the team. If you do not record the starting point before launch, you will never be able to prove the effect, either way.

In practice, before launching an agent, write down three things: the volume handled today, the time it takes and whose time it is, and the current error rate when you can measure it. That is three lines in a document. Without them, you will end up debating the agent's effect instead of observing it.

Key takeaways

Before launching an agent, plan for five things.

1. Human sign-off where mistakes are expensive: money, clients, people.
2. Audits: regularly review a sample of the AI's decisions and count its errors.
3. Monitoring: every agent reports its own failures. A run that fails silently is the real danger.
4. An off switch: the ability to stop an agent without breaking anything, as I did with voice.
5. A metric set before building: the number that should move, and its starting value.