The worst failure mode for a back-office agent is not a crash. A crash gets noticed. The failure that costs money is the quiet one: the timesheet chaser ran every morning for five months, then a token rotated, and for eleven days it did nothing at all. Nobody filed a ticket, because nothing appeared broken. The chases just stopped arriving, and the first sign was a pay run with forty missing timesheets instead of the usual four.
Agents are unusually good at failing quietly, for a boring reason: their output is an absence. No exception email looks exactly like a clean day. So the monitoring has to watch two separate things — did the agent run, and is the underlying work actually getting done — because either one can be fine while the other rots.
This is the layer we build alongside every agent, before handover. It is a few tables, three alerts and one email. It is not observability in the vendor-conference sense.
Part one: a run log the agent cannot skip
Every scheduled run writes one row when it starts and updates it when it finishes. The row is written first, on purpose — a run that dies in stage two still leaves evidence.
create table agent_run (
run_id bigserial primary key,
agent text not null, -- 'timesheet_chaser'
started_at timestamptz not null default now(),
finished_at timestamptz,
status text not null default 'running', -- running|ok|failed
stage text, -- last stage entered
error text,
trigger text not null default 'schedule' -- schedule|manual|replay
);
create table agent_run_counter (
run_id bigint not null references agent_run(run_id),
name text not null, -- 'timesheets_missing', 'chases_sent', 'escalated'
value bigint not null,
primary key (run_id, name)
);
Counters matter more than logs. A log line tells you something happened; a counter tells you how much, and a number that moves to zero is the cheapest anomaly detector there is. For a timesheet chaser we count contractors in scope, timesheets missing at cutoff, chases drafted, chases sent after approval, escalations raised, and API errors swallowed.
That last counter is not optional. Retries are good practice and they are also how a broken integration hides. Count every retry and every error you decided to tolerate, and put both in the digest.
Part two: three alerts, and only three
Alerting on everything trains people to ignore alerts. We ship three, and each one names the human it wakes.
1. The run did not happen. A dead-man's switch, not a failure handler. If no agent_run row for timesheet_chaser reached status = 'ok' inside its expected window, something outside the agent broke: the scheduler, the container, the credential.
select a.agent,
max(r.finished_at) as last_ok
from agent_expectation a
left join agent_run r
on r.agent = a.agent
and r.status = 'ok'
group by a.agent, a.max_gap_minutes
having max(r.finished_at) is null
or max(r.finished_at) < now() - (a.max_gap_minutes * interval '1 minute');
Keep agent_expectation as a real table — one row per agent, with the expected cadence and the escalation contact — and run this check from something that is not the agent. A process cannot credibly report its own absence.
2. The run happened and did nothing. Compare today's headline counter against its own recent history rather than a hand-set threshold, because volumes swing with the week and the season.
with d as (
select r.run_id,
r.started_at::date as day,
max(c.value) filter (where c.name = 'chases_sent') as chases
from agent_run r
join agent_run_counter c on c.run_id = r.run_id
where r.agent = 'timesheet_chaser'
and r.status = 'ok'
and r.started_at > now() - interval '60 days'
group by r.run_id, r.started_at
)
select (select chases from d order by day desc limit 1) as today,
percentile_cont(0.5) within group (order by chases) as median_60d
from d
where day < current_date;
Alert when today is zero and the median is not, or when today is below a fraction of the median you picked deliberately. We usually start at a quarter and tighten once we have a season of data. Expect false positives around holidays; a Monday after a client shutdown looks like an outage.
3. The work is stale, whatever the agent thinks. This is the one that catches the failure the other two miss: the agent runs, sends its chases, reports success, and the chases are going to a mailbox nobody reads. So check the outcome, in the systems of record, independently of the agent.
select c.contractor_id, c.full_name, t.week_ending, t.chased_count, t.last_chased_at
from timesheet_status t
join contractor c using (contractor_id)
where t.state = 'missing'
and t.week_ending < current_date - interval '10 days'
and t.chased_count >= 3
order by t.week_ending;
Three chases and still missing after ten days is not an automation problem, it is a person problem, and it belongs in front of the account manager. An agent that keeps chasing indefinitely is a robot arguing with a wall.
Part three: the daily digest
One email, one named recipient, every morning, including the quiet days. A digest that only arrives when something is wrong is indistinguishable from a digest whose sender has stopped working — the same trap the agent fell into.
What goes in it:
- Ran at 06:05, finished 06:09, status ok.
- Counters, with yesterday and the 60-day median beside each.
- Escalations raised, with the human each one went to.
- API errors and retries, by endpoint.
- The staleness list: work the agent cannot finish on its own.
What stays out: anything that reads like a score. Nobody needs a health percentage. Operators want the numbers they already argue about — missing timesheets, approvals outstanding, credentials expiring inside thirty days.
Field renames, deprecations and the other slow leak
Tokens are the fast failure. Schema drift is the slow one. An ATS adds a status value, a back-office vendor renames a field, a custom field your matching rules depend on gets retired by an admin who had good reasons, and the agent carries on producing plausible nonsense.
Two cheap defences. First, validate the shape of what you pull and count rejects: any row that fails validation increments rows_rejected rather than getting dropped in silence. A reject count that goes from zero to two hundred overnight is a schema change with a timestamp on it. Second, pin the API version where the vendor offers one, and subscribe a real mailbox to the platform's developer changelog — Loxo, JobAdder, Crelate, Vincere and Recruiterflow all ship changes on their own cadence, and none of them will call you.
The audit overlap
If the agent touches screening, this monitoring layer is doing double duty. The run log already answers most of what an NYC Local Law 144 or Illinois HB 3773 record request asks about process: when the tool ran, on which records, what it produced, which human acted. Keep run_id on every decision row you write and the audit export becomes a join instead of an archaeology project. That is not a compliance claim about the tool — it is just the records being in one place when someone asks for them.
What this costs
A day or two of engineering on top of the agent, and about fifteen minutes of somebody's morning. Skip it and you are running an unattended process against live client data on the assumption that silence means success.
We build this into every engagement rather than selling it separately, because an agent without it is not finished. If you are running something homegrown and you cannot answer "when did it last do useful work" from a query, that is the place to start — before the next agent, not after.
If you want a second pair of eyes on what you have running, get in touch with your ATS, contractor headcount and what the agent is supposed to do each morning. We reply within one business day.