Every agent we build sits on a copy of somebody's ATS. The redeployment radar reads placements and end dates. The chaser reads timesheet submissions and credential fields. The screener reads applications. None of them read the ATS live at the moment of decision, because an API that is down for eleven minutes should not take a Monday-morning work list down with it.
So the real question underneath "can you build us an agent" is usually: how fresh is your copy, and how do you know? This walkthrough builds that sync layer. It is deliberately boring, and it is the part that decides whether an operations lead trusts the daily list six weeks after handover.
Pick the freshness you actually need, per record type
Before choosing a transport, write down how stale each record is allowed to be. Not "real time": a number.
| Record | Tolerable staleness | Why |
|---|---|---|
| Placement end dates | 24 hours | The 30/60/90-day buckets do not move overnight |
| Timesheet submissions | 1-2 hours during a pay run | The chase list has to be right on Monday and Tuesday |
| Credential expiries | 24 hours | Lead times are measured in weeks |
| New applications | 15-30 minutes | Candidate response rates fall away fast |
| Job status changes | 1 hour | A closed req should stop generating outreach drafts |
Most of that table is satisfied by a nightly pull. One or two rows are not, and those are the only places worth paying the complexity cost of webhooks. Firms tend to arrive asking for event-driven everything and leave with a nightly job plus two subscriptions.
Option one: polling on a schedule
Polling is a cron job that asks for what changed since the last run. It is the default because it is debuggable at 7am by somebody who did not write it, and because it survives being switched off for a day.
Use the vendor's modified-since filter if there is one, and keep a watermark of your own:
create table sync_state (
resource text primary key, -- 'placements', 'timesheets'
last_synced_at timestamptz not null,
last_run_status text not null,
last_error text
);
SINCE=$(psql -At -c "select last_synced_at from sync_state where resource='placements'")
curl -s "$API_BASE/placements?updatedAfter=$SINCE&limit=100" \
-H "Authorization: Bearer $ACCESS_TOKEN"
Three things to get right, all learned the hard way:
- Overlap the window. Ask for changes since the last run minus an hour. Clocks disagree, vendors index asynchronously, and an upsert keyed on the record id makes duplicates free. A gap is expensive; an overlap costs nothing.
- Advance the watermark only on success. Write
last_synced_atafter the last page has landed, in the same transaction if you can. A job that crashes on page 7 of 9 and still claims it is current will quietly lose a day of end-date changes. - Respect the rate limit, and run it at night. The same discipline as pulling end dates from an ATS API: read the reference, honour the response headers, back off, and do not compete with recruiters mid-search.
What polling cannot do is notice a deletion. If a placement is removed rather than cancelled, a modified-since query never mentions it again. That is what the reconcile in the last section is for.
Option two: webhooks, when minutes matter
A webhook is the ATS calling you. Loxo, Crelate, JobAdder, Vincere and Recruiterflow all offer some form of event subscription, and the details differ enough that you should read your vendor's docs rather than trust a generic tutorial, this one included. What is consistent is the shape of the receiver.
Rule one: the receiver does not do the work. It verifies, writes the event to a ledger, and returns 200 in single-digit milliseconds. Matching, drafting and API calls happen in a worker afterwards. A receiver that processes inline will eventually time out, and most vendors respond to timeouts by retrying, which is how one late payload turns into four.
create table ats_events (
id bigserial primary key,
provider text not null,
event_id text not null, -- vendor's id, for dedupe
event_type text not null,
resource_id text,
payload jsonb not null,
received_at timestamptz not null default now(),
processed_at timestamptz,
attempts int not null default 0,
last_error text,
unique (provider, event_id)
);
That unique constraint is the whole deduplication strategy. Vendors deliver at least once, not exactly once; retries after a network blip are normal, not a fault. An insert ... on conflict do nothing makes a repeat delivery a no-op.
Verify the signature before you read the body
An unauthenticated webhook endpoint is an open write path into your work lists. Check the signature header the vendor sends, over the raw request bytes, with a constant-time comparison:
import crypto from 'node:crypto';
export function verify(rawBody, signatureHeader, secret) {
const expected = crypto
.createHmac('sha256', secret)
.update(rawBody) // raw bytes, before any JSON parsing
.digest('hex');
const a = Buffer.from(expected, 'hex');
const b = Buffer.from(signatureHeader, 'hex');
return a.length === b.length && crypto.timingSafeEqual(a, b);
}
If your framework parses and re-serialises JSON before you see it, the bytes change and every signature fails. Capture the raw body on that route specifically. Also reject payloads with a timestamp older than a few minutes, so a captured request cannot be replayed at you next week.
Treat the payload as a hint, not as data
Most event payloads are thin, and some are stale by the time you read them: two edits inside a second can arrive out of order. So do not trust the values in the body. Take the resource id from the event, then re-fetch that record from the API and upsert it. The event tells you what changed; the API tells you what it is now. This one habit removes an entire category of "the agent used an old bill rate" incidents.
Process out of the ledger, with a bounded retry
The worker claims unprocessed rows, does the work, and stamps them. Keep the retry ladder short and visible, and stop:
update ats_events
set attempts = attempts + 1
where id = $1;
After three or four failures, leave the row unprocessed, put it on the daily exceptions list with its error, and let a person look. Infinite retries hide bugs; a dead-letter row on somebody's morning list gets fixed.
Run both: the nightly reconcile
Webhooks are for latency. Polling is for truth. Every production sync we run does both, and the nightly pull exists specifically to repair what the event stream got wrong:
- Events dropped while your endpoint was down for a deploy.
- Changes the vendor does not emit an event for at all, which is usually the interesting custom field.
- Deletions and merges. Candidate merges are the classic: two records become one, and your copy keeps a ghost.
- Bulk edits made in the ATS UI or by an import, which some platforms do not emit per-record.
So the reconcile pulls the full active set once a night, upserts it, and marks anything active in your copy but absent from the ATS as gone rather than deleting it. Then it writes down how bad the drift was:
create table sync_health (
resource text not null,
checked_at timestamptz not null default now(),
records_in_ats int not null,
records_local int not null,
drift_count int not null, -- rows the reconcile had to fix
max_lag_minutes int not null -- oldest unprocessed event
);
drift_count is the number that matters. On a healthy sync it is a handful of rows a night. When it climbs, something changed at the vendor end: a new field, a revoked token, a subscription silently disabled. You want to hear that from your own dashboard rather than from a recruiter who called a contractor whose assignment was extended a fortnight ago.
What to put in front of the operations lead
One line, on the same daily email as the exceptions list:
Placements synced 04:12, 0 events pending, 3 records corrected overnight. Timesheets synced 08:05, 2 events pending.
That sentence is what makes an agent's output arguable. Without it, every wrong row is a question about the agent's judgement. With it, most wrong rows resolve into a sync problem or an ATS-hygiene problem, both of which have an owner and a fix. It is the same reason our Redeployment Agent and Timesheet and Compliance Chaser report their own freshness: an unexplained list gets ignored by the second month.
And the honest not-yet: if you have one ATS, no separate timesheet system, and a nightly pull satisfies every row of your staleness table, do not build a webhook receiver. You would be adding a public endpoint, a secret to rotate and a queue to monitor in exchange for latency nobody in the office can use.
If you want this wired up against your own stack, contact us with your ATS, your timesheet system and which lists have to be right by 8am. We reply within one business day.