DynamoDB scheduler: how we decide when a recurring event fires
Our DynamoDB scheduler never sleeps until a run time. It writes every instant of a recurring rule as a row up to 90 days ahead, and a sweep that runs every 300 seconds on a single leader fires the rows whose time has passed, through one DynamoDB transaction that claims the row and emits the event. An event therefore runs at its fire_at instant rounded up to the next 300-second tick, plus the delivery time of a durable queue, and it fires once across the whole worker fleet.
This post covers the five stages of that path in our backend's calendar module, the window math that decides which rows exist, the transaction that makes each fire single, and a 6 to 24 hour gap where a row is written but never fired.
The short version
- A rule is expanded into one PENDING row per instant, from 24 hours back to 90 days ahead, with a cap of 1,000 instants per pass.
- A leader-gated tick every 300 seconds queries seven hourly buckets of a sparse index and fires the rows whose
fire_athas passed. - One
TransactWriteItemscall with two writes emits the event and flips the row from PENDING to FIRED, so two workers can never fire the same occurrence. - Materialize reaches back 24 hours but the tick reaches back six, so a row whose instant is 6 to 24 hours old at schedule time is never fired.
Why a sleep loop is the wrong scheduler
The first scheduler in most codebases is a loop that sleeps and checks what is due. We wrote about how that loop multiplies in how codebases grow three schedulers nobody designed. It fails in three specific ways: it keeps no record of what it fired, it fires once per process instead of once per fleet, and it loses its place on every restart.
DynamoDB has no primitive for "run this at 07:30". So the design writes the instants down as data and sweeps that data on a fixed tick. No process holds a timer for a specific event.
| Approach | What decides the run time | Fires once per fleet | Survives a restart | Record of what fired |
|---|---|---|---|---|
| Sleep loop in the app process | An in-memory timer | No, once per instance | No | None |
| Cron on every host | The host crontab | No, once per host | Yes | Logs only |
| Materialize and sweep (ours) | fire_at on a stored row, checked by the tick | Yes, by a conditional claim | Yes, rows are durable | Every occurrence row, with its status |
The five stages from schedule call to handler
Stages 1 and 2 run inside the API call; stages 3 to 5 run later in the worker. Stage 3 is the only one that reads the clock to decide whether something runs.
The schedule call stores the rule, not a time: a recurrence, an anchor_at UTC instant that is both the first run and the wall-clock template for later runs, an IANA timezone, an optional cron string and an until_at hard stop. The event id is source_module#external_ref, so a repeat call is a no-op. If either half is missing, the id falls back to a fresh UUID and every call creates a new event, which is the easiest way to get duplicates by accident.
Materializing instants into a 90-day window
Materialize turns one rule into concrete instants and writes one occurrence row per instant. The expander is a pure function with no clock and no database: it takes a rule, a now and a window. That makes DST and month-end cases plain unit tests, with no time mocking.
after = max(anchor_at - 1s, now - 24h) # never replay more than 24 hours
horizon = now + 90 days # materialize_horizon_days
keep = after < fire_at <= horizon
The left edge is the later of two values. Without the 24-hour floor, expanding an old daily rule for the first time would write, and then fire, weeks of missed history.
Expansion rules
HOURLY steps in fixed UTC hours and ignores DST. DAILY and coarser rules add N units to the anchor in local time and convert back, so the local hour survives DST changes. MONTHLY clamps the day, so an anchor on the 31st runs on April 30 and February 28. CRON tests each minute against the five fields in the event's timezone.
One pass stops at 1,000 instants. A cron that fires every minute reaches that cap after about 17 hours of horizon, and the next top-up continues from the last row.
The row and the sparse index
Each occurrence id is <fire_at_iso>#<event_id>, written with put_if_absent, so one (event, instant) pair can exist once. A PENDING row also gets gsi1pk = DUE#YYYY-MM-DDTHH, the UTC hour of its fire_at. Rows skipped by a holiday policy are written as SKIPPED without that key, so they show in the UI and never fire.
Only PENDING rows carry the DUE key, so the index is sparse and holds only work still to do. The tick never scans the base table; it queries a handful of hour partitions.
The tick: a 300-second sweep behind a DynamoDB lease
Every worker runs this loop, and only the lease holder gets past the first step. The lease is a row in a DynamoDB locks table rather than a Redis lock, so the scheduler depends only on a store the rest of the platform already uses.
One failure here is silent. If the ARGOS_TABLE_LOCKS environment variable is not set, no worker ever wins the lease, no tick drains, and nothing in the calendar raises an error. Events simply stop firing.
The seven hourly buckets, the current UTC hour and the six before it, are the catch-up budget after an outage. A row older than that is in no swept bucket and stays PENDING on the base table.
The claim: one transaction, two writes
For each due row the scheduler binds the row's tenant and runs one TransactWriteItems call. The first write puts a cal.event_fired event row on the durable lane. The second updates the occurrence to FIRED, removes its DUE keys and carries the condition status = PENDING.
# Simplified: the emit and the claim commit together or not at all.
client.transact_write_items(TransactItems=[
{"Put": {"TableName": EVENTS_TABLE,
"Item": event_fired_row(occ)}}, # lane = DURABLE
{"Update": {"TableName": CAL_TABLE, "Key": occ.key,
"UpdateExpression": "SET #s = :fired, fired_at = :now "
"REMOVE gsi1pk, gsi1sk",
"ConditionExpression": "#s = :pending",
"ExpressionAttributeNames": {"#s": "status"},
"ExpressionAttributeValues": {":fired": {"S": "FIRED"},
":pending": {"S": "PENDING"}, ":now": {"S": now_iso}}}},
])If a second worker with a stale lease reaches the same row, its condition fails and DynamoDB rolls back both writes, so no second event row exists. The same write drops the row from the DUE index, and an occurrence can never be FIRED without its event row. We lean on the same conditional-write pattern for device commands with durable acks over MQTT.
If the transaction does not commit, the row stays PENDING and the next tick retries it. A FAILED status exists in the enum, but no code path sets it today.
From event row to handler
The event row reaches handlers through the relay described in our outbox on DynamoDB: DynamoDB Streams, EventBridge Pipes, then an SQS FIFO queue. A worker binds the tenant from the envelope, claims {event_id}:{subscription} in an idempotency inbox and runs each matching durable subscription. Subscriptions filter on layer_code, so a module sees only its own layer, and tenant binding follows the same tenant isolation rules as the rest of the backend.
Delivery is at least once. The calendar guarantees the occurrence fired once. It does not guarantee your handler ran once, so every handler must be idempotent.
The six-hour top-up keeps 84 to 90 days of rows ahead
Every six hours the loop runs two passes under separate leases. reconcile_declared writes every module's declared specs with put_if_absent, so a spec that ships in a deploy becomes rows without an API call. materialize_horizon reads every recurring event from the ACTIVE_EVENTS index and runs materialize again.
Because the row id starts with fire_at, rewriting an existing instant is a no-op, and only instants that newly entered the horizon are added. A recurring event therefore always has 84 to 90 days of rows ahead of it. After an outage, every instant still in the future is intact; instants that passed during the outage are caught up only if they fall inside the six-hour lookback.
Where it fails today: the 6 to 24 hour gap
Materialize backfills 24 hours. The tick looks back six. Any rule whose anchor is between those two numbers at schedule time gets a PENDING row that no tick will ever read. This came out of documenting the firing path against the code, and here is the exact case:
The stranded row still shows in list_occurrences, so it looks scheduled, and no recovery tool exists today. Until the backfill and the lookback agree, anchor recurring events in the future.
Two smaller gaps sit next to it. A ONCE event anchored more than 90 days out is outside the window and is not on ACTIVE_EVENTS, so the top-up never revisits it and it never fires. And the interval settings exist, but the loop hardcodes both intervals.
What this lets EverBowl™ do
EverBowl™ learns how a dog eats and drinks over months. Per-day work on that record, such as the daily summary of how a dog's day went, runs on this calendar, because it is the only scheduler our lints allow. DAILY rules expand in local time, so a 07:30 IST job runs at 07:30 local every day, and the claim transaction means each day produces one record, not two.
The honest limit is timing and the first day. A job runs within one 300-second tick of its instant, not to the second, and a job anchored 6 to 24 hours in the past loses its first day. EverBowl does not diagnose; the value is the record.
Conclusion
If you build recurring jobs on DynamoDB, store the rule, materialize instants into a bounded window, and let one leased sweep fire them through a conditional transaction. Keep the expander pure so DST and month-end cases stay unit tests. Make the backfill window and the sweep lookback the same number, or you will strand rows the way we do today. Make every handler idempotent, because the calendar fires once but delivery is at least once. More posts on how our backend is built live in the Software category.
FAQs
How do you schedule recurring jobs in DynamoDB?
DynamoDB has no timers, so store each future run as a row with its run time in the key. A periodic sweep queries the rows that are due and claims each one with a conditional write. A sparse GSI keyed by hour keeps the sweep from scanning the table.
How does the scheduler avoid firing an event twice?
Two layers. A DynamoDB lease lets only one worker sweep per tick, and the claim is a transaction whose update requires status = PENDING. If two workers race, the second condition fails and both of its writes roll back.
What happens to scheduled events during an outage?
Future instants are already stored as rows, so none are lost. Instants that passed during the outage fire on the first tick back only if they are within the six-hour lookback; older ones stay PENDING.
How do you cancel a recurring event?
cancel_event marks the event cancelled, removes it from the top-up index, and moves every PENDING occurrence to SKIPPED with its index key removed. Rows that already fired do not change, and the event row stays for audit.
