DynamoDB read capacity: three background jobs that wasted it

DynamoDB read capacity: three background jobs that wasted it

By Rupam Biswal

Three background jobs, none of them serving a single user request, consumed nearly all of the DynamoDB read capacity on our production account. One scheduler table absorbed a GetItem count in the millions over a single week. A second job read a full batch from every index shard on every sweep in order to archive nothing. A third let more than one worker run the same scheduled tick, multiplying the first two. The same audit surfaced a functional bug hiding inside the cost: a tenant list with a page-size limit and no cursor meant most households never received their daily summary.

This post covers how we found it from metrics rather than symptoms, what each loop was actually asking DynamoDB to do, why the first proposed fix would have broken a feature, and the 3 rules we now apply to any background read.

The short version

  • A leader lease released the moment its work returned, so more than one worker ran the same scheduled tick, and a longer lease alone would not have fixed it: coordination is not an idempotency key.
  • A periodic sweep made 7 GetItem calls per tenant for cursors that only change at midnight, and emitted an event with 0 subscribers.
  • An archiver read a full batch per shard and filtered by date in code, billing every row read and archiving 0.
  • The tenant list behind the sweep stopped at one page with no cursor, so every tenant past that page never got the pulse that writes its daily summary.

How we found it

We started from the bill, not from a slow endpoint. A week of per-table, per-operation metrics showed a small scheduler table carrying orders of magnitude more read capacity than the table holding all transactional app data, which is roughly fifty times larger. That inversion is the tell: bookkeeping should never outweigh the thing it books. The operation mix settled it, because the GetItem count tracked the tick cadence multiplied by a few hundred, which nothing in the product could produce.

Three background jobs on DynamoDB and what each asked the database to do before and after the fix

Each job was ordinary code that was correct at a smaller scale.

Job one: a lease released too early

Our workers scale horizontally. Anything that must happen once across the fleet guards itself with a DynamoDB conditional-write lease: one item per lock with an owner and an expiry, acquired by a PutItem that succeeds only when the lock is absent or expired. That primitive was fine.

The bug was in the helper around it, which released the lease in a finally block the moment the action returned. For one-shot work that is correct. For a loop that sleeps between runs, it goes wrong as soon as there is a second worker. Worker A finished its tick in seconds and deleted the lock; worker B, on a phase-shifted timer, woke, found the lock free, and ran the whole tick again. PutItem calls on the lock table ran well above the single-worker baseline, which is how we sized the duplication.

The first correction inverts the default so a successful run keeps the lease for the rest of the window, and only a failure releases it early:

python

async def run_if_leader(name, action, *, ttl_seconds, release_on_success=False):
    lease = await lock.acquire(name, ttl_seconds=ttl_seconds)
    if lease is None:
        return False
    try:
        await action()
    except BaseException:
        await lock.release(lease)   # let another worker retry this window
        raise
    if release_on_success:          # one-shot callers opt in
        await lock.release(lease)
    return True

Releasing on failure has a condition attached. If the action can fail partway through its work, another worker picking up the window immediately will repeat whatever already happened, so the action must be idempotent or carry its own recovery semantics. Ours is, and the next section is why.

How a leader lease released too early let a second worker re-run the same scheduled tick, and why a longer lease alone is not enough

A lease is coordination, not an exactly-once guarantee.

A longer lease is not a correctness argument

It is easy to ship that change and believe the problem is solved. It is not. Consider a lease that expires just before the next tick: worker A completes its action, the lease lapses, worker B wakes on a phase-shifted timer before A's next run, acquires the free lease, and executes the same logical window again. The lease reduced contention; it did not make the window run once.

What makes the window run once is naming it. Each sweep identifies its slot, a date for daily work, and records that slot on completion. A worker that acquires the lease and finds the slot recorded returns immediately. The lease keeps two workers from colliding; the slot record makes a collision harmless when it happens anyway.

One smaller consequence is easy to miss: the expiry comparison is strict, so a lease exactly as long as the sleep lets a worker waking on the expiry second lose its own next run. The TTL now sits a little under the interval.

Job two: seven reads per tenant for a value that moves once a day

A calendar component emits boundary pulses, one per tenant when a period changes, so other components can do end-of-period work without each running its own timer. The daily summary in how a day of bowl events becomes one trustworthy paragraph is triggered by exactly this day boundary.

On every tick the sweep listed the tenants, bound each one, and for each of 7 period grains did a GetItem on that grain's cursor to see whether the period had advanced. Tenants times 7 reads times the tick cadence times the duplication from job one lands within a few percent of what we measured. Six of the 7 cursors can only change at midnight, so nearly every tick read them to learn nothing.

The finest grain was worse than wasteful. Nothing subscribed to it, yet it was emitted per tenant on the durable lane every period, a transactional write plus a cursor update each time. It was the largest write load on the account and nothing consumed it.

The fix that would have broken the product

The first proposal was obvious: emit each pulse once, globally, instead of once per tenant. It would have cut the writes to almost nothing. It would also have silently stopped every daily summary, because the summary handler relies on the pulse binding the tenant it belongs to. The handler's own docstring says so.

A cost fix that removes the tenant from a per-tenant event is not a fix. It is the same failure, arriving faster.

What shipped keeps the per-tenant pulse and removes the waste around it. The sweep reads all of a tenant's cursors in 1 Query instead of 7 GetItem calls, and the calendar lookup runs once per date, only when something advanced. It pulses the day-or-coarser grains only; the finest grain stays declared but no longer fans out. Because all the swept grains move only at midnight, a single watermark row records the date the sweep last completed, so every tick in between costs 1 read.

The watermark is written last, and only if every tenant succeeded, because writing it first leaves the remaining tenants starved when a worker dies mid-sweep. Writing it last has the opposite consequence, worth stating plainly: a crash before the write makes the next sweep repeat tenants that already succeeded. That is safe only because emitting a pulse is idempotent per tenant and date. Trusting the watermark alone would have been the same mistake as trusting the lease alone.

Job three: a filter is not a key condition

Our change-data feed keeps one row per tracked write and archives rows past an age threshold to object storage. On each sweep the archiver asked every index shard for its oldest batch and kept the rows past the cutoff. For a week it read every one of those rows and archived 0, because nothing was old enough yet.

DynamoDB bills a Query for the items the key condition matches, before any filter runs, whether that filter is a FilterExpression or a list comprehension afterwards. Reading a batch to discard it costs what reading a batch to use it costs.

The index sort key already began with the timestamp, so the cutoff belonged in the key condition, where the index narrows the read before anything is billed. An empty sweep now matches no items and consumes the operation's minimum. The same change added cursor pagination, so a backlog larger than one batch drains in a single sweep rather than leaking a batch per shard per run, the behavior we wanted in turbo catch-up.

Read patternItems billed when nothing is dueItems billed with 5 due rowsDrains a backlog in one sweep
Oldest batch per shard, filtered in codethe full batch, every shardthe full batch, every shardNo, one batch per shard per sweep
Cutoff in the key conditionnone matched5Yes, pages by cursor

The bug the cost audit was hiding

The tenant list feeding the sweep queried a global index with a page-size limit and no cursor, written when that limit was generous. It is not generous now. Every tenant past the first page was never visited, so its day pulse never fired and its daily summary was never written. No error, no alarm, just an empty journal for most of our households.

A paginated query with a page limit and no cursor silently truncates, so tenants past the first page receive nothing

The query returned a page of valid rows every time. Validity is not completeness.

The helper now follows the index cursor until it is exhausted. The first sweep after deploy visits every tenant once; after that the watermark keeps the steady state at 1 read per tick.

Three rules we now apply to any background read

A list with a limit and no cursor is a bug waiting for growth. It returns the right answer until the day it does not, and that day arrives silently. Every cross-tenant enumeration now pages until the cursor is exhausted, or states why it cannot.

Anything filtered after a DynamoDB read was billed as if returned. If the discriminator is in the key, put it in the key condition. If it is not, that is a schema question, not a filtering question. The same reasoning drove the sort-key design in why pet data needs a knowledge graph.

Coordination is not idempotency, and neither is a watermark. A lease stops two workers colliding. A watermark skips redundant work. Only a per-unit idempotency key makes a repeat harmless, and scheduled work needs all three to be both cheap and correct.

We expect the scheduler table's reads to fall by more than an order of magnitude and the archiver's index reads to fall to 0, confirmed from the same metrics that found the problem. More backend patterns sit in the software category.

Why this matters at Hoomanely

Hoomanely builds connected pet-care products that help a pet parent understand the animal in front of them. Almost everything a pet parent sees in the app is produced by scheduled work: the end-of-day summary, the periodic insight, the nudge timed to a sensible local hour. Background jobs are not an internal concern here; they are how most of the product's value is delivered.

That is why this audit mattered beyond the invoice. Read capacity is a shared, finite budget, and three jobs spending it on bookkeeping were taking it from work that reaches a household. The functional bug sharpens the point: the job was expensive while not working, and nothing was positioned to say so.

What this lets EverBowl do

EverBowl™ from hoomanely learns a dog's eating pattern from the bowl and, at the end of each day, writes one summary of what it understood into the pet's journal. That summary is triggered by the day boundary this post is about. Before the fix, most households never received the pulse, so EverBowl had learned the day and never wrote it down. Now every household gets its pulse, and the fleet spends its read budget on recognizing patterns rather than re-reading cursors.

The honest limit is that the pulse only fixes delivery. If a household had no bowl events that day, there is nothing to summarize and no entry is written. EverBowl does not diagnose; the value is the record.

Conclusion

Three background jobs cost us most of our read capacity and a feature that quietly did not work for most households. All three were ordinary code that was correct at a smaller scale, the kind of defect no review catches and no alarm fires on. If you run scheduled work on DynamoDB, pull a week of per-table, per-operation metrics and compare the GetItem and Query counts against what your product could plausibly generate. Then check every enumeration for a missing cursor, every post-read filter for a key it could have used, and every lease for what happens when a second worker wakes one second after it expires.

FAQs

Does releasing the lease on failure risk duplicate work?

Yes, if the action is not idempotent, because another worker can pick up a window that already partially completed. We release on failure because our per-unit work is idempotent and a stalled window is worse than a replayed one. If yours is not idempotent, hold the lease and let it expire instead.

Why not use a different lock backend for the scheduler?

The bug was never the primitive. A conditional write gives one serverless, in-account coordination mechanism with no extra dependency to operate or fail over, and swapping it would have changed nothing about release timing or window identity.

Would a FilterExpression have been cheaper than filtering in code?

No. DynamoDB applies a FilterExpression after reading the items matched by the key condition, and bills for what was read rather than what was returned. The only way to reduce billed items is to narrow the key condition, which is why the cutoff moved into the sort-key bound.

How did a page-size limit survive as the tenant count grew?

Because it never errored. The helper returned a full page of valid rows every time, the sweep emitted a full page of valid pulses, and the tenants it skipped produced no event at all. The signal that finally exposed it was a read count explainable only by exactly one page of tenants.

References

More on Software →

Read more