One Way, and Never a Second: How Codebases Grow Three Schedulers Nobody Designed

Somewhere in most mature codebases there are three ways to schedule a recurring job.

Nobody chose that. There is no design document proposing three schedulers, no meeting where a team weighed the options and landed on "several." Look closely and each one had a good reason at the time, each was added by a competent engineer solving a real problem, and each was smaller than the alternative on the day it shipped.

The interesting part is not that codebases drift. It is that the drift is assembled entirely out of reasonable decisions. Every individual step is defensible. The destination is not.

The commit that would pass review

A backend needs a daily summary of a user's activity. Ordinary feature, no architecture required. Here is the implementation most people reach for first:

while True:
    await asyncio.sleep(3600)
    await run_due_summaries()

Six lines. No new tables, no new concepts, no new dependency. It works the first time you run it locally. Landing in a PR alongside the actual feature work, it would pass review in most teams, and blocking it would look like pedantry.

It is also a second scheduler.

Not in intent. In every consequence that matters. It fires on its own clock with no record of what it fired, so nothing can answer whether last Tuesday's run happened. It has no idea other instances of the process exist:

And it is invisible. Nobody outside that one file knows it exists, so six months later somebody in a different module writes the same loop with a different interval and a different opinion about what happens when the handler throws. Now there are two. The third is a formality.

The rule that was already being broken

Our team had a rule about this. Not about schedulers specifically, but the same shape: modules were supposed to talk to each other only through published interfaces, never by reaching into each other's internals. It was written down. It was in bold. Anyone on the team could recite it, and would have said with real confidence that we followed it.

Then we went looking. Eight call sites were importing another module's internals directly, straight past the interface that existed for exactly this purpose.

Nobody had decided to break the rule. Each of those eight was somebody who needed one field, saw a working import two lines away, and took it. The boundary was documented as enforced and enforced by nothing at all.

The lesson is smaller and more boring than the architecture it appears to be about:

A boundary that isn't linted is a suggestion.

Not a weak rule. Not a rule people ignore because they are sloppy. A suggestion, in the literal sense, which is what any rule becomes when following it depends on everybody remembering it under deadline.

One way, and never a second

So the principle went into the engineering guide as a hard rule, followed by the part that makes it real.

The rule reads: one way to do each cross-cutting thing, and never add a second. One event engine. One task engine. One realtime plane. One settings registry. One audit trail. One logger. Two more of the same kind arrived later, establishing exactly one long-term scheduler and exactly one user-facing historical record.

The wording matters more than it looks. It is not "prefer the standard one," because a preference tolerates two things coexisting through a migration that quietly never finishes. More on that failure below, because it is the one that has cost the most.

Enforcement is a script of custom AST lints, close to thirty of them, running in the same gate as every other check on every push. A good share exist purely to prevent a second implementation of something that already exists. You cannot construct an event engine outside the one package that owns it. A setting may only be declared in its module's settings file, a realtime channel only in its channels file. The call that publishes to the realtime provider is reachable only from inside the realtime module. The scheduler's tables, the scheduler's events, and the durable-timer primitive are fenced inside the modules that own them.

The newest one is the most on the nose. It fires if you declare an enum with a GLOBAL member anywhere outside the file that already defines the scope vocabulary, on the assumption that you are reinventing it. So far the assumption has been right every time.

None of these are clever. They are AST checks that take an afternoon to write. Their entire value is that they run whether or not anybody remembered the rule.

What the rule costs, and what it buys

The cost is not small, and it lands all at once.

That six-line loop becomes a declared scheduled event with a recurrence, a materialised occurrence row, a conditional claim so two workers never double-fire, and a durable subscription that has to be idempotent because delivery is at-least-once. Call it forty lines and a concept the next person has to learn. For "run this at midnight."

The first time you pay that, it feels absurd. It is absurd, if you only ever schedule one thing.

Everything after the first is a declaration against a scheduler that already knows how to be leader gated and crash safe. The loop does not amortize. It multiplies, and each copy multiplies its own bugs.

That is the whole economic argument, and it is why the rule has to be enforced at the moment of the first violation rather than the third. At the first violation the loop is genuinely cheaper. The rule asks somebody to pay now against a problem they cannot see yet, and no amount of documentation makes a person do that reliably.

Where it broke, part one: applying it too hard

The principle survived contact with reality in three interesting ways, each of them instructive for the wrong reason.

The first failure was over-application. Once you write "there is one scheduler," the natural next move is to route everything time-shaped through it. That is what happened. Then a periodic archival sweep was needed for a change feed the data layer emits, and the obvious answer was a scheduled event.

It was the wrong answer, and it took two occurrences before that was clear. The sweep is the event engine's own housekeeping. It has no domain meaning, no tenant, and no user-visible schedule anybody would ever want to inspect or change. Putting it in the scheduler would have made the scheduler a dependency of the very thing it exists to stay decoupled from. It lives in the worker instead, guarded by a distributed lease, and it is neither a timer loop nor a scheduled event.

The correction: one way per concern is not one thing for everything.

The journal has the same restraint built into it deliberately. It is the one user-facing historical record and it does no AI and no scheduling. A daily summary is just a record in it; producing that summary is somebody else's job. The moment the journal grows a summarizer it stops being a record and quietly becomes a second place where AI features live.

The event engine tests the same boundary from the other side. One engine does not mean one delivery guarantee. It carries three lanes: in-process local, durable over a FIFO queue, and a best-effort cross-worker lane added for a high-volume feed. Three lanes, one engine, one catalog, one subscription API. The alternative was a separate engine for change data capture, and that came closer to being built than it should have.

Where it broke, part two: the migration that never finished

This is the failure that has actually cost money, and it is the reason for the "never a second" wording.

A secondary generation step was split out of a request path so the main response would not wait on a second model round-trip. New endpoint, new method, docstring updated to say the old field now returns empty. Clients migrated. By every normal measure this was a completed piece of work.

Except the original path still awaited the inline generation. Nobody had deleted it.

So every migrated client paid for that work twice, and the response was still gated on the slower of two model calls. Investigating "why does onboarding feel slow" turned this up as one of four things in the request doing work nobody had asked for. Another was a full retrieval round-trip for a feature the tenant had switched off, because the flag defaulted to false and the only code that ever read it was a status endpoint.

The lesson: a refactor that lands the new path and leaves the old one is worse than not refactoring, because it costs you both designs and buys you neither.

Where it broke, part three: the rule was simply wrong

The last case is the one teams usually skip, and skipping it is how rules rot. A rule that can never be amended gets broken in silence, which is worse than having no rule.

The plan was to run the workflow engine on a managed Cassandra service, reusing a datastore already in the stack. Then it turned out that service forbids the conditional batch writes the engine's schema depends on. Not a tradeoff that had been underweighted. Just wrong.

What did not happen is quietly swapping in a relational database and leaving the document saying Cassandra. The entry recording the change is explicitly titled as a supersession, names the earlier decision, and explains why it was wrong. Then it does the part that matters more: it narrows the old rule instead of deleting it. Application data still lives on the original store and only there. The one sanctioned use of the new database is the workflow engine's own persistence. The rule survived with a documented exception rather than eroding into "well, we use both now."

Design documents get the same treatment. One plan designed the model gateway one way, a later plan designed it better, and the first is still in the repo with a SUPERSEDED banner at the top pointing at the second. Nobody has to guess which one won, and nobody has to wonder whether the old one was rejected or forgotten.

The hardest version arrived with a feature whose entire purpose is letting separate tenants read shared content, which inverts the isolation rule the whole backend rests on. The resolution was not an exemption. It used the cross-tenant mechanism a few other modules already use, and the guarantee that does still apply was test-locked: content written under one tenant has to be unreadable when that tenant is bound. An exemption would have been faster and would have left no way to tell, a year later, whether the isolation guarantee still held.

What we would do differently

Three things, in order of how much they would have saved.

Write the check the same day as the rule. There were months where the rules existed and nothing enforced them, and every piece of drift that later had to be cleaned up lived precisely in that gap. The lint does not need to be good. It needs to exist before the second person needs the rule.

Put a deletion step in the plan template from the beginning. Every half-migration traces back to a plan that carefully described what to build and said nothing about what to remove. A plan is not done when the new path works. It is done when the old call site is gone.

Be slower to expand a principle than to enforce it. The instinct after landing one scheduler was to make everything time-shaped go through it, and that instinct was wrong twice before anybody caught it. "One way per concern" is a much narrower claim than it sounds like on the day it gets written down.

None of this is free. Every time a lint blocks a six-line solution and hands somebody a forty-line one, the team is paying against a problem that may never arrive. Some of those payments are wasted. Some features never get a second instance.

But thirty-odd modules in, nobody has had to ask which of three schedulers a job is running on. Plenty of codebases cannot say that after a year, and it is rarely a discipline problem. The rule just never got written down early enough to be cheap, and by the time the second way appeared, so had the third.