Four Lanes and a Hash Chain

We run a modular monolith. One deployable, a dozen or so modules inside it, and one hard rule between them: a module may import another module's interfaces (frozen DTOs and a Protocol) and nothing else. No reaching into another module's service, its models, or its tables.

That rule is easy to write down and hard to keep, because sooner or later a module needs something to happen elsewhere. A user registers and three other modules care. A device reading arrives and a notification should go out. If the registration code calls the notification service directly, the seam is gone. It went quietly, in a one-line import that looked reasonable in review.

So events are not a convenience here. They are the load-bearing thing that keeps the seams sealed: the mechanism by which a module can cause effects it knows nothing about. That framing sets the bar. A best-effort in-memory pub/sub is fine when events are a nice-to-have. When events are how your architecture holds together, losing one is losing a state transition.

This is a tour of what we built on DynamoDB to make that safe. And, because this is the part nobody writes up, it is also about the two ways it failed silently before we made it prove itself.

The envelope

Every event in the system is the same shape.

class EventEnvelope(BaseModel):
    model_config = ConfigDict(frozen=True)

    id: str
    type: str                      # "<module>.<name>", e.g. "user.registered"
    tenant_id: str
    occurred_at: str
    correlation_id: str | None
    actor_id: str | None
    payload: dict[str, Any]

    # populated by the store on append
    seq: int | None
    prev_hash: str | None
    hash: str | None

Two details carry more weight than they look like they do.

The type is namespaced by module. user.registered, not registered. You can read a subscription and know which module's contract you are coupled to, and a glob like user.* is a meaningful subscription rather than an accident.

Tenant, actor and correlation come from ambient context, never from the caller. emit() reads them out of contextvars bound at the edge of the request. A caller cannot pass a tenant_id, which means a caller cannot pass the wrong one. It is the same principle as the rest of our data layer: tenancy is established once, at authentication, and after that it is not an argument anyone can get wrong. It also means an event emitted during a webhook whose signature failed has no honest tenant to be stamped with. That is a real constraint we hit, and we will come back to it.

One write path

The hardest problem in event-driven systems is not delivery. It is the gap between changing state and announcing it. Save the entity, then publish, and crash in between: you have a user who exists and a world that never heard about it. Publish first and you get the inverse. Neither is recoverable after the fact, because nothing recorded that the pair was supposed to be atomic.

The classic answer is the transactional outbox: write the state change and the outbox row in one database transaction, and let a separate process drain the outbox. The usual assumption is that you have a relational database to do it in. We don't.

DynamoDB's TransactWriteItems turns out to be enough.

A caller hands its own writes to emit as extra_writes, and the store prepends two items of its own:

await default_engine().emit(
    "thing.created",
    {"thing_id": thing_id, "owner_id": owner_id},
    extra_writes=[{"Put": {"TableName": ..., "Item": ...}}],
)

Three kinds of item, one commit:

All of it commits or none of it does. There is no window. There is also no publisher: nothing in the application writes to a queue, because the commit is the publish and the table's stream does the rest.

Two things we learned making this pleasant to use.

Name the failures in the caller's vocabulary. DynamoDB reports transaction cancellations by item index. Because the store prepends two items, the caller's first write is item 2, and an error saying "item 3 failed a condition" sends whoever is reading it hunting through the event store's internals for a bug that is in their own guard. The store re-labels every reason before raising: extra_writes[1] ConditionalCheckFailed. Small change. It is the difference between a five-minute diagnosis and an hour.

Not every cancellation is a retry. Losing the race for the chain head is normal and expected (another request in the same tenant committed first), so the append re-reads the head and tries again within a bounded budget. But a caller's refused condition is an answer, not a delay. "This user already liked this post" will be refused identically five more times. Our first version retried everything, which spent the whole budget and then reported contention on the chain: blaming the infrastructure for a business rule working correctly. The predicate that tells the two apart is worth writing carefully. A conflict, or a failed condition on the head row specifically, is transient. Anything else is a real answer, and is raised immediately with nothing written.

The chain

Every event carries hash = sha256(prev_hash + canonical(envelope)), chained per tenant, with the head row tracking last_seq and last_hash.

The chain buys us two things. The obvious one is tamper evidence. The log is insert-only in production at the IAM level (the role has no UpdateItem or DeleteItem on that table), and the chain means that even if that were bypassed, a rewritten event is detectable. A reconciliation endpoint recomputes the chain on demand and reports whether it is intact. For a system carrying health data, "we can prove this record was not altered" is worth the sha256.

The less obvious one is that the chain forces the ordering discipline to be real. The head update is conditional on last_seq, so two concurrent appends in the same tenant cannot both win. The log has no gaps and no forks, and seq is a genuine total order per tenant.

Which sets up the caveat that matters, the one we wrote into the module docstring so nobody has to rediscover it:

The log is strictly ordered. Delivery is not. DynamoDB Streams preserve order only per partition key, not across transactions. Two events committed in sequence can reach a consumer in either order.

So downstream consumers must be idempotent however ordered the log looks. We get per-tenant ordering back on the queue by using the tenant as the FIFO message group, but the general rule stands: order the log for auditing and reconstruction, and never let a handler's correctness depend on having seen its predecessor.

Four lanes

Not every event wants the same delivery. A handler that must run before the caller returns is a different animal from one that can be a few seconds late, which is different again from a high-volume change feed where losing one is fine and blocking a write is not.

Rather than one delivery mode with flags, there are four lanes.

• INLINE: awaited inside the caller's flow. Exceptions propagate. Fail-closed, for the critical path.
• LOCAL: scheduled on the event loop, fire-and-forget, isolated. A failure is logged and never reaches the caller.
• DURABLE: delivered out-of-process, at-least-once, idempotent, dead-lettered on failure.
• BEST_EFFORT: also out-of-process and on any worker, but unordered, dup-tolerant, with no dead-letter and no back-pressure. The substrate for the change feed.

The diagram's real content is that left column. Each lane has exactly one dispatcher. emit() dispatches INLINE and LOCAL and nothing else. The durable consumer dispatches DURABLE and nothing else. The change-feed consumer dispatches BEST_EFFORT and nothing else.

That makes the lane a reachability decision rather than a preference. A handler declared on the wrong lane is not slow or unreliable. It is unreachable. It binds without error, it shows up in the generated catalog, it dead-letters nothing, and it never runs. Hold that thought.

The durable lane, end to end

Committed row → table stream → EventBridge Pipe → SQS FIFO → a consumer on Fargate that calls one function: deliver_durable(envelope).

That function binds the tenant from the envelope, so the handler's own writes land in the right tenant without anyone passing it around. It then selects the matching DURABLE subscriptions, and for each one claims an idempotency key of {event_id}:{subscription} before running the handler. The claim is a row with a status: pending → completed, or dead if the handler raised.

Per subscription, not per event. At-least-once delivery means a redrive can re-present an event whose first three handlers already succeeded and whose fourth failed; keying on the pair means the replay runs only the fourth. A handler that raises is dead-lettered (its status marked dead, plus a tenant-scoped row an operator can query) and its siblings still run. One broken subscriber does not take the others down with it.

The reconciliation endpoint returns both halves of the health picture:

{ "chain_intact": true, "dead_letter_count": 0 }

Locally and in tests, the consumer function is called directly. Same code path, no queue, no emulator. The lane is exercised in CI without any of the infrastructure that carries it in production.

The wire shape, and a 93% loss

The pipe's input template makes exactly one substitution: the DynamoDB stream image, verbatim. The consumer un-types it back into an envelope on the other side. This looks lazy and is not. Naming envelope fields individually in the template cannot work, for two independent reasons. The payload is a DynamoDB map whose keys differ per event type, and a template substitutes JSON paths; it cannot reshape a typed map. And the store writes with exclude_none=True, so optional fields are simply absent from real rows, leaving a template that names them with unresolvable paths.

Both failures are silent. A pipe that cannot resolve a path does not error into your logs; it just stops delivering. The queue stays empty, every durable subscription stops running, and the test suite is green.

We learned the same lesson a second, more expensive way. The head row is written in the same transaction as the event, so it hits the stream just as often. The pipe reads two target parameters off each record, dedupe id and message group, and on a head row those paths resolved to nothing, so SQS rejected the send with a 400. A rejected send fails the whole Pipes execution, discarding the genuine events batched alongside it. Roughly 93% of all durable messages were being dropped, and nothing anywhere said so.

The fix was to filter head rows out at the pipe and, belt and braces, to stamp the head row with the two attributes, so the lane is correct even where that filter hasn't been applied. With one subtlety worth the comment it got in the code: the head row's id is derived from the event id, never equal to it. The two rows reach the queue together, and an identical dedupe id would make SQS drop one of the pair. Arbitrarily. Sometimes the real event.

The catalog, and why we gate it in both directions

Every module declares its event surface in one file: EMITS (what it produces) and SUBSCRIPTIONS (what it reacts to, on which lane, via which handler method). The registration code binds handlers from the same specs, so the declaration and the live wiring cannot drift. A generated catalog records the whole surface across modules, and a drift check fails the build if it is stale.

The point of declaring choreography up front is reviewability. Ad-hoc pub/sub grows event storms and circular subscriptions that nobody can see, because the wiring only exists at runtime. A catalog makes the whole graph a file you can read in review.

But a declaration only helps if something proves it matches the code, and for a long time only half of ours was checked. We verified that every subscription wired into the engine was declared. We never checked the other direction, and it had drifted three ways at once: two modules emitted ten types declared nowhere (and therefore absent from the generated catalog, while three live subscriptions pointed at them), two declared emitters had no producer at all, and one subscription named a type nothing could ever emit.

It is the same lesson as the rule register in Every Lint Rule Is a Scar, arriving from a different side: a declaration is worth exactly as much as the check that proves it still matches the code.

Four static checks now close it:

1. Every emitted type is declared. An emit(CONST, ...) whose type is in no EMITS fails the build.
2. Every declared emitter has a producer. A catalog entry nothing emits fails, unless it is on a short, reasoned allowlist that is also tracked in the backlog.
3. Every type_glob can match something: the declared emitters, plus the change-feed namespace derived from the live repository entities.
4. Every change-feed subscriber is on BEST_EFFORT, and only those are.

The last two exist because of one bug, and it is the failure mode worth internalising. Conditions match with fnmatch, so a glob with no wildcard is an exact string compare. One subscription had two defects at once: a typo in the entity name, and the LOCAL lane where it needed BEST_EFFORT. It bound without complaint. It appeared in the catalog. It dead-lettered nothing. It simply never ran, so a user who corrected their email address kept getting notifications at the old one. And because it was the change feed's only declared consumer, the entire feed was delivering to nobody while looking fully built.

That is the detached probe from A Taxonomy of Silence in its purest form: a pattern that matches nothing, and nothing is not a violation.

There is a related one we fixed in the engine rather than the gate. An event that reaches a worker and matches nothing used to be indistinguishable from an event delivered to five handlers: the consumer deletes the message either way, and silence looks the same. It now logs a warning naming the type, and, for the fan-in types where the event type alone doesn't identify the job, the payload discriminator too. A related gap in the boot log, which named the modules that registered but never their subscriptions, cost five days of chasing a missing feature through the pipe, the filter, the stream and the condition matcher before anyone could ask the only question that mattered: is the subscription there at all?

One declaration we deleted rather than allowlisted: a webhook_rejected event. Every envelope is stamped with the tenant from context, and a webhook whose signature just failed offers only an attacker-controlled tenant. There is no honest tenant to stamp, so the rejection belongs in the structured log, not the tenant's event log. When the architecture forbids a declaration, the fix is to remove it, not to carve out an exception.

The change feed, next door

One more subsystem shares the engine without sharing the log. Every write to a tracked table emits a record.written.<ENTITY> / record.deleted.<ENTITY> change event, so modules can react to data changes without the producer knowing its consumers. An address change cascades into notifications, commerce and location, and the module that owns addresses imports none of them.

It is deliberately built to a different standard. It is emitted post-commit and fire-and-forget, so it never blocks a write. It goes to its own TTL'd table, not the hash-chained log, because a change feed at that volume would swamp an audit log. It is delivered on BEST_EFFORT, because a lost or duplicated change notification is acceptable and back-pressure is not. Older rows are archived to object storage by a leader-gated worker.

Same engine, same envelope, same subscription model. Different guarantees, chosen on purpose. That is the argument for four lanes instead of one delivery mode with a retry setting.

What we'd tell you to steal

• Make the commit the publish. If your database can write two items atomically, you can have a transactional outbox. Passing the caller's writes into emit, rather than emitting after a save, removes the window entirely, and removes it for every call site at once.
• Read tenancy from context, never from arguments. A value a caller can't pass is a value a caller can't get wrong.
• Order the log, but never let a handler depend on it. Streams give you per-partition order at best. Idempotency is not optional; it is the price of the lane.
• Treat a delivery mode as a reachability property. If each lane has exactly one dispatcher, a mis-laned subscription is unreachable rather than degraded. That is much worse than slow, and much easier to gate against.
• Gate the declaration in both directions. The whole value of a declarative catalog is that it matches reality, and nothing at runtime will tell you when it stops. Check that everything emitted is declared, everything declared is emitted, and every pattern can match something.
• Be suspicious of silence. Almost every expensive bug in this subsystem presented identically: no error, no dead letter, green tests, and a feature that quietly did not happen. A glob that matches nothing, a lane with no dispatcher, a pipe template that can't resolve a path, a module whose registration never ran. The common fix was not better error handling, because the errors did not exist. It was making the absence of work observable: log the event that matched nothing, name the subscriptions at boot, fail the build on a pattern that can't match.

The events were always the thing holding the seams shut. What we had to learn was that an event system's most important output is the one it doesn't produce, and that the thing you most need to see is the thing that didn't happen.