Keyspaces looked like the safe choice. Cassandra was already running in the stack, so letting Temporal reuse it meant no new engine and everything staying in-account. Then we hit the wall: Temporal's Cassandra schema depends on conditional batch writes, the lightweight-transaction mechanism behind shard-ownership fencing, and Keyspaces doesn't support them. The obvious fallback was Postgres. But Postgres meant Aurora, and Aurora was blocked by two standing rules: application data lives on DynamoDB alone, and no stateful tier is allowed to run always-on. So the actual work wasn't picking a database. It was renegotiating the rules that had ruled Aurora out in the first place.
The wall
Temporal's Cassandra schema leans on conditional batch writes for shard-ownership fencing, the mechanism guaranteeing only one worker owns a shard at a time. Keyspaces doesn't support that pattern, which took Cassandra off the table entirely and pointed at Postgres as the only other schema Temporal ships support for.
Our rule was two rules
"Never Aurora" had always been said as one rule, but it was really two: a data-gravity rule (don't let application state split across two transactional stores) and a cost rule (nothing always-on, no billing runaway). Checked separately, only the data-gravity concern actually applied here, since Temporal's database is engine internals, not something the application ever queries directly.
The decision path
Cassandra already in the stack → chose Keyspaces (reuse, no new engine)
↓
Wall: Temporal's Cassandra schema needs conditional batch writes (LWT) for shard-fencing — Keyspaces doesn't support them
↓
Fallback: Postgres → requires Aurora
↓
Blocked: two standing rules — "Never Aurora" (data-gravity) and "Nothing always-on" (cost)
↓
Split the rule: data-gravity still applies · cost gets solved structurally, not by blocking
↓
Result: DynamoDB always for app data, plus exactly one sanctioned Aurora cluster for the Temporal engine only
Move 1 - narrow the rule instead of abandoning it
Rather than dropping "never Aurora," we scoped it: application data stays on DynamoDB, always, and there is now exactly one sanctioned Aurora cluster in the account, the Temporal engine database. That exact phrasing now lives in CLAUDE.md, task-engine.md, and a header comment at the top of the Terraform module, so the distinction doesn't get re-litigated later.
Move 2 - kill the cost objection structurally
The cost rule got satisfied structurally rather than argued away, using Aurora Serverless v2 with the scaling floor doing the actual work:
Serverless v2 never runs as a fixed-size node, so the "nothing always-on" stance survives untouched. The counter-intuitive detail worth calling out: enabling v2 scaling requires engine_mode = "provisioned", not "serverless" — the naming runs backwards from what you'd guess.
Move 3 - simplify the compute tier while we were in there
Since the persistence layer was already being rebuilt, the compute side got simplified too. An EC2 Auto Scaling Group became an ECS Fargate service running temporalio/auto-setup:1.25, with DB=postgres12 and POSTGRES_SEEDS wired from the cluster endpoint, sized to one or two tasks depending on the deployment profile. Fewer moving parts, same scale-to-floor property.
Move 4 - never hold the password
manage_master_user_password = true hands the credential straight to RDS and Secrets Manager. The ECS execution role gets GetSecretValue scoped to that one secret ARN, never a wildcard, and the container pulls the value through the task definition's secrets block with a JSON key selector:
{
name = "POSTGRES_PWD",
valueFrom = "${aws_rds_cluster.temporal.master_user_secret[0].secret_arn}:password::"
}
No plaintext password ever touches state, tfvars, or the task definition.
The final architecture
TEMPORAL ENGINE · the one sanctioned Aurora
ECS Fargate service temporalio/auto-setup:1.25
↓↑
Aurora Serverless v2 (Postgres) min/max ACU floor, scales to a floor
No domain code imports temporalio — only executors/temporal.py does.
The part that made the reversal safe
A few guardrails kept this from being a risky rewrite. Profiles are driven by tfvars-per-profile rather than count toggles, since flipping a count can destructively replace a stateful resource. The cluster carries both prevent_destroy and deletion_protection, with skip_final_snapshot tied to whether the profile is HA. CI plans both profiles over OIDC with no auto-apply, and tfsec was swapped for tflint, Trivy, and Checkov in the same pass, with every suppression carrying a written reason and a pointer to where it gets wired at prod deploy.
Why it cost ~zero application code
This is the part worth ending on: swapping the persistence engine underneath a workflow engine touched no domain code at all. Only executors/temporal.py is allowed to import temporalio, enforced by a hard lint rule, and the inline executor stays the default — so local dev and the test suite never needed Temporal running in the first place. The only local change was flipping the docker-compose Temporal profile from Cassandra to Postgres, for parity with prod. Everything validated clean: terraform validate on both profiles across dev and prod, fmt -check clean, and the full suite green at 159 passed, 89.20% coverage.
What we'd do differently
Validate the persistence engine against the workflow engine's actual schema requirements before committing to it, not after. That's now written down as the standing do/don't for the next migration.
One honesty note
The old entry in learnings.md wasn't edited to fit the new story; the file is append-only, so the newer entry says it supersedes the previous one and both stay readable. And the module isn't in prod yet. The CMK, enhanced-monitoring, and backup-vault items are marked as deliberate skips, not gaps nobody noticed. Better to say that plainly than to imply it already shipped.
