Everything That Produces Output in Time Is a Sensor

Or: what happened when we made a physical transducer and a call to a language model the same kind of object.

The fourth table

At one point last year we had three unrelated pieces of code doing the same thing badly.

A fleet of devices streamed telemetry into one store, and the read path was “fetch a page of rows, then filter in application code.” A model-generated artifact was produced on a schedule, one per subject, and kept in its own table with its own shape. A set of derived metrics accrued over time in a third place, with a third query pattern.

Each was defensible on its own. Together they were a pattern, and the pattern was the problem: there was no shared contract, no uniform retention or lineage model, and — most importantly — nothing at all stopping the next data-producing feature from inventing a fourth table, a fourth write path, and a fourth read query that doesn’t scale.

You cannot fix that with a code review. Code review catches the fourth table if the reviewer remembers the first three. What we wanted was for the fourth table to be impossible to write, and for the correct alternative to be more convenient than the wrong one.

So we wrote down one sentence and then spent a plan cycle taking it completely literally:

A sensor is any encapsulated unit with a declared output schema that produces stamped outputs in time. Everything that produces an output in time is a sensor.

Taking it literally

Here is the test we use on new work. Which of these is a sensor?

1. A physical transducer reporting a measured quantity continuously.
2. A third-party API we poll on a fixed interval.
3. A rolling window derived from another sensor’s history.
4. A language model that writes a paragraph of guidance per subject, per period.

The useful answer is all four, and the fourth is the one that does the work. If a model’s output is a sensor reading, then a whole set of questions nobody asks about an AI feature suddenly have to be answered, because the envelope demands them:

• What time does this reading pertain to? (Not “when did we write it.”)
• What produced it? Which model version, which prompt lineage.
• How confident is it, and is it degraded?
• What happens when it’s wrong? Is it mutable, or is a correction a new record?
• How long do we keep it, and can we delete it for one user?
• Who downstream consumes it, and what happens to them when it changes?

Those are ordinary time-series questions. They are not ordinary “we added an AI feature” questions, which is exactly why the framing was worth adopting. Modelling the inference as a reading imports a discipline the AI feature would otherwise have had to invent, badly, on its own.

Every sensor — transducer or model — ships one declaration:

manifest(
    code           = <stable sensor code>,
    scope          = SUBJECT | GLOBAL,
    archetype      = HARDWARE | POLLED | DERIVED | MODEL,
    output_shape   = SCALAR | TUPLE | STRUCTURED | BLOB_REF,
    output_schema  = <declared schema>,
    sampling       = <declared peak rate>,
    event_policy   = <when a reading also raises an event>,
    ingest_profile = STREAMING | DURABLE,
    retention      = <tier + horizon>,
    sizing         = <estimated rate, estimated row size>,
    partitioning   = <bucket granularity, shard count>,
    code_version   = <function version>,
)

A model inference and a strain gauge fill in different values for archetype, output_shape and sizing. They do not differ in kind, and they do not differ in vocabulary.

The envelope, field by field

The paradigm is only as good as the envelope every reading shares, and every field in ours exists because of a specific failure it prevents.

event_time distinct from ingestion_time. This is the field pair people collapse into one created_at and regret. A device that loses connectivity and then flushes its buffer produces readings whose event times are in the past. If you have one timestamp, you have silently decided that late data is new data — your ordering is wrong, your replay is wrong, and your audit trail can’t distinguish “the source was slow” from “the value changed now.”

version and immutability. Readings are never mutated. A correction is a new row at a higher version superseding the same (sensor_id, event_time), and reads resolve to the highest version seen. That is what makes the store append-only and the write path idempotent — a retry, a replay and a correction become three shapes of the same safe operation rather than three different bugs.

seq. The clustering tiebreaker within one event time, and quietly the most important replay-safety primitive we have. A sensor that passes a deterministic id as its seq can re-run its own generation for a period and upsert the same row instead of appending a duplicate. A repository-backed feature has to construct that property deliberately; a sensor gets it from the primary key.

quality and confidence. A coarse OK / DEGRADED / SUSPECT flag travels with every reading so consumers can refuse degraded data rather than averaging it in. Our model sensor stamps DEGRADED when the inference comes back without a usable result. The reading is still recorded — we want the evidence — but nothing downstream is obliged to treat it as equal.

provenance. The inputs, as (sensor_id, event_time, seq) references, plus the code version and the model version. For a model sensor this is the difference between “the model said something odd last quarter” and “the model said something odd last quarter, on this prompt lineage, on that model version, from these three inputs.”

Gaps are typed, not absent. Missing data is its own record:

gap_reason: MISSING | LATE_DROPPED | SOURCE_DOWN

A polled source that times out emits an explicit SOURCE_DOWN gap. It does not quietly hold the last value forward, because “the reading has been unchanged for hours” and “we have not heard from the source for hours” are opposite facts and must not render identically.

Record and raise are two verbs

The most load-bearing rule in the design is boring to state: recording a reading is always separate from raising an event about it. Every reading is recorded. Whether it also raises an event is a declared, per-sensor policy.

event_policy: EVERY_READING | ON_CHANGE | ON_THRESHOLD | ON_WINDOW_CLOSE | ON_EMIT

Conflating the two is how time-series work destroys an event bus. A high-rate sensor that emits a domain event per sample is a denial-of-service attack on your own subscribers, and it will be discovered in production by the on-call engineer rather than at design time. So the split is enforced rather than recommended: a sensor whose declared rate is too high for the event policy it picked fails registration, at bootstrap, in CI, with a message saying what to pick instead.

Raising also never breaks recording. The emit path swallows its own failures deliberately:

async def raise_event(manifest, reading, event_type=READING_EVENT):
    """Raise a sensor event on the durable lane.

    A raise failure must never break recording.
    """
    try:
        await bus.emit(event_type, payload(manifest, reading), lane=DURABLE)
    except Exception:
        log.exception("sensor_event_emit_failed", code=manifest.code)

Data capture is the thing you cannot get back. Notification is retryable. Treating them as one operation means the less important half can lose you the more important half.

The manifest is an admission test

A declaration is only worth writing if something checks it. Registration validates every manifest and refuses the ones that would hurt us later, which turns a set of scaling conventions into a design review that runs on every boot.

Write-budget sizing. A single partition in the store tolerates a documented number of write units per second, and we budget a fraction of it to keep headroom. So a sensor must declare its honest peak rate and row size, and the required shard count falls out of arithmetic rather than opinion:

def required_shards(peak_rate, row_bytes):
    """One row costs ceil(row_bytes / WRITE_UNIT) units."""
    units_per_row = max(1, ceil(row_bytes / WRITE_UNIT_BYTES))
    peak_units    = peak_rate * units_per_row
    return max(1, ceil(peak_units / PARTITION_WRITE_BUDGET))

A slow scalar sensor needs one shard. A sensor at the top of the workload band we designed for spreads across many. Declare a shard count below what your own numbers require and registration raises.

Money safety. If a sensor’s output can affect a charge, it may not shrug at missing inputs:

if manifest.affects_commerce and manifest.missing_data_policy in (None, EMIT_MARKER):
    raise ContractError(
        f"sensor {manifest.code!r}: a money-affecting sensor must declare a "
        f"fail-closed or hold-last-valid missing-data policy"
    )

FAIL_CLOSED, or HOLD_LAST_VALID with a maximum staleness. Emitting a gap marker and carrying on is forbidden on a money path — a billing pipeline that treats “I don’t know” as a value is how you refund a quarter.

Acyclicity. Sensors declare dependencies, and derived sensors trigger recompute cascades, so a cycle is not a slow query — it is an infinite loop with a storage bill. The registry runs a depth-first cycle check on every registration and refuses to hold a graph with a cycle in it.

Four guards, all at registration: partition budget, event policy versus rate, missing-data policy for money, and acyclicity. None of them can be forgotten by a reviewer, because none of them are a reviewer’s job.

Shape tables, not per-sensor tables

The instinct when a new data producer arrives is to give it a table. We give it no table. There is a small fixed set of tables organised by the shape of the value, and every heterogeneous sensor in the system shares them — scalar, tuple, structured, and a reference row for anything large enough to live in object storage. Alongside them sit rollup tables at a few granularities, so dashboards read downsampled aggregates and the raw partitions absorb writes only.

Every one of those tables carries the same key:

partition key : (sensor_id, time_bucket, shard)
clustering    : (event_time asc, seq asc)

Three properties fall out of that key, and they are the reason the constraint is worth it.

Tenant isolation lives in the partition key. Runtime sensor ids are tenant-leading and scope-qualified — {tenant}:{scope}:{code}. There is no way to address a partition without naming the tenant, and the id is derived from the request’s bound tenant at the storage floor, never from a caller-supplied parameter. The registry keys manifests by the trailing code alone, so every subject and every device shares one declaration while getting its own series.

One place computes the partition, so the reader and writer cannot disagree. This is the bug the design is most proud of preventing. A sharded write path and a scatter-gather read path that each derive the shard independently will drift the moment someone changes the bucket granularity, and the symptom is silent partial reads. So both call the same registry method:

def partition_key(sensor_id, event_time):
    m      = registry.manifest_for(sensor_id)
    bucket = epoch(event_time) // m.bucket_granularity
    shard  = stable_hash(sensor_id) % m.shard_count
    return (sensor_id, bucket, shard)

The shard hash is a stable cryptographic digest rather than the language’s built-in hash, which is salted per process and would give you a different shard on every worker — a genuinely nasty way to lose data across a fleet.

A read is a bounded fan-out with a merge rule. Because writes spread across shards by hash, a range read must query every shard for every bucket the interval touches, then merge by event time, keeping the highest version per event time. That merge is where reads-resolve-to-latest-version actually happens:

def merge_latest(rows, limit):
    best = {}
    for row in rows:
        current = best.get(row.event_time)
        if current is None or row.version > current.version:
            best[row.event_time] = row
    return sorted(best.values(), key=lambda r: r.event_time)[:limit]

All datastore access lives in exactly one file — a generic, datastore-pure floor that knows about rows and partitions and nothing else. It has no idea what a manifest or an archetype is. A thin adapter in the sensor module maps the typed envelope onto it. That separation is what would let us move the store without touching a single sensor.

Two execution tiers, and no new scheduler

Sensors run in one of two places, chosen by a declared profile at a rate boundary:

• Durable — slow and schedulable — rides our existing calendar, the one long-term scheduler in the system. A sensor declares a calendar layer and its scheduled event specs, and subscribes to the calendar’s fired event on the durable lane.
• Streaming — faster, continuous, event-driven — runs on a long-lived worker loop.

The interesting part is the thing we refused to build. The obvious way to run a sampling loop is a sampling loop: an in-process timer per sensor. We had a hard rule against growing a second scheduler, so the durable tier reuses the calendar instead, which means sensor sampling inherits leader election, at-most-once firing across the fleet, and durable timers for free. Every architectural rule you keep is a category of infrastructure you don’t operate.

Two consequences worth knowing if you copy this. A durable calendar handler binds a tenant but no principal, so any principal-scoped read inside it returns nothing; our per-subject fan-out is therefore scheduled per owner, with the handler resolving that owner’s subjects in a system context. Discovering this by watching a scheduled job produce zero readings for zero reason is a memorable afternoon. And system-wide sensors ride a global tenant partition with the scheduler rebinding the tenant before firing — the pending→fired conditional claim already makes firing at-most-once fleet-wide, so no extra lock is needed.

Corrections cascade, but only so far

When a correction lands, the sensors that depend on the corrected input have to re-derive their affected window. That fan-out is bounded two ways, because unbounded recompute is how a backfill becomes an incident: a per-sensor recompute horizon, beyond which corrections are still recorded but trigger no dependent work; and a cascade depth guard, belt-and-braces over the registry’s acyclicity guarantee.

The cascade runs on the durable event lane, where at-least-once delivery plus per-subscription idempotency gives effectively-once execution, and recompute emits new versioned readings — it never mutates an existing one. A derived sensor that wants to participate implements one method:

class RecomputableSensor(Protocol):
    async def recompute(self, from_ts, to_ts) -> tuple[Reading, ...]: ...

Reproducibility is the whole point: given the same inputs and the same code version, the same window re-derives to the same readings.

A paradigm without enforcement is a preference

We have been doing this long enough to know that an architectural principle documented in a markdown file has a short half-life. So the paradigm is backed by lints that run in the same gate as everything else.

No raw datastore sessions outside the floor. Open one anywhere else and the check fires. There is one writer, and it is checked by a machine, not remembered by a person.

Table ownership. The sensor tables may only be named inside the sensor module.

A shrinking migration backlog. This is the one I’d steal if I were reading someone else’s post. We keep an explicit list of modules that still produce time-series outside the contract, and the lint flags them by name. The paradigm shipped with its own to-do list, machine-checked, and it only ever gets shorter. “Everything is a sensor” would otherwise mean “everything new is a sensor and the old stuff is whatever it was” — which is the same fragmentation with better documentation.

Plus the registration-time guards above. The rule and its enforcement were written in the same change, which is the only way we have found to make an architectural claim still true a year later.

What it cost

It would be dishonest to present this as free. Four things hurt, and three of them we chose.

A second storage floor. We already had exactly one sanctioned write path for transactional data. Sensors introduced a second one for time-series, with its own operational surface, its own provisioning and its own failure modes. We accepted that in exchange for one uniform substrate for everything time-shaped. Two floors are a real cost; four ad-hoc tables and counting was a bigger one.

Weaker isolation than the transactional store. Our transactional path enforces tenancy in several layers, including infrastructure-level constraints on leading keys. The time-series store has no equivalent, so isolation there is by construction — the tenant-leading partition key, derived only from the bound tenant at the floor — plus namespace-level access control. That is an honest reduction in defence-in-depth, written down in the decision record as a consequence rather than buried.

Retention is not erasure. We shipped the store with a time-to-live and considered privacy handled. It isn’t: a per-user delete request for data that lives only in the time-series store needs a real hard delete, so the floor grew a partition-scoped delete path built on the same partition enumeration the reader uses. A module whose data is time-series-only cannot honestly claim erasability without it.

And one genuinely misleading failure. The table names are fixed but the namespace carries an environment prefix, so every deployed environment must set it explicitly. Miss it and the client connects, discovers every host, fails to bind the namespace on each one, marks the entire cluster down, and surfaces as a connectivity error — a stack of tracebacks pointing at your network instead of your config. The lesson generalises past our stack: when a client fails per-host, a single misconfiguration is indistinguishable from an outage unless you have written the mapping down somewhere a human will read during an incident.

There is also a second-order effect worth naming, because it cuts both ways. Once the model’s output is a sensor reading, the time-series store becomes a hard dependency of that endpoint rather than a soft one that degrades. That is correct — the inference is the data — but it does mean a storage misconfiguration takes down an AI feature that used to fail softer. Unifying substrates unifies blast radius.

When something is not a sensor

The framing is expansive, not universal, and a paradigm that swallows everything explains nothing. A name is not a sensor reading. An order is not a sensor reading. A user’s role in a group is not a sensor reading. Those are referential and transactional state — mutable, authoritative, read by key — and they belong in the transactional store behind the repository floor with its own provenance and audit rules.

The line is the phrase in time. If the answer to “what is this?” is a value that pertains to an instant, and yesterday’s version stays interesting after today’s arrives, it is a sensor. If the answer is a fact about the world right now that gets updated, it isn’t.

Model inference sits on the interesting side of that line, and noticing so is the whole idea. A model call feels like a request and a response — you ask, it answers, you store the answer. But the answer pertains to a moment, is superseded rather than corrected, has provenance, has confidence, gets consumed by things downstream, and needs retention and erasure. That is a reading. Treating it like one meant our AI feature got event-time semantics, idempotent replay, versioned corrections, lineage and privacy erasure without a single line of AI-specific infrastructure — it inherited them from the paradigm, the same way a load cell does.

The test we hold the docs to

One line in our authoring guide sets the bar, and it is the standard I would suggest for any paradigm you try to establish:

An engineer who has never seen the system should be able to build a correct, well-behaved sensor of any archetype from this guide alone.

That is the real deliverable. Not the store, not the shard math, not the archetype taxonomy. The deliverable is that “is this a sensor?” is now a question with an answer, and the answer arrives carrying a checklist: declare your rate and row size honestly, pick an event policy your subscribers can survive, keep event time separate from ingestion time, make your writes idempotent, say what happens when an input is missing, and record where your value came from.

Any one of those is something a careful engineer might have done anyway. The point of a paradigm is that they no longer have to be careful — and neither does the next person, on the next feature, at the end of a long week.