LittleFS Tail-Latency for Burst Writes: Commit + Prealloc
When you first validate a bursty image pipeline on LittleFS, it usually looks great: average write times stay low, and short tests feel done. Then you run a 6-to-12-hour soak, flash creeps toward full, and suddenly P95/P99 write latency starts spiking. Frames arrive in bursts, your producer thread blocks longer than expected, and the rest of the system starts paying for it, sensor scheduling jitter, CAN/USB backpressure, dropped frames, even watchdog risk.
This post is a practical firmware playbook for reducing that long-tail latency without replacing your storage pipeline. The key is to stop treating "write time" as one number and instead make the system explain itself: how much time goes into pushing bytes versus updating metadata, and when garbage collection or compaction quietly steals your latency budget as the device ages and free space shrinks.
From there, two small patterns change stability dramatically. A commit marker makes each image set "publishable" only when it's fully persisted and remains safe under power loss, and best-effort preallocation reduces allocation churn so burst writes don't repeatedly trigger worst-case GC behavior. The result is a storage path that stays predictably fast even when flash is hot, fragmented, and nearly full.
The problem
LittleFS is designed for embedded flash realities, wear leveling, power-loss resilience, log-structured behavior. That design creates a common tail-latency failure mode for burst workloads: bursty producers create many short-lived files or frequently new extents; as the filesystem fills, allocation gets harder and internal cleaning or compaction happens more often; metadata operations like directory updates, rename, close, and sync can suddenly take much longer than the data write itself. The result: average throughput looks fine, but the tail is unpredictable.
In image pipelines combining camera and thermal frames with metadata sidecars, the real performance truth is that your system doesn't fail on averages, it fails when the worst 1-5% of writes stall the whole pipeline.
Approach
An incremental approach that's safe, measurable, and compatible with existing pipelines. First, measure what matters and separate it: data write time (payload bytes), metadata time (open/close, directory operations, rename), sync time (if you explicitly fsync), and a GC/compaction signal inferred from long metadata time or block-device counters.
Second, use a commit marker pattern for set correctness, writing image sets so downstream readers never see partial sets, especially after power loss, either via rename-based commit (.tmp to final) or footer-based commit (writing a verified footer last).
Third, use best-effort preallocation to reduce churn under pressure: keeping a small file pool of reused slots, or reserving a write budget for space discipline, or using pre-created empty containers you fill and commit. None of this requires touching LittleFS internals, it's firmware-level engineering around tail behavior.
Step 1: add instrumentation that isolates tail spikes
If you only time the whole "write set" call, you'll miss the real source of stalls. Segment your write path into phases that map to filesystem work. For each image set, measure t_open_us, t_write_us (sum of payload writes), t_close_us, t_commit_us (rename or footer finalization), and t_total_us. Also record fs_used_pct (estimated), set_size_bytes, and the current "mode" (empty FS versus partially full versus near full).
Two metrics keep you honest: P99 set commit latency in milliseconds, and sustained sets per minute after 6 hours. That's enough to validate stability without turning your post into a dashboard.
A minimal-overhead instrumentation sketch:
typedef struct {
uint32_t open_us, write_us, close_us, commit_us, total_us;
uint32_t bytes;
uint16_t fs_used_pct;
} write_trace_t;
static inline uint32_t now_us(void);
#define TRACE_START(var) uint32_t var##_t0 = now_us()
#define TRACE_END(var, out) do { (out) += (now_us() - var##_t0); } while(0)Used like this:
write_trace_t tr = {0};
TRACE_START(total);
TRACE_START(open);
lfs_file_open(&lfs, &f, path_tmp, LFS_O_WRONLY | LFS_O_CREAT | LFS_O_TRUNC);
TRACE_END(open, tr.open_us);
TRACE_START(write);
lfs_file_write(&lfs, &f, buf, len);
TRACE_END(write, tr.write_us);
TRACE_START(close);
lfs_file_close(&lfs, &f);
TRACE_END(close, tr.close_us);
// commit marker step (rename or footer verify)
TRACE_START(commit);
commit_file(path_tmp, path_final);
TRACE_END(commit, tr.commit_us);
TRACE_END(total, tr.total_us);For estimating fill level: if you already track blocks used at the block-device layer, use that; otherwise approximate from lfs_fs_size() (where available) and your configured block count, or maintain a coarse allocator watermark in your own storage service. You don't need perfect accuracy, just enough to correlate spikes with "near full."
Step 2: use a commit marker so readers never see partial sets
Even after optimizing latency, you still need correctness, readers should only consume complete image sets. Tail latency and correctness are tied together, if your pipeline retries or the device reboots mid-burst, you don't want partially written files interpreted as real data.
Rename-based commit writes everything to a temporary name, then renames to the final name once complete: sets/set_1234.tmp/ commits to sets/set_1234/ (or a set_1234.done marker). Renames are typically metadata operations LittleFS handles in a power-loss-safe way, and the key benefit is semantic: downstream readers only scan final names. Write image payload files to a temp directory (set_1234.tmp/cam.bin, therm.bin, meta.json), then after all closes succeed, rename the directory to the final name, optionally creating a COMMIT file inside for easy scanning.
Reader rule: ignore .tmp directories entirely, only process final set_* names, and if a set_* directory is missing expected files, treat it as corrupted and quarantine it.
int commit_dir(const char* tmp, const char* final) {
// Ensure all files are closed before this point.
// Rename is your atomic-ish "publish" step.
int rc = lfs_rename(&lfs, tmp, final);
return rc;
}If rename or directory operations are your tail spikes, footer-based commit avoids rename by making the file self-validating instead: append a fixed footer at the end (magic, version, length, crc32), and only consider the set valid if the footer exists and verifies. This works well for a single "container file" per set, camera plus thermal plus metadata plus footer, with the footer written last, if power is lost mid-write, verification simply fails and the reader skips it. A footer layout example: magic = 0x53455421 ("SET!"), payload_len, crc32(payload). Reader rule: scan the file, read the footer, verify the CRC, then consume, and skip on any invalid footer.
In image-heavy products like EverBowl, commit semantics are what let downstream transfer and analytics stay simple: readers never need partial-write heuristics. The pipeline can treat storage as a sequence of published sets, even across reboots.
Step 3: best-effort preallocation to reduce allocation churn
Now we attack the main source of tail spikes in long runs, allocation and compaction pressure. You don't need perfect preallocation, just enough to avoid panic allocation under near-full conditions.
Strategy 1, a reusable file pool, is the most practical: instead of creating brand-new filenames forever, create a bounded pool and reuse slots, say slot_0000 through slot_0255, with a small index file mapping sequence_id to slot_id, and each slot overwritten in a controlled way like a ring buffer. Why it helps: fewer directory entries changing over time, less allocation churn, predictable metadata patterns, and smoother GC pressure. Implementation idea: pre-create the slot files once, at format time or first boot, then on each set pick the next slot, write there, and commit with a footer or rename within the slot namespace.
Strategy 2, reserved space discipline, is simple and effective: tail spikes get brutal when you're writing into the last few percent of free space, so maintain a reserve (say 5-10% free), and if you drop below it switch to degraded mode: drop optional frames, reduce burst depth, prioritize commit markers, or pause writes until upload/transfer frees space. This isn't giving up, it's enforcing predictability. Stable systems protect their future self.
Strategy 3, pre-created directories for burst batches: if your burst writes create many directories, pre-create a limited number of batch directories (batch_00 through batch_15) and write sets into the current batch until it's sealed, then rotate. This reduces directory-create bursts at the worst possible times.
An important implementation detail: "best-effort" means you never block forever. Preallocation is meant to smooth tail latency, so it can't become a new source of stalls. It's an optimization path, not a dependency. Attempt to allocate or pick a slot quickly using a strict time budget or a small bounded retry count, and if that fails, immediately fall back to the normal write path, accepting a latency hit in that case rather than deadlocking or starving the producer. Record a metric whenever the fast path misses, things like prealloc_hit_rate, prealloc_fallback_count, and a fallback_reason. That turns "it felt slower today" into an actionable signal telling you whether the pool is undersized, cleanup is lagging, or you're routinely operating too close to full.
In production pipelines where image sets get uploaded off-device for ML validation or post-processing, preallocation makes on-device behavior boringly consistent during long soak runs, exactly what you want when devices are deployed in homes and you can't babysit flash state.
Results: validating improvements without vanity benchmarks
This is where teams often get stuck: run a 2-minute test, declare victory, ship. Tail latency needs a different validation style. A proper soak test setup matches your real burst pattern: same set size distribution (camera, thermal, metadata), same burst cadence (say 3 FPS bursts, then idle), same downstream interactions if your reader scans storage. Test at three fill levels: freshly formatted, mid-fill (50-70%), and near full (85-95%).
What you should expect after these changes: commit correctness becomes deterministic, readers never see partial sets. P99 commit latency becomes much tighter, especially near full. Overall throughput may stay similar, but the system becomes stable under stress. A clean target worth tracking: improving P99 set commit latency by 3-5x under near-full conditions, a typical outcome when allocation churn is the real culprit. Exact numbers depend on block size, wear level, and burst size, but multi-x tail improvement is common once you stop fighting the allocator at the worst possible moment.
Common issues and how to avoid them
If rename is expensive on your device, use footer commit with container files, or reduce rename frequency by committing per batch instead of per file. If your reader still sees weird artifacts, enforce strict reader rules, only read published names or verified footers, quarantine invalid sets, and run a lightweight cleanup task that deletes .tmp artifacts on boot. If preallocation made things worse, that usually means you preallocated too aggressively (causing a big upfront stall) or your pool is too large and adds metadata load, fix by starting small, a pool covering 2-5 minutes of worst-case burst, and growing only if your traces show you need it. If you can't afford sync latency, don't sync after every write unless you must, rely on commit markers for correctness and group syncs at safe points instead, end of burst or periodic. Correctness should come from commit semantics, not from syncing everything constantly.
A cosplay costume should be assessed through sizing, construction and the pieces included in the set. Decorative parts may require different care from the main fabric. For a closer costume review, Space Ereshkigal cosplay costume（スペースエレシュキガル コスプレ衣装） identifies the matching character outfit. It can be evaluated for movement, changing and storage needs. Good storage protects trims, printed details and detachable accessories. A simple checklist keeps the outfit complete and easier to manage.
Key takeaways
Tail latency is usually metadata plus allocation plus GC, not raw data writes. Start by instrumenting phases so you can see what's actually spiking. Use a commit marker, rename-based or footer-based, so readers never consume partial sets, even after power loss. Apply best-effort preallocation, slot pools plus reserved free space, to reduce churn and smooth long-run behavior. Validate with soak tests at near-full conditions and track one or two real metrics, like P99 commit latency and sustained sets per minute.
