What The Training Runs Taught Us About IMU Bark Detection

In our previous post we built the first version of an ML bark classifier: 28 features extracted from collar IMU windows, a RandomForest winning a model bakeoff, bark F1 of 0.714 on a held-out test set. That number looked good. It was wrong.

The Number That Was Lying to Us

The 0.714 came from a held-out test set drawn from the same recording days as the training data. That sounds rigorous until you consider what sharing a day means: the dog's activity level, collar fit, recording environment, and mood all correlate across clips from one session. A model trained on 80% of a day and tested on the remaining 20% is partly memorizing the day, not learning to bark.

When we ran the same model against a genuinely new recording session it had never seen, F1 dropped to 0.10. Same model, same threshold, seven times worse. Same-day splits inflate results roughly 2x on this dataset. Every model that looked promising in early training looked terrible on the first honest new day.

The Protocol Fix

We switched to leave-one-session-out (LOSO) as the primary evaluation metric. Each recording session is held out in turn, the model trains on everything else, and results aggregate across all held-out sessions. Every number we report now is cross-day. Same-day splits survive only as a smoke test during development.

The collection protocol changed too. When a new labeled session arrives, we first score every saved model against it before ingesting anything. That session is pristine data, never seen by any model. We record the scores, then add the session to training. Each day earns its honest benchmark exactly once, and the scoreboard builds itself automatically.

Eight Runs, One Clear Lesson

Between the initial classifier and today we ran eight training iterations, adding sessions from three collar devices and one dog. We also ran a full model bakeoff: nine configurations including Random Forest, Gradient Boosting, SVM, logistic regression, and MLP. The bakeoff result was humbling. At matched recall, every competitive configuration landed within about 8% false positives of each other. The classifier was not the bottleneck. The data was.

Three data interventions moved the LOSO numbers far more than any model change:

On-dog filtering. Not every clip in a session was recorded while the collar was on a dog. Collars sit on desks and get set down. We keep a not_bark clip for training only if it fell within 1 minute of a verified bark (proving the dog was wearing the collar at that moment), or if the session is explicitly vouched. Everything ambiguous is dropped. The effect on cross-day generalization was immediate.

Verified negatives from monitored sessions. A fully monitored bark session has every bark clip marked. Every unmarked clip in the same session is therefore a verified negative: the collar was on the dog and the dog was not barking. We extracted 98 training-grade negatives from sessions we already had, with no new collection needed.

Ruthless exclusions. A mislabeled bark found during review means the full session gets audited. An ambient-audio test session where the collar was hand-rotated goes to evaluation only. The motion is real but the dog is not. Small datasets amplify label noise, so data hygiene is effectively a feature.

The LOSO trend across all eight runs:

Model  Data                                        recall       precision
v1     4 days, 1 device, unfiltered negatives       0.20          0.14
v4     +3 days, 2 devices, on-dog filtering         0.74          0.18
v6     +2 bark sessions (new device), label fix          0.78          0.24
v7     +/-1-min verified negatives                  0.87          0.24
v8     +98 verified session negatives               0.88          0.36

What the IMU Can and Cannot See

A collar IMU does not sense the bark. A 0.3-second acoustic event barely moves a collar. What it senses is the behavioral state around barking: an aroused, often pacing dog barking in bouts. We measured this directly: bark windows show sustained elevated motion across sub-windows, while many false-positive triggers are calm windows with a single brief movement spike.

The practical consequence: a dog that barks while standing still is nearly invisible to the IMU. Every model we trained scored those clips around p=0.18, well below any useful threshold. That is the recall ceiling for IMU-only detection, and it is a physics constraint rather than a modeling one.

The Signal That Was Already There

While debugging false positives, we looked at the on-device audio classifier output more carefully. Its final label was not useful for our dog (it usually returned "eating" because our dog's barks rarely crossed its per-segment confidence gate). But its raw bark-family probability, bark and yip class scores summed, turned out to rank our labeled clips with AUC 0.904. The IMU model achieved AUC 0.67. The better signal had been computed on every clip since day one. Only the decision layer was discarding it.

Audio and IMU fail in complementary ways. Audio hears the bark but cannot confirm the wearer is the one barking. IMU confirms the wearer is moving in a bark-like pattern but misses still-dog barks entirely. Combining them with a simple fusion rule:

fused_is_bark = audio_family_prob >= 0.12
                OR (audio_family_prob >= 0.03 AND imu_p >= 0.25)

On a 196-clip evaluation set (43 barks, 153 verified negatives, IMU scores strictly out-of-fold):

System                Precision  Recall  F1    False Positives
Pure IMU (thr=0.25)   0.29       0.86    0.43  91
Audio + IMU fusion    0.46       0.86    0.60  43  (-53%)

Same recall. 53% fewer false positives. No new sensors, no new models, one rule over numbers already present in every event.

Above both sits a bout rule: send a notification only when at least two flagged clips arrive within 5 minutes. Real barking arrives in bouts; isolated false positives tend to arrive alone. Full-day backtests showed roughly 4x improvement in notification-level precision with no real bark episodes missed.

Where Things Stand

The fusion rule is running in shadow mode: it logs its verdict alongside the current live verdict on every incoming event without changing anything the user sees. First forward validation on a fresh labeled session: 6 of 6 barks correctly flagged.

The current model is v8, a RandomForest on 28 IMU features trained on 75 barks and 192 verified negatives (no undersampling needed once negatives are verified), decision threshold 0.25. Every event it produces carries the model version, the probability, and the threshold, so any question about a specific verdict is answerable from the stored data.

What Is Next

Shadow mode runs until we have enough forward validation data to set fusion thresholds with confidence rather than on pooled evaluation data. Per-device onboarding is already in the rollout plan: every new collar device produces a false-positive surge until one of its labeled sessions enters training, so new deployments will collect a labeled warm-up session first.

75 confirmed barks from one dog is a real number with real limits. The model generalizes well enough to be useful, but the precision numbers will tighten substantially as bark count grows past 150 and per-device sessions accumulate. That collection is already underway.