Audio classification on a microcontroller: mic to NPU to storage

Our new camera module, running on its evaluation board, now classifies dog sounds on its own microcontroller. It turns the microphone into 16 kHz audio and computes a 64-band log-mel spectrogram on the CPU in about 320 µs per frame. Every 480 ms, an int8 eight-class network on the neural accelerator (NPU) scores the last 0.97 s of sound in 66.4 ms, with a typical result 125 ms after the window's last sample. Each 10 s of audio is stored on the module's eMMC with its decisions and read back for a checksum, and in a 680 s labeled bench run, 68 of 68 records verified with 0 samples lost.

This post covers each stage in order: continuous capture, the spectrogram, the network on the NPU, and the record store. It also explains how latency is measured on the audio clock, then closes with the bench results and what this lets EverBowl do.

The short version

• The microphone is captured without gaps into a 4.096 s ring of 16 kHz audio, and a 680 s run missed 0 audio blocks.
• The on-board log-mel spectrogram matched the Python reference (librosa) on 19 of 19 clips, with a worst difference of 0.00002 dB.
• An int8 eight-class network scores a 0.97 s window every 480 ms in 66.4 ms on the NPU, with a typical result 125 ms after the last sample.
• Each 10 s record is stored with its decisions and verified by reading it back, and 68 of 68 records passed on the bench.

Why the camera module classifies its own audio

The bowl's main processor already runs an eight-class dog-audio model: bark, yip, howl, panting, eating, drinking, sneeze, and none. The camera module carries its own NPU, which already runs the dog-image network described in our post on the first neural network on the camera module. Running the same audio model there means the module can attach a decision to every second of sound before any file leaves it.

The approach follows the same idea as our on-device acoustic event detection work: decide close to the microphone. The difference here is that a full neural network does the deciding, on microcontroller-class silicon, alongside a camera.

Continuous capture without gaps

The microphone's converter delivers 48 kHz stereo into a circular DMA ring of four quarters, each holding 4,096 frames (about 85 ms). The DMA never stops. The main loop drains finished quarters, takes the microphone channel, filters it with a 47-tap low-pass, and keeps every third sample. The result is a 65,536-sample ring of 16 kHz mono audio, which holds 4.096 s.

The low-pass is a 47-tap windowed-sinc filter with a 6.8 kHz cutoff, built in 16-bit fixed point. It attenuates by 26 dB at 8 kHz and by 54 dB at 9 kHz, so sound above the new 8 kHz limit does not fold back into the audio the model hears.

The drain has to stay fast and has to notice if it ever falls behind. It handles at most two quarters per call, so the camera loop keeps its cadence:

488	    uint32_t done = s_q_done_isr;
489	    uint32_t pending = done - s_q_done_cpu;
490	    if (pending > AUD_QUARTERS - 1U) {                       /* the DMA has lapped us: skip to the oldest intact quarter */
491	        s_missed += pending - (AUD_QUARTERS - 1U); s_skip_total += pending - (AUD_QUARTERS - 1U);
492	        s_q_done_cpu = done - (AUD_QUARTERS - 1U);
493	        pending = AUD_QUARTERS - 1U;
494	    }
495	    for (uint32_t n = 0; n < pending && n < 2U; n++) {       /* at most 2 quarters per call (keeps the capture loop cadence) */
496	        aud_process_quarter(s_q_done_cpu % AUD_QUARTERS);
497	        s_q_done_cpu++;
498	    }

Any lap is counted, never hidden, so a clean run proves itself. The filter and decimation take about 1.5 ms per 85 ms quarter. In a 90 s bench run beside the live camera loop at 15 fps, the ring logged 0 overruns and 0 DMA errors at a steady 16,000 samples per second. The same principle of letting DMA feed the data while the model waits is covered in our post on DMA-first pipelines.

A spectrogram that matches the training pipeline

A network only works on device if its input is computed exactly as it was in training. The front end follows the model's own definition: a 400-sample Hann window centered in a 512-point FFT, a hop of 160 samples, 64 mel bands from 125 to 7,500 Hz, and power converted to decibels with a floor of 10⁻¹⁰.

Frames stream as audio arrives. One new frame is computed for every 160 new samples, as soon as its last sample is in the ring, so the board produces 100 frames per second. Each frame is a float FFT plus 64 short dot products against a precomputed filterbank, and takes about 320 µs. The frames land in a 256-frame ring, and the classifier reads a 97-frame patch from it every 48 frames.

To prove the match, the board can send a clip together with the spectrogram it computed for that clip. A host script then runs librosa on the same WAV file and compares every value. On 19 of 19 clips, the worst difference was 0.00002 dB. A separate check confirmed that frames recomputed for a clip were identical to the frames from the live stream.

Running the classifier on the NPU

The network is a MobileNet-style transfer model built on YAMNet, quantized to int8 and compiled for the NPU. It takes one 64 × 97 patch, performs 80 million multiply-accumulates, and returns eight raw int8 outputs. The CPU then converts those outputs to probabilities with a softmax.

Each patch is quantized to int8 on the CPU and laid out band by band, the same orientation the model was trained on. The main loop starts a run as soon as a new patch exists and records each decision for the file being built:

648	        if (ms.patches > s_patch_done) {
649	            if (ms.patches - s_patch_done > 1U) { s_bark_skipped += ms.patches - s_patch_done - 1U; }
650	            uint32_t p = ms.patches - 1U; s_patch_done = ms.patches;
651	            if (g_bark_ready && App_Audio_MelPatch(p, s_bark_patch) == 0) {
652	                /* Build A6: age of the window's last audio sample (frame 48p+96 needs 16 kHz samples up to its centre + 199) */
653	                const uint32_t w_last = (p * 48U + 96U) * 160U + 199U;
654	                const uint32_t lat_ready = App_Audio_AgeUs(w_last);
655	                AppBarkResult br; int brc = App_Bark_Run(s_bark_patch, &br);
656	                const uint32_t lat_done = App_Audio_AgeUs(w_last);
657	                if (brc == 0) {
658	#if APP_AUDIO_REC
659	                    App_Rec_AddDecision(p, br.logit_q, lat_ready, br.us, lat_done);
660	#endif

Memory is shared carefully. The 3.3 MB weights file is loaded from the eMMC into external memory at boot, and the network is enabled only after its size and CRC-32 match. The 147 KB of activations sit in NPU memory that the dog-image network also uses, so the audio network never starts while the image network is running.

We checked the port number by number. On 37 of 37 patches, the int8 input the board fed the network was identical to the host's. The board's outputs were as close to the original float model as a desktop int8 runtime (ONNX Runtime) was, with a mean logit error of 0.466 against 0.470. On a second run of 45 patches, the board's bark probability stayed within 0.008 of the host's int8 result on every one.

Recording with the decisions inside the file

Audio is cut into back-to-back 10 s files on the eMMC. Each file is a standard 16-bit mono WAV followed by a custom chunk listing every decision made while it was recorded. Each entry is 24 bytes: the patch index, the eight raw outputs, and three timings. A 10 s file holds 20 or 21 decisions and is 320,548 bytes.

Step 1: build and checksum

The WAV header, samples and decision chunk are assembled in external memory, and a CRC-32 is computed over the whole file. This takes about 41 ms.

Step 2: write and read back

The file is written and flushed in about 20 ms, then read back into a second buffer and checked against the CRC in about 50 ms. The audio ring is drained between steps, so capture continues throughout.

Step 3: send and prune

The verified file is queued on the bench link in 128 KB parts, one part per loop pass, so classifier decisions keep their cadence. After the last part, the oldest file beyond the newest 30 (about 10 MB) is deleted.

Measuring latency on the audio clock

Wall-clock timers cannot say how old a sample is. Instead, the firmware reads the DMA's position in the stream, works out how many 48 kHz frames have arrived since a given 16 kHz sample was completed, and converts that to microseconds. Every stage below is measured that way, on the board:

What this lets EverBowl do

EverBowl™ learns a dog's routine from what happens at the bowl, and sound at the bowl is one of its inputs. With this pipeline, the camera module scores every half second of sound near the bowl against eight classes, such as eating, drinking or barking, and stores the audio with every decision inside the same verified file. Each moment of the record carries both what was heard and what the module understood it to be.

The honest limit is that this runs on our evaluation board today. The model is still being tuned on real bowl recordings, and the module currently keeps every record rather than choosing which to keep from its scores. EverBowl does not diagnose; the value is the record.

Conclusion

Audio classification on a microcontroller works when every stage is built to be checked. Keep capture on DMA and count every lap, compute the spectrogram exactly as the model was trained and prove it against the reference library, and verify the network's input number by number. Measure latency on the audio clock rather than the wall clock, and read every stored file back before trusting it. More of our device-side work is in the embedded systems category.