How We Distilled SOTA Transformers for Real-Time Edge Audio Classification

Audio classification on edge devices presents a unique engineering challenge that doesn't get enough attention in the ML community. While everyone's building massive transformer models that need GPU clusters, we needed something that could run on a Raspberry Pi and accurately detect different types of dog sounds in real-time. This is the story of how we bridged that gap using knowledge distillation with CLAP transformers.

The edge audio problem nobody talks about

Audio classification on constrained hardware is fundamentally harder than image classification, and here's why: you can't just look at an audio file and understand what's wrong. With images, a developer can visually inspect mislabeled data. But audio? It's just a waveform or spectrogram that requires domain expertise to interpret.

Add to this the real-world complications, background noise, varying recording quality, different microphone characteristics, and the computational constraints of edge devices. You need a model that's both small enough to run on limited hardware and accurate enough to be useful. That's a tough combination. Most edge audio solutions compromise on one of three things: accuracy (simple models that miss nuanced sounds), latency (cloud-based inference that introduces delays), or generalisation (over-fitted models that fail in new environments). We needed all three: high accuracy, real-time inference, and robustness across different acoustic environments.

The state of edge audio classification

The standard approach for audio classification on edge devices typically involves lightweight CNN architectures or simplified recurrent models. Traditional CNNs on mel spectrograms convert audio into visualised frequency-over-time representations and run a small convolutional network, this works but lacks the contextual understanding that makes modern AI powerful. Quantised mobile models take a larger model like YAMNet or VGGish and compress it through quantisation, but you're starting with models that weren't designed for your specific use case, and compression inevitably hurts performance. Feature engineering plus classical ML extracts handcrafted features, MFCCs, zero-crossing rates, spectral features, and uses SVMs or Random Forests, interpretable but brittle. Transfer learning from general models fine-tunes pre-trained audio models on your specific task, sounds good in theory, but these models are usually too large for edge deployment, or lose critical knowledge during aggressive compression.

None of these methods effectively transfer the knowledge of state-of-the-art models to edge-compatible architectures. Quantisation and pruning help with size, but they're crude tools, you're essentially hoping important information survives compression. We needed a principled way to distill the understanding of a massive, accurate model into something that could run on a $50 piece of hardware.

Our approach: knowledge distillation with CLAP

We designed a knowledge distillation pipeline using CLAP, Contrastive Language-Audio Pretraining, as our teacher model and a custom MelCNN as our student model for edge deployment.

Why CLAP as the teacher? CLAP is a transformer-based model that understands audio in context, it's been trained on massive datasets to align audio and text representations. This means it doesn't just classify sounds, it understands them conceptually. Why MelCNN for the edge? We needed something computationally efficient that could process mel-spectrograms with minimal overhead. The key insight: we don't need the student to replicate the teacher's architecture, just its understanding. CLAP's job was to provide high-quality labels and soft targets, the MelCNN's job was to learn those patterns in a computationally efficient way.

The data pipeline: quality over quantity

Raw audio data from the internet is messy. Really messy. Dogs barking with background music, outdoor recordings with wind noise, videos with human speech overlapping barks, none of this would train a reliable model.

Our three-stage data pipeline: stage 1, large-scale scraping, we collected thousands of dog sound audio samples from various sources, YouTube, sound libraries, user submissions, prioritizing volume and diversity across breeds and recording conditions. Stage 2, noise cancellation with AudioSep, we ran every audio sample through AudioSep, a source separation model that can isolate specific sounds from noisy backgrounds, think of it as extremely advanced noise cancellation, it doesn't just remove noise, it separates out the dog sounds from everything else. This step was critical, without it CLAP would label ambiguous sounds incorrectly, and those errors would propagate to our student model. Stage 3, high-quality labeling with CLAP, clean audio samples went through CLAP for classification, because CLAP has such strong generalisation from its pre-training, it could accurately classify dog sounds into our target categories even without fine-tuning.

The beauty of this pipeline? We automated the creation of a high-quality labeled dataset without manual annotation. CLAP's strong zero-shot capabilities did the heavy lifting.

Training the student: distillation in practice

With our CLAP-labeled dataset ready, we trained the MelCNN using knowledge distillation. The student model learned from both hard labels, the class assignments from CLAP, and soft targets, the probability distributions over classes, which contain richer information about similarities between classes.

The MelCNN's architecture was designed specifically for edge deployment: efficient depth-wise separable convolutions, optimised input resolution with smaller mel-spectrograms, minimal fully-connected layers, and quantisation-friendly operations. The resulting model was under 2MB and could perform inference in under 50ms on a Raspberry Pi 4. But more importantly, it maintained over 90% of CLAP's accuracy on our test set, far better than training a small model from scratch would achieve.

What we learned, and what surprised us

AudioSep was non-negotiable, in early experiments without noise separation, the model would pick up on background correlations instead of the actual dog sounds, clean data made all the difference. CLAP's zero-shot power surprised us, we initially planned to fine-tune CLAP on our dog sounds, but its zero-shot classification was already so accurate that fine-tuning provided minimal gains, saving significant compute time. The student exceeded expectations, we expected some accuracy degradation with such aggressive model compression, but the knowledge distillation framework preserved much more information than traditional compression methods. Edge deployment has hidden complexity, model size is just one factor, memory bandwidth, cache utilisation, and quantisation strategies mattered as much as the architecture itself.

Key takeaways

• Audio classification on edge requires different thinking than cloud-based ML, you need to prioritise efficiency without sacrificing accuracy.
• Knowledge distillation bridges the capability gap between SOTA models and deployable ones, don't just compress, teach.
• Data quality matters more than quantity for specialised tasks, noise separation and high-quality labeling were force multipliers.
• SOTA models like CLAP can be powerful labeling tools even if you can't deploy them, use them as teachers, not just as endpoints.
• And edge constraints force better engineering, leading to systems that aren't just small, but robust and efficient.
• Building production ML systems is about making intelligent tradeoffs.
• For us, that meant accepting we couldn't run transformers on a Raspberry Pi, but we could teach a much simpler model everything those transformers knew.