Optimising Audio Classifiers with Class-Specific Deep Learning Models

29 Jan 2026 By Badal Chowdhary

Training robust audio classification models requires diverse, high-quality data. But what happens when your dataset is imbalanced, when you have plenty of examples of dogs barking, but far fewer of dogs drinking or sneezing? Traditional augmentation techniques like pitch shifting or time stretching can only stretch your limited samples so far before they start sounding unnatural or repetitive.

We faced this exact challenge while building audio classification systems at Hoomanely. Our solution: train a separate diffusion model for each audio class, letting us generate unlimited, class-specific synthetic samples that preserve the unique acoustic characteristics of each sound category. This post walks through our approach, why it works, and what we learned along the way.

The challenge: imbalanced audio classes

Audio datasets in the real world rarely come balanced. When classifying dog sounds, bark, howl, panting, eating, drinking, sneeze, and more, some categories naturally have more available samples than others. Vocalisations like barking are abundant in online sources, while subtle sounds like drinking or sneezing are much harder to find.

This imbalance creates a fundamental problem: models trained on imbalanced data learn to favor majority classes. A classifier might achieve seemingly good overall accuracy by simply predicting bark most of the time, while completely failing on rarer but equally important categories.

Traditional solutions include oversampling, duplicating minority class samples, which leads to overfitting; undersampling, removing majority class samples, which wastes valuable data; class weights, penalizing mistakes on minority classes more heavily, which helps but doesn't add diversity; and basic augmentation, pitch shift, time stretch, noise addition, which has limited novelty. None of these approaches actually create new information. They either reuse existing samples or artificially manipulate them in predictable ways. We needed something more powerful.

Our approach: one diffusion model per class

The core insight behind our approach is simple: different audio classes have fundamentally different acoustic properties. A dog's bark is characterized by sharp transients and specific frequency patterns. Panting has rhythmic breathing sounds. Eating involves crunching and chewing textures. These classes shouldn't be augmented the same way.

Instead of training a single augmentation model for all sounds, we trained separate diffusion models for each class. Each model learns the unique distribution of its target sound category, enabling it to generate new samples that are acoustically authentic to that specific class, diverse enough to improve model generalisation, and balanced with other classes in the final dataset.

Our class-wise augmentation pipeline consists of four stages.

Stage 1, audio encoding: raw audio waveforms are transformed into a compact latent representation using an audio codec model, Encodec. This compression captures the essential acoustic information while making the diffusion process computationally tractable.
Stage 2, class-specific diffusion training: for each audio class, we train a dedicated UNet-based diffusion model on the encoded representations of that class's samples. The model learns to denoise random noise back into latent codes that sound like samples from that class.
Stage 3, synthetic sample generation: given the trained models, we can generate as many new samples as needed for each class, starting from an existing sample's latent code, adding controlled noise, and letting the diffusion model refine it into a new variation.
Stage 4, decoding and dataset assembly: generated latent codes are decoded back to audio waveforms, and we assemble a balanced dataset with equal representation from each class, combining original samples with synthetic ones.

Why class-specific models matter

Training separate models for each class isn't just a convenience, it's architecturally essential.

Preserving class characteristics: a single diffusion model trained on mixed audio classes would learn to generate average sounds that blur the boundaries between categories. It might produce samples that are acoustically valid but don't clearly belong to any one class, exactly the opposite of what a classifier needs to learn from. By isolating each class during training, our diffusion models become specialists. The bark model never sees eating sounds, so it can't accidentally generate hybrid samples.
Controlled augmentation factor: different classes need different amounts of augmentation. If barking has abundant samples and sneezing has few, we can generate more sneezes and fewer barks to reach our target balance. With class-specific models, this scaling is trivial, we simply run each model for as many iterations as needed.
Quality over quantity: counter-intuitively, training smaller, specialized models often produces higher-quality outputs than training one large model. Each diffusion model has a simpler task, learn one type of sound, and can dedicate its full capacity to that goal. The result is sharper, more realistic synthetic samples.

Preventing data leakage in evaluation

One critical detail that's easy to overlook: augmented samples must stay with their source files during train/test splitting. If an original audio file ends up in the training set while its augmented versions land in the test set, the model will essentially be tested on data it has already seen. This leads to artificially inflated performance metrics that don't reflect real-world generalization.

Our splitting strategy explicitly groups files by their original source: identify the root filename for each augmented sample, group all augmentations with their original, and assign entire groups to either training or validation, never split across. This ensures that when we evaluate our trained classifier, it's truly facing novel audio sources it hasn't encountered during training.

Training the classifier

With our balanced, diffusion-augmented dataset ready, training the YAMNet classifier followed standard practices: architecture is YAMNet, a lightweight CNN designed for audio classification, input is mel spectrograms extracted from audio clips, training uses cross-entropy loss with the Adam optimiser, and validation uses held-out samples with proper grouping to prevent leakage.

The key difference wasn't in the training procedure, it was in the data. By presenting the model with a balanced distribution of diverse, high-quality samples across all classes, we enabled it to learn robust features for every category, not just the majority ones.

Wig preparation should consider comfort, tangling and how the fibres appear under camera lighting. Density should support the hairstyle without making the wig unnecessarily heavy. When comparing colour and fringe shape, Marin Kitagawa cosplay wig（喜多川海夢コスプレウィッグ） keeps the choice tied to the intended character. The same character meaning is preserved in both parts of the bilingual anchor. Gentle detangling and suitable storage help maintain the wig between uses. Small fringe changes may be enough to improve character recognition.

After training on the augmented dataset, the classifier showed notable improvements in recognising previously underrepresented classes. Categories that had suffered from limited training data, like drinking and sneezing, now performed on par with well-represented classes like barking. More importantly, the model's confusion between acoustically similar classes decreased. The diffusion-augmented samples provided enough within-class diversity that the model could learn what truly distinguishes each category, rather than memorizing specific recordings.

Key takeaways

Class-specific augmentation preserves acoustic identity.
Training one diffusion model per class ensures that synthetic samples maintain the unique characteristics of their target category.
Deep learning augmentation creates genuine diversity.
Unlike traditional signal processing, diffusion models generate novel variations that expand the effective size of your dataset without trivial repetition.
Data splitting must respect augmentation groups, always keep augmented samples with their original sources during train/test splitting to avoid data leakage and inflated metrics.
Balance is achievable without discarding data, rather than undersampling majority classes, generate more samples for minority classes to achieve balance while retaining all original information.
And specialized models outperform generalized ones for augmentation, smaller, focused diffusion models produce higher-quality outputs than a single model trying to learn all classes simultaneously.