How BERT Learned Pet Questions (And Got Really Good at It)

How BERT Learned Pet Questions (And Got Really Good at It)

By Surya

Why we needed a question classifier

At Hoomanely, pet parents send us thousands of questions covering health, behavior, grooming, food, and training. Those questions tell us what people actually care about and where the product falls short, but reading every one by hand is slow, inconsistent, and doesn't scale.

So we built an internal BERT-based classifier that tags incoming questions into defined categories the moment they arrive. It's trained on 4,401 questions that GPT-5 labeled first, which our team then checked by hand to make sure the training data was reliable.

The dataset: 4,401 questions, reviewed by hand

We had GPT-5 label thousands of historical questions into seven categories, then our team audited and corrected the results to remove drift and mislabeling.

The distribution skews toward Health and General, which matches what we'd expect in the real world, and it became the backbone of the fine-tuned classifier.

Bar chart of question category distribution across the labeled dataset
Bar chart of question category distribution across the labeled dataset

Why BERT for this

BERT works well here for a few reasons: it reads context rather than just matching keywords, it handles short and noisy queries well, fine-tuning needs comparatively little data, and lightweight variants like DistilBERT or MiniLM are fast enough for real-time use.

Instead of hand-written rules or a TF-IDF baseline, BERT learns the semantic pattern in a question, recognizing that "My dog keeps shaking its head" is Health, "How do I stop leash pulling?" is Training or Behaviour, and "Can puppies eat curd?" is Food.

Example questions mapped to their predicted categories
Example questions mapped to their predicted categories

How the classification works, conceptually

Tokenization converts the text into WordPiece tokens. Embedding and transformer layers encode contextual meaning. The CLS token becomes a summary vector for the whole sentence. A small feed-forward classification head maps that vector to a category.

Input Question → Tokenizer → BERT Encoder → CLS Vector → Dense Layer → Softmax → Category
Flow diagram of tokenization through the classification head
Flow diagram of tokenization through the classification head

Building the training pipeline

Step 1, clean and standardize: remove duplicates, lowercase and strip punctuation only where it helps, convert categories to numeric labels, and split roughly 85/15 with stratification.

Step 2, model choice: DistilBERT or MiniLM as the base, one or two linear layers with dropout on top, AdamW as the optimizer, cross-entropy loss, and a max length of 64 tokens, which is plenty for a question.

Step 3, fine-tuning: batch size 16-32, three to five epochs, learning rate warmup and decay, early stopping. BERT converges fast once the labels are clean, so we didn't need long training runs.

Training loop diagram showing batching, epochs, and early stopping
Training loop diagram showing batching, epochs, and early stopping

The pipeline, end to end

Historical Questions ──► GPT-5 Labeling ──► Human Audit
                                         │
                                         ▼
                                 Clean Labeled Dataset
                                         │
                                         ▼
                                  Fine-tune BERT Model
                                         │
                                         ▼
                         Real-Time Question Categorization Service

How we use the classifier at Hoomanely

  • Understanding user needs at scale: we can see in real time which domains, health, food, training, people struggle with most.
  • Prioritizing content and product work: if grooming questions spike, our content and product teams know exactly where to focus.
  • Routing inside our AI stack: Health questions pull from clinical RAG data, Behaviour routes to behavioral models, General gets handled by lightweight FAQs.
  • Better long-term data quality: consistent classification makes historical analytics something we can actually trust.

What we learned

  • GPT-5 labels are a good starting point, but human review still matters. Small labeling noise hurts fine-tuning more than you'd expect.
  • Four to five thousand clean examples is enough for a solid BERT classifier.
  • Smaller BERT variants hold up well on short queries.
  • A consistent taxonomy matters more than which architecture you pick.

Final thoughts

This project made our understanding of user questions something we can measure instead of guess at. Reliable automatic categorization means better answers, better routing, and sharper product decisions. As the volume grows into the tens of thousands, the classifier will keep evolving with new categories and richer signals.

If you're building something similar: start with clean labels, pick a small BERT variant, and fine-tune just enough. Not more than that.

Read more