Identifying Dogs by Facial Landmarks: Building a Multi-Stage Vision Pipeline

How we're building a dog facial recognition system from scratch using landmark alignment, metric learning, and a custom-trained embedding model.

The problem

EverBowl is Hoomanely's precision healthcare device for dogs. It monitors eating and drinking habits, ocular temperature, sounds, and behavioral patterns over time, turning routine visits to the bowl into a continuous health signal for pet owners.

That works well in a single-pet household. But a chunk of our users have two or more dogs sharing the same bowl, and once multiple pets use the same device, the data gets ambiguous: we know something happened, but not which dog caused it. Tracking individual health metrics in a multi-pet household means knowing, per visit, which dog is actually at the bowl.

The obvious first idea, bolting a classifier onto the camera, breaks down fast. A classifier needs a fixed set of known dogs at training time, but users add new pets, households change, and the system has to keep working without retraining every time that happens. What we actually needed was identity-based recognition, not classification, which meant building a face recognition pipeline for dogs.

Why not just use a pretrained model

Dog face recognition isn't a solved problem the way human face recognition is. There's no dog equivalent of FaceNet with comparable quality and generalization. The few public datasets that exist, like the Zenodo DogFaceNet dataset, help with pretraining, but they don't capture the visual conditions our cameras actually produce: fisheye distortion, overhead angles, low-contrast infrared-adjacent imagery, and dogs sitting very close to the lens. Most of this we had to build ourselves.

System overview

The system runs in three distinct stages, each triggered at a different point in the product lifecycle, all sharing the same preprocessing pipeline.

Training runs infrequently, only when we need to improve or retrain the embedding model. It processes labeled image collections from known pets through the preprocessing pipeline to produce face crops, then trains the model on those crops combined with public data.

Gallery building runs whenever a new pet needs to be handled by the system. It takes the already-trained model and embeds that pet's face crops into a mean identity vector, then appends it to the gallery file. No retraining needed. Adding a new pet takes seconds.

Inference runs in real time on every new camera frame. It pushes the frame through the same preprocessing pipeline, embeds the resulting crop with the trained model, and matches it against the gallery to identify which pet is at the bowl.

The preprocessing pipeline is the shared foundation underneath all three stages, and getting it right is what makes everything downstream actually work.

The shared preprocessing pipeline

All three stages start here. Raw camera frames are messy: motion blur, lighting changes, partial occlusion, profile poses, multiple dogs in frame at once. We built a seven-step pipeline to filter all of that down to good, frontal face crops.

Step 1, quality gate: every frame gets rejected if its Laplacian variance (a proxy for sharpness) falls below threshold, or if mean brightness is too dark or blown out. This alone filters out a large share of nighttime and fast-motion frames before any model even runs.

Step 2, dog detection: YOLOv8-nano, pretrained on COCO, detects the dog's bounding box in the bowl camera image. We keep only the highest-confidence detection per frame and pad it with a 15% margin so downstream models get full context.

Step 3, eye gate: we reuse the eye segmentation model already built for EverBowl's ocular temperature pipeline. It runs on the dog crop and checks for a geometrically plausible eye pair: horizontally aligned (vertical ratio below 0.6), at least 15 pixels apart, both eyes in the upper half of the crop. This gate is fast, so frames without a valid eye pair get skipped before the slower landmark model ever runs.

Step 4, landmark detection: a 46-point TFLite landmark model runs on the dog crop at 384x384 resolution, returning normalized (x, y) coordinates across four anatomical groups: outer head and ear periphery (points 0-5), cheek and face boundary (6-11), eyes (12-23), and nose and muzzle (24-45).

Step 4a, structure check: before alignment, a rotation-invariant check confirms the raw landmarks actually form a coherent dog face. The eye cluster centroid and nose cluster centroid need meaningful separation, each cluster needs to be internally compact relative to that separation, and the two sub-centroids of the eye cluster (split left and right) need enough distance between them to confirm both eyes are actually visible. Profile views, where only one eye shows, get rejected here.

Step 5, face alignment: a similarity transform (rotation, uniform scale, translation, no shear) maps the detected left-eye centroid, right-eye centroid, and nose centroid onto a fixed canonical template, eyes at (0.30, 0.38) and (0.70, 0.38), nose at (0.50, 0.62). OpenCV's estimateAffinePartial2D gives us the transform matrix, and warpAffine produces a 224x224 crop from it. If more than 20% of the output pixels come out as black fill, meaning the face was far off the canonical position, the frame gets rejected.

Step 6, pose filter: on the aligned 224x224 crop, we check coverage (the landmark bounding box has to span at least 30% of the crop on both axes) and yaw symmetry (the left and right spread of eye landmarks can't differ by more than 4x). Frames failing either check get discarded.

Step 7, output: the 224x224 crop exits differently depending on the stage. During training and gallery building it gets saved to disk, organized by pet identity. During inference it goes straight to the embedding model.

The annotated frame below is a typical example: the green box is the YOLO dog detection, the red dots are the validated eye pair, and the inset shows the 46 landmark points, color-coded by group, on the aligned face crop.

Training flow

Building the dataset: the preprocessing pipeline runs across all images from known pets captured by EverBowl cameras, saving one 224x224 aligned face crop per passing frame per pet. We combine that with the public Zenodo DogFaceNet dataset, which adds roughly 1,254 additional dog identities and improves the embedding model's general discriminability.

Here's what the resulting training set looks like for our own pets: each row is one dog, each column is one aligned 224x224 crop.

Architecture: MobileNetV2, pretrained on ImageNet, serves as the backbone. We replace the classification head with a linear projection down to 512 dimensions, followed by batch normalization and L2 normalization, giving us a unit-norm embedding vector as output.

Loss function: we use ArcFace (Additive Angular Margin Loss) with scale s=64 and an angular margin of 0.5 radians. ArcFace adds a fixed angular penalty to the target class logit during training, which forces the model to learn a much tighter angular separation between identities than standard softmax would give us. At inference time we discard the classification head entirely, and identity matching is purely cosine similarity between embedding vectors.

Handling data imbalance: the Zenodo dataset is roughly 12 times larger than our own EverBowl crops, which would cause the model to underfit on our actual pets if left alone. We use a WeightedRandomSampler that gives EverBowl identities a 4x sampling boost, so they show up frequently in every batch regardless of the imbalance.

Augmentation: training applies random horizontal flips, color jitter, and small random rotations. Validation uses only resizing and normalization, nothing else.

Training loop: we run 60 epochs with AdamW (lr=1e-3, weight decay=1e-4) and cosine annealing down to lr=1e-5. Every 5 epochs we evaluate rank-1 nearest-neighbor retrieval on two held-out validation sets: Zenodo identities held out at the identity level (an open-set evaluation), and EverBowl identities with two images held out per pet. We save the checkpoint with the best EverBowl rank-1 accuracy as the production model.

Gallery building

Gallery building runs independently of training, any time a new pet joins a household, or whenever we want to refresh a pet's embeddings after collecting more photos.

For each pet, we run all of their aligned face crops through the trained embedding model and average the resulting 512-dimensional vectors into a single mean embedding, which then gets L2-normalized. That mean embedding represents the pet's identity in the space the model has learned.

The gallery itself is stored as a .pt file mapping each pet's name to its mean embedding, and it's completely separate from the model checkpoint. Adding a new pet just means running their crops through the existing model and appending one vector to the gallery file, with no retraining and no disruption to any of the existing identities already in there.

Inference flow

Inference runs on every new camera frame across two stages.

Stage 1, preprocessing: the new frame runs through the same seven-step pipeline described above. If any step fails, whether it's blurry, has no dog, no valid eye pair, a landmark structure failure, an alignment failure, or a pose failure, the frame produces no identification result, and we log which stage it failed at.

Stage 2, matching against the gallery: for a frame that clears preprocessing, the aligned 224x224 crop gets embedded by the model into a 512-dimensional unit-norm vector. We compute cosine similarity between that vector and every mean embedding in the gallery. The highest similarity wins, provided it clears a configurable threshold (0.50 by default); frames below that threshold get marked unknown.

What's next

This post covers the pipeline design and the reasoning behind it. In the next post, we'll share results from our own EverBowl dataset: identification accuracy, a breakdown of failure modes by stage, threshold calibration, and where the system still struggles.

This pipeline is the first step toward making EverBowl work well for multi-pet households.