Dog Identification by Facial Landmarks: Real-World Results and Failure Analysis

In our previous post we described the multi-stage vision pipeline we built to identify dogs by their facial landmarks - covering preprocessing, embedding model training, gallery building, and real-time inference. This post covers what happened when we ran it on real camera footage and reviewed every result manually.

Setup

A quick recap of the system: the embedding model was trained on a combined dataset - our own labeled face crops plus the public Zenodo DogFaceNet dataset, which adds around 1,254 additional dog identities to improve general discriminability. Once training was complete, the gallery was built as a separate step: face crops from 33 known pets were passed through the trained model, each pet's embeddings averaged into a single mean vector, and the result saved as a gallery.pt file. The model and gallery are decoupled - adding a new pet means updating the gallery, not retraining.

We evaluated on images from four days: June 19, 20, 22, and 23, 2026. Every detection was manually reviewed and marked as correct or incorrect through an annotation tool. In total, 149 images were evaluated across those four sessions.

Detection Rate

Of the 149 images, 100 produced a face detection and gallery match (67%).

The remaining 49 frames failed at one of three stages:

Dog not detected (20 frames, 41% of failures). The YOLO dog detection step found nothing. These are typically frames where the dog is too far from the bowl, the camera angle shows only a partial body, or the image is blurry enough that the quality gate passes but detection still fails.

No gallery match (19 frames, 39% of failures). The pipeline completed all seven preprocessing steps and produced a face crop, but the cosine similarity against every gallery entry fell below the 0.50 threshold. This is the correct behavior for an unknown or out-of-gallery dog - the system returns nothing rather than a wrong answer. We found that one pet, Sophie, was visiting the bowl on several days but had not been enrolled in the gallery. Most of her frames correctly fell into this bucket.

Eye pair not found (10 frames, 20% of failures). The eye gate rejected the frame before landmark detection even ran. These are mostly profile-angle shots where only one eye is visible, which the geometric validation correctly rejects.

The detection rate varied noticeably by day - 79% on June 19 dropping to 56% on June 20 and 23. We believe this reflects real differences in the angles and distances captured across sessions rather than a model issue.

What Is Holding Back Detection Rate

The 33% non-detection rate is the main area for improvement. The three failure modes point in different directions:

Dog not detected is largely a framing and angle problem. Frames where the dog is at a sharp angle or too far from the sensor are less likely to present a full frontal face to the camera. The current detection step uses YOLOv8-nano pretrained on COCO, which was not designed for our specific camera conditions. The most principled fix here is to train our own dog detection model on frames from our own sensor setup - this would give the detector exposure to the exact angles, distances, and lighting it will encounter in practice, rather than relying on threshold tuning of a general-purpose open-source model.

No gallery match has two components: genuinely unenrolled pets (correct behavior) and enrolled pets whose similarity falls below threshold (missed matches). Lowering the threshold would recover some of the latter but would introduce more false positives. Per-pet threshold calibration is one direction we want to explore.

Eye pair not found reflects real geometric constraints of the pipeline. Profile-angle images will always fail this gate by design - it is a feature, not a bug, since profile crops do not align well and produce unreliable embeddings. Addressing this would require a separate profile-view recognition path.

Accuracy on Detected Frames

Of the 100 detections, 94 were confirmed correct by human review (94%) across all four days.

The average cosine similarity on correct detections was 0.82 - comfortably above the 0.50 threshold, which suggests the model has good margin on identities it knows well.

Where It Gets Wrong

Six detections were assigned the wrong pet across all four days:

• Roopup was confused with Sophie on two occasions (June 19)
• Janu was confused with Sky (June 19)
• Scamp was confused with Sophie (June 22)
• Kutta was confused with Jake (June 22)
• Kiki was confused with Rani (June 22)

Sophie appears in three of the six confusions. This is not surprising: Sophie is simply not in the gallery. The fix is not retraining - it is a gallery update. Her crops need to be embedded using the existing model and added to gallery.pt, which takes seconds. Until that happens, the system has no representation to match her against. When her face clears all preprocessing steps and her embedding happens to sit close enough to an enrolled pet in the embedding space, it crosses the similarity threshold and gets assigned that pet's identity instead of being returned as unknown. This is a known edge case in open-set recognition - a pet not in the gallery who visually resembles an enrolled one will be misidentified rather than rejected.

The Kutta/Jake and Kiki/Rani confusions are more interesting. Both pairs are dogs of similar build and coloring. They are the cases we want to study more closely in future evaluation, as they suggest the embedding space may not have sufficient separation for visually similar breeds.

Missed Detections

32 frames were not detected by the pipeline but were labeled by the human reviewer as containing a known dog. Many of these were pets not currently in the gallery (Sophie, Sky, Rani), which the system correctly returned as no match - the right behavior. Adding them to the gallery requires only a gallery rebuild using the already-trained model, no retraining. The remainder were pets that are in the gallery but whose frame failed at the detection or eye gate stage - legitimate pipeline misses where a usable face crop could not be extracted.

Next Steps

The identification accuracy of 94% on detected frames is a strong starting point. The gap to close is the detection rate. Our next focus areas are:

• Adding all active pets to the gallery (a gallery rebuild, not a retrain) so that unenrolled-pet frames return as unknown rather than being misidentified as a similar-looking enrolled pet
• Calibrating the cosine similarity threshold per identity based on historical similarity distributions
• Investigating the visually similar pairs (Kutta/Jake, Kiki/Rani) with targeted evaluation

We will share threshold calibration results and per-pet accuracy breakdowns in a follow-up post once we have more reviewed data. The current evaluation was run on images captured manually with the sensor held by hand. A key next step is running the same evaluation in the natural bowl environment, where dogs approach on their own and camera framing is determined by the actual hardware placement rather than a person holding the device.

All results are from EverBowl camera footage reviewed through a manual annotation tool. Accuracy figures reflect human-reviewed ground truth only.