Similarity Search: A Deep Dive into Image Similarity Search

Similarity Search: A Deep Dive into Image Similarity Search

By Surya

Introduction

Image similarity search is one of the more versatile techniques in computer vision. Whether you're spotting duplicate images, powering a search engine, catching tampering, or running a lightweight recognition task on an edge device, the idea stays the same: turn images into numbers and measure how close those numbers are.

At Hoomanely, we use this in a small part of our food-detection workflow inside the smart bowl, specifically to tell whether the bowl contains food and how similar its current state is to a previous one. That's the only place our product comes into this post; the rest is a standalone look at how similarity search systems work.

The problem space

Traditional image classification assumes you already know the categories. A lot of real tasks don't fit that mold: you want to know how similar two images are, not which bucket they belong to; objects vary subtly in texture, color, or how much of them is visible; and lighting, angle, and occlusion keep changing. Similarity search handles these cases by comparing images in a continuous space instead of forcing them into discrete labels.

Why image similarity

Image similarity works by embedding images into a metric space, where closer embedding vectors mean more visually similar images. That gets you a few advantages: it holds up under lighting and orientation changes, it doesn't need strict class definitions, it scales easily since you just add more reference images, and it's efficient enough to run on mobile or edge hardware.

Diagram of images mapped into an embedding space with similar images clustered together
Diagram of images mapped into an embedding space with similar images clustered together

How image similarity systems work

A clean similarity search pipeline generally has four steps.

Preprocessing: before comparing anything, resize images to a standard shape (224x224 is common), normalize pixel values, and optionally crop, denoise, or augment. Consistent preprocessing is what makes similarity scores trustworthy.

Embedding generation: a model turns each image into a compact vector, often 128 to 1024 dimensions, using an architecture like MobileNetV3, EfficientNet-Lite, a CLIP ViT encoder, or a ResNet-based encoder. These vectors capture shape, color distribution, and texture.

Similarity computation: once you have embeddings, you compare them with a chosen distance metric.

Ranking and decision: the system ranks gallery images by similarity, or applies a threshold to decide whether two images are close enough to count as a match. This same architecture powers reverse image search, duplicate detection, visual recommendation, lightweight object recognition, and state comparison workflows.

Pipeline diagram showing preprocessing, embedding, similarity computation, and ranking stages
Pipeline diagram showing preprocessing, embedding, similarity computation, and ranking stages

Similarity metrics: how closeness is measured

Not every distance metric behaves the same way, and the choice affects performance a lot. Here are the ones that show up most often in production.

Cosine similarity, the most widely used: it measures the angle between two vectors.

cosine(A, B) = (A · B) / (‖A‖ ‖B‖)

It's insensitive to overall brightness, stable across small variations, and works well with the direction-rich embeddings most neural encoders produce, which is why it's the default choice for high-dimensional image embeddings.

Euclidean distance (L2) measures the straight-line distance between two vectors.

d(A, B) = sqrt( Σ (A_i - B_i)^2 )

It suits shape-driven images and embeddings that are already L2-normalized, but it's sensitive to magnitude differences if you skip normalization.

Manhattan distance (L1) sums the absolute differences between vectors. It's more robust to noise than Euclidean and works well on datasets with high variance or mixed textures, often paired with cosine for tie-breaking.

Structural Similarity Index (SSIM) compares perceptual quality rather than raw pixel differences, evaluating contrast, luminance, and structural pattern. It suits comparing near-identical images or catching distortion, compression artifacts, and tampering.

Learned similarity (metric learning) trains a model directly to understand similarity instead of classification, using approaches like triplet loss, contrastive loss, or ArcFace and CosFace style margin losses. It learns task-specific similarity and creates tight clusters even for subtly different classes, which is ideal when the domain images vary a lot. The drawback is that it needs curated positive and negative pairs and a substantial training set.

Comparison of cosine, Euclidean, Manhattan, SSIM, and learned similarity metrics
Comparison of cosine, Euclidean, Manhattan, SSIM, and learned similarity metrics

Visual search and retrieval, searching by example the way Google Lens or e-commerce "search by photo" does. Duplicate and near-duplicate detection, useful for content moderation, dataset cleaning, or cloud storage. Quality inspection in manufacturing, spotting deviations and subtle pattern changes. Medical imaging, comparing pathology samples, X-rays, or scans against known references. Lightweight recognition workflows, comparing against a reference gallery instead of training a classifier for every category. And state comparison systems: a small part of our work at Hoomanely uses this idea to compare bowl states, and similar techniques show up widely in robotics and autonomous systems.

Advantages and limitations

On the plus side, similarity search is flexible across domains, works even with unlabeled data, expands easily, runs fine on lightweight models, and strikes a good balance between accuracy and compute. On the other hand, it needs a diverse reference gallery, its quality depends heavily on the embedding model, similarity decisions can be hard to explain in some cases, and it's sensitive to major viewpoint changes unless it's been trained for them.

Best practices for building robust similarity systems

  • Maintain a high-quality gallery. Diverse samples make the system more robust.
  • Normalize embeddings. L2-normalize unless you're using cosine directly.
  • Use multiple metrics. Cosine plus Euclidean often catches edge cases the other misses.
  • Threshold carefully. Dynamic thresholds adapt to lighting and noise better than a fixed cutoff.
  • Benchmark regularly. Track recall@k, precision@k, and false positives over time.

Why similarity search matters

Image similarity search is one of the more practical ideas in computer vision. Instead of forcing images into rigid classes, it builds a continuous understanding of visuals that lets systems work across environments, categories, and subtle variation. It shows up in search engines, industrial inspection, and comparing real-world physical states, and it's a foundational piece of most modern vision pipelines. A small part of Hoomanely's own work draws on this idea, but the technique itself is far broader than any one product: a versatile, scalable approach to visual understanding.

Key takeaways

  • Image similarity search maps images into a vector space and measures closeness.
  • Cosine similarity is the most stable and widely used metric.
  • Euclidean, Manhattan, SSIM, and learned metrics each bring their own strengths.
  • Similarity search powers everything from search engines to industrial and medical systems.
  • A strong gallery and a good embedding model are the core of a system that actually works.

Read more