Monocular Depth Estimation: Seeing the Third Dimension Through a Single Eye

Intro

Look at a photograph and your brain instantly works out what's near and what's far, even though the image itself is flat. Machines don't have that instinct built in. Monocular depth estimation teaches a computer to infer depth from a single camera, no stereo pair, no LiDAR. This post covers how it works, why transformers changed the field, and how it runs efficiently on small edge devices.

Why depth matters

Depth sits at the center of perception. Robots use it to navigate, AR systems use it to place objects correctly, self-driving cars rely on it for safety. At Hoomanely, depth helps our smart bowls and cameras read a dog's posture, movement, and interaction without needing multiple sensors. Depth traditionally came from stereo cameras or LiDAR, both dependent on geometry or active sensing. Monocular estimation gets similar results by learning depth cues from a single image instead.

How one image encodes depth

A single image carries subtle depth hints: perspective, where distant lines appear to converge; relative size, where familiar objects look smaller the farther away they are; and texture and blur, where fine detail fades with distance. These are the same cues your brain uses looking at a photo, and in computer vision they get captured mathematically.

The perspective relation comes from the pinhole camera model:

h_i / H_i = f / Z_i

An object's apparent height on the image (h_i) is proportional to its true height (H_i) and the camera's focal length (f), and inversely proportional to its distance from the camera (Z_i). In plainer terms, the farther something is, the smaller it looks. Rearranged:

Z_i = f * (H_i / h_i)

So if we know the focal length and an object's true size, we can work out its distance. In real scenes we rarely know every object's true size, which is where machine learning comes in. The network learns a function F mapping each pixel's intensity to a depth value:

F: I(x, y) -> D(x, y)

Instead of relying on exact geometry, it learns from data, seeing many images with known depths and absorbing those relationships over time.

From CNNs to transformers

Early models used CNNs to infer depth locally, scanning small patches and piecing together a larger scene. They captured fine detail well but often failed to connect distant parts of an image, like how the horizon relates to nearby objects.

Self-supervised training improved on this by learning depth through reconstructing one camera view from another:

L_photo = | I_L - I_R^warp(D) |

This photometric reconstruction loss predicts depth D, warps one view (I_R) to match the other (I_L), and minimizes the difference. If the reconstruction looks right, the depth prediction is probably accurate, and this lets a model learn depth without any labeled data.

Then transformers changed the picture. Instead of processing small patches in isolation, they reason globally through self-attention:

Attention(Q, K, V) = softmax((QK^T) / sqrt(d_k))V

Each pixel, treated as a query Q, interacts with every other pixel (keys K and values V), learning how strongly they influence one another. That lets the network connect distant regions, for instance how the sky's brightness might affect ground shading. Transformers also pick up object size priors, recognizing that people, cars, or furniture tend to have certain sizes and using that to infer relative distance, though this can occasionally confuse scale, mistaking a toy car for a distant real one. Overall, combining global reasoning with semantic understanding gives transformers smoother, more coherent depth maps.

Training and deployment

Depth models train on datasets with known ground-truth depth per pixel, such as NYU Depth V2 for indoor scenes and KITTI for outdoor driving. The training objective balances accuracy and smoothness:

L_total = Mean(|D - D*|) + λ * |∇D|

The first term is a pixel-wise loss keeping each predicted depth close to ground truth, and the second is a smoothness loss encouraging neighboring pixels to have similar depths so flat surfaces stay smooth. The coefficient λ balances the two, which together produces depth maps that are both accurate and visually natural.

On edge devices, efficiency matters as much as accuracy, and a few techniques make that possible: distillation, where a smaller student model learns from a larger, accurate teacher; quantization, reducing 32-bit weights to 8-bit with little accuracy loss; and pruning, removing redundant connections to shrink the model. These optimizations let a model like MiDaS infer depth at around 15 frames per second on a Raspberry Pi, effectively turning a plain camera into a 3D sensor.

Limitations and what's next

Monocular depth estimation isn't perfect. Scale ambiguity means the model struggles with absolute distance, since a small object up close can look like a big object far away. Reflective and transparent surfaces confuse it because reflections distort the light cues it relies on. And in video, predictions can flicker between frames, a form of temporal inconsistency.

The next wave of research pairs monocular depth with Neural Radiance Fields and diffusion-based 3D reconstruction, bridging 2D learning and 3D geometry. These hybrid methods combine physical consistency with learned priors, moving depth estimation closer to how people actually perceive space.

Key takeaways

• Monocular depth estimation pulls 3D understanding out of a single image.
• Transformers outperform CNNs by reasoning globally and learning object priors.
• Edge optimizations like distillation and quantization make real-time use possible.
• Future models will likely merge semantic and geometric depth reasoning for better accuracy.