Yocto Recipes That Actually Compile: Lessons from Porting a Custom Application to a Minimal Image

The build failed at 11 PM on a Thursday. Not from our code, from a missing pkg-config path. The CMake configure step for Mosquitto was resolving openssl.pc from /usr/lib/x86_64-linux-gnu instead of from the target sysroot. We'd seen this class of error before. Fix it, rebuild, wait forty minutes for bitbake to grind through its dependency graph, and something else would break, a different recipe, a different missing symbol, an equally maddening reason.

This was the early phase of porting our sensor hub daemon to a Yocto-built minimal image, the same project where we eventually cut our OS footprint from 3.3 GB to 150 MB. That number gets attention when we talk about it. What doesn't get attention is what came before it: a month of recipe failures, missing libraries at runtime, and rebuild cycles so slow they made iteration feel impossible. This post is about those failures and the three techniques that finally made the workflow manageable.

Problem

Yocto isn't a package manager, it's a complete cross-compilation framework. Writing a recipe for your own application means describing not just where the source lives but how it should be configured, compiled, staged, and installed into a sysroot that's structurally different from your host machine. When things go wrong, the error messages often point at paths that have no business being involved in a cross-build, because at some layer the build system leaked back to the host environment.

Our application was the sensor hub daemon described in earlier posts, the C++ process that runs the BLE bridge, manages the two-tier sensor queue, and forwards frames over CAN to the Linux host. On a full Raspbian image it just worked: Mosquitto, our ring buffer library, libgpiod, libserialport, all present. On our stripped Yocto image with a custom machine config, none of them were. Writing recipes for each exposed a different category of failure.

The three categories we kept hitting were pkg-config contamination from the host environment, RDEPENDS versus DEPENDS confusion that left libraries missing at runtime even when the build succeeded, and iteration speed, since a full bitbake rebuild cycle took 30 to 50 minutes per attempt. All three are solvable, but none of them are obvious from the documentation, and none of them announced themselves clearly in the error message.

Approach

The pkg-config contamination problem. The first failure looked like this. Our Mosquitto recipe pulled in OpenSSL as a dependency. CMake's FindPkgConfig module ran during configure, called pkg-config --cflags openssl, and got back a path pointing at the host machine's OpenSSL headers. The build succeeded, because the headers were structurally compatible, but the binary got linked against the host's libssl.so path, which didn't exist on the target. The device would boot, the daemon would start, and then it would immediately segfault on the SSL handshake.

# The symptom: pkg-config resolving to host paths during cross-compile
$ bitbake -c compile mosquitto 2>&1 | grep "pkg-config"
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.8.1")
-- Checking for module 'openssl'
--   Found openssl, version 3.0.2
-- OpenSSL include dir: /usr/include/openssl   <- HOST path, not sysroot

The fix is to never let pkg-config see the host. Yocto's cross-compilation environment should handle this automatically, but CMake's FindPkgConfig module bypasses the Yocto-injected wrapper if the recipe doesn't explicitly pass the right flags. The correct pattern sets PKG_CONFIG_PATH to point only at the sysroot and exports PKG_CONFIG_SYSROOT_DIR so any path the tool returns gets automatically prefixed with the sysroot root:

# In your recipe (.bb file)
# recipe: mosquitto_2.0.18.bb

inherit cmake pkgconfig

# Tell CMake to use the cross-compile pkg-config wrapper
OECMAKE_C_FLAGS += "-I${STAGING_INCDIR}"
OECMAKE_LINK_FLAGS += "-L${STAGING_LIBDIR}"

# Export sysroot-aware pkg-config environment
export PKG_CONFIG_PATH = "${STAGING_LIBDIR}/pkgconfig:${STAGING_DATADIR}/pkgconfig"
export PKG_CONFIG_SYSROOT_DIR = "${STAGING_DIR_TARGET}"

DEPENDS += "openssl"

The rule we settled on: every recipe calling CMake or autoconf with a native library dependency needs an explicit PKG_CONFIG_SYSROOT_DIR export. Don't assume the inheritance chain handles it. Check the configure log for any pkg-config line resolving to a path not under ${STAGING_DIR_TARGET} and treat it as a bug.

RDEPENDS vs. DEPENDS: the failure that lets builds succeed. The second category is subtler because the build finishes cleanly. DEPENDS declares a compile-time dependency, something that must be built and staged before this recipe runs. RDEPENDS declares a runtime dependency, something that must be present on the target image when this recipe's output gets installed. Mixing them up produces a build that works perfectly and a device that fails at runtime.

We hit this with libgpiod. Our recipe listed it under DEPENDS, correctly, since we needed its headers and static library during the build, but left it out of RDEPENDS, incorrectly, since we also needed its shared object on the target at runtime. The build produced a clean binary. bitbake world finished without warnings. The image got generated and flashed. The daemon started, tried to open /dev/gpiochip0 through libgpiod, and failed with:

sensor-hub: error while loading shared libraries: libgpiod.so.2:
cannot open shared object file: No such file or directory

The binary was linked. The library was staged in the sysroot during the build. It just never made it into the final rootfs, because nothing told Yocto the image needed it at runtime. The fix, once understood, was straightforward:

# sensor-hub.bb — correct DEPENDS and RDEPENDS

DEPENDS += "libgpiod libserialport mosquitto jansson"

# Runtime dependencies: everything the binary links against dynamically
RDEPENDS:${PN} += "libgpiod libserialport mosquitto jansson"

# For libraries we only need at build time (static linking),
# DEPENDS alone is correct — RDEPENDS is not needed for those

The pattern that caught most of our RDEPENDS omissions: after flashing a new image, run ldd on every binary from your custom recipes and check each listed .so against what's actually in /usr/lib on the target. Any .so not found is a missing RDEPENDS entry. That check became part of our image validation step before any real hardware testing.

Iteration speed: devshell and externalsrc. The third problem was the feedback loop. A full bitbake rebuild to test a recipe change took 30 to 50 minutes on the machines we used for embedded builds, most of it rebuilding dependencies that hadn't changed. The sstate-cache helps a lot for CI, but even with a warm cache, waiting on bitbake to re-run tasks for every small recipe edit was slow enough to make debugging feel adversarial.

Two tools fixed this. The first is devshell. Running bitbake -c devshell your-recipe drops you into a shell inside the exact cross-compilation environment Yocto would use, correct sysroot, correct cross-compiler on PATH, correct CFLAGS and LDFLAGS exported. From inside that shell you can run cmake, make, or your build system directly and iterate in seconds instead of minutes. This is how we debugged the pkg-config contamination: we opened a devshell for the Mosquitto recipe, ran cmake by hand, and watched exactly what pkg-config resolved to.

# Open a devshell for your recipe — iterates in seconds, not minutes
$ bitbake -c devshell sensor-hub

# You're now inside the cross-compilation environment:
$ echo $CC
aarch64-poky-linux-gcc --sysroot=/path/to/sysroot

$ echo $PKG_CONFIG_PATH
/path/to/sysroot/usr/lib/pkgconfig

# Run cmake manually to test pkg-config resolution:
$ cmake .. -DCMAKE_TOOLCHAIN_FILE=$TOOLCHAIN_FILE
-- Found PkgConfig: /usr/bin/aarch64-poky-linux-pkg-config  <- correct wrapper
-- Found openssl, version 3.0.7
-- OpenSSL include dir: /path/to/sysroot/usr/include  <- correct sysroot path

The second tool is externalsrc. Once you're done debugging in devshell and want to iterate on your own application's code without running the full bitbake fetch-unpack-patch cycle every time, externalsrc lets you point the recipe directly at your local source tree. Bitbake uses it as the source, skips the fetch step, and rebuilds only the compile and install tasks when files change.

# In your local.conf or a conf/local.conf override:
# Point the recipe at your live source tree
EXTERNALSRC:pn-sensor-hub = "/home/dev/hoomanely/sensor-hub/src"
EXTERNALSRC_BUILD:pn-sensor-hub = "/home/dev/hoomanely/sensor-hub/build-yocto"

# Now editing any source file and running:
#   bitbake -c compile sensor-hub && bitbake -c install sensor-hub
# skips fetch/unpack/patch and only recompiles what changed.
# A clean rebuild of just your application takes ~90 seconds
# instead of 45 minutes.

The combination changed the development model from "make a change, wait an hour to see if it compiles" to "edit in your normal IDE, run a 90-second bitbake compile, test on hardware." The sstate-cache underneath means the rest of the dependency tree stays untouched.

Process

The recipe structure we landed on. After working through the failures above, every recipe for our own application code and its first-party dependencies follows the same pattern, encoding all three lessons:

# sensor-hub.bb — full recipe structure after lessons learned

SUMMARY = "Hoomanely sensor hub daemon"
DESCRIPTION = "BLE bridge, two-tier sensor queue, CAN forwarder"
LICENSE = "CLOSED"

# Build-time deps: headers + static/import libs needed during compile
DEPENDS = "libgpiod libserialport mosquitto jansson openssl"

# Runtime deps: every .so the binary will dlopen or link against at runtime
RDEPENDS:${PN} = "libgpiod libserialport mosquitto jansson openssl"

inherit cmake

# Sysroot-aware pkg-config — never let host paths leak into the sysroot build
export PKG_CONFIG_PATH = "${STAGING_LIBDIR}/pkgconfig:${STAGING_DATADIR}/pkgconfig"
export PKG_CONFIG_SYSROOT_DIR = "${STAGING_DIR_TARGET}"

# Explicit sysroot for CMake so find_package() also resolves correctly
OECMAKE_C_FLAGS += "-I${STAGING_INCDIR}"
OECMAKE_LINK_FLAGS += "-L${STAGING_LIBDIR}"

do_install() {
    install -d ${D}${bindir}
    install -m 0755 ${B}/sensor-hub ${D}${bindir}/sensor-hub
    install -d ${D}${sysconfdir}/sensor-hub
    install -m 0644 ${S}/config/default.conf \
        ${D}${sysconfdir}/sensor-hub/sensor-hub.conf
}

FILES:${PN} += "${sysconfdir}/sensor-hub"

Making sstate-cache work across team members. One iteration-speed win paid off the moment we moved from a single developer working on the image to the full firmware team using it: a shared sstate-cache. That's Yocto's build artifact cache, and when a recipe task hasn't changed, bitbake can restore its output from cache instead of rerunning it. By default it lives on the local machine. Pointing it at a shared NFS mount or S3 bucket means any developer starts with a warm cache and a first build measured in minutes rather than hours.

# In site.conf (shared across all team members via version control):
SSTATE_MIRRORS = "file://.* https://sstate.hoomanely.internal/yocto/PATH;downloadfilename=PATH"

# Or for a local NFS share:
SSTATE_MIRRORS = "file://.* file:///mnt/yocto-cache/sstate/PATH"

# To populate the cache after a successful build (run on the build server):
# bitbake world  — builds everything; populates the cache
# Subsequent developers get most tasks as cache hits

Our build server runs a nightly bitbake world against a clean checkout. Every developer's first build after pulling the latest image layer changes hits the sstate-cache for everything except the recipes they've modified. The 45-minute first build dropped to under 8 minutes once the cache warmed up.

Results

After implementing the three fixes, correct PKG_CONFIG_SYSROOT_DIR exports, explicit RDEPENDS for every dynamically linked library, and externalsrc plus devshell for iteration, the recipe workflow changed measurably.

Before: a typical recipe debugging session took 3 to 4 hours. Write the recipe, run bitbake, wait 35 to 50 minutes for a build, find the error, fix one thing, rebuild. Three or four iterations to get a new recipe working from scratch.

After: the same session takes 45 minutes. devshell lets us verify CMake configuration in under 2 minutes. externalsrc cuts application rebuild cycles to about 90 seconds. The recipe structure above catches RDEPENDS issues before flashing, since we added a post-install check that runs ldd on all installed binaries and cross-references the .so list against the image manifest.

The final image size, 150 MB from a starting point of 3.3 GB, came downstream of this work. You can't aggressively strip an image if you don't have precise control over what each recipe actually pulls in. The recipe discipline we built to solve the compilation failures turned out to be the same discipline that made image minimization tractable.

Why it matters at Hoomanely

Our home hubs run a stripped Linux image because the alternative costs us in three ways: OTA update size, since a 3.3 GB image over a home broadband connection is a very different problem from a 150 MB one, flash storage on the device, and attack surface. The OTA post described how we built a dual-path firmware update system for the wireless SoC on the hub, and that whole effort gets easier to reason about when the Linux host image it lives on is minimal and under tight control.

The Yocto recipe discipline also connects to the reliability stories in our other hardware posts. The sleep-mode coordination described in the sleep modes post, and the reset-event observability in the reset architecture post, both require exact control over which kernel modules, udev rules, and systemd services are present on the image. A bloated image with uncontrolled package dependencies means you can't be sure which daemon is holding a GPIO line, which kernel module loaded a conflicting driver, or which init script changed a peripheral's default state at boot. A minimal image built from recipes you control removes that uncertainty.

It's also a maintenance discipline. Every package in a Yocto image is one someone has to keep building, keep patching for CVEs, and keep testing across kernel and toolchain upgrades. Fewer packages means fewer surprises eighteen months after the first hub ships.

Key takeaways

Host paths in a cross-build are always a bug. Any pkg-config resolution, CMake find_package call, or linker path that doesn't resolve under STAGING_DIR_TARGET is wrong. Export PKG_CONFIG_SYSROOT_DIR in every recipe with external library dependencies, and never assume the inheritance chain handles it.

RDEPENDS and DEPENDS aren't interchangeable. DEPENDS is for the build, RDEPENDS is for the runtime image. If a library is dynamically linked, it needs RDEPENDS. Run ldd on your installed binaries and compare against the image rootfs as part of build validation, don't find out on the device.

devshell is the right debugging environment for recipe failures. It gives you the exact cross-compilation shell bitbake uses, in seconds, without a full rebuild. If your recipe is broken, open a devshell before touching the .bb file.

externalsrc changes the development model. A bitbake world rebuild is for CI. For active development on your own application code, externalsrc plus a targeted bitbake -c compile cuts iteration time from 40 minutes to 90 seconds. Use it from the start of a porting effort, not after you've already suffered through ten full rebuilds.

A shared sstate-cache is team infrastructure, not a personal optimization. Set it up before a second person joins the image development work. A warm cache means the first build is fast; without it, every new developer runs the full multi-hour cold build.

Recipe discipline is what enables image minimization. You can't confidently strip a package from an image if you don't know why it's there. The same attention to DEPENDS and RDEPENDS that fixes runtime failures is what makes it safe to remove things.

Author's note

The first version of this porting effort had none of the structure above. The recipe for our application was a copy-pasted template with the wrong DEPENDS, no PKG_CONFIG_SYSROOT_DIR export, and no understanding of how Yocto's staging directory differed from a normal sysroot. It built. It ran on the bench. It broke on the target in ways that took days to trace back to the recipe.

The right framing isn't "write the recipe and see if bitbake succeeds." bitbake succeeding tells you the build was structurally valid. It doesn't tell you the resulting binary will run on the target, that every shared library it needs is in the image, or that the pkg-config paths it resolved during the build came from the sysroot rather than the host. Those are three separate things to verify, and none of them happen automatically.