Yocto Recipes That Actually Compile: Lessons from Porting a Custom Application to a Minimal Image

The build failed at 11 PM on a Thursday. Not from our code — from a missing pkg-config path. The CMake configure step for Mosquitto was resolving openssl.pc from /usr/lib/x86_64-linux-gnu instead of from the target sysroot. We'd seen this class of error before. Fix it, rebuild, wait forty minutes for bitbake to grind through its dependency graph, and something else would break — a different recipe, a different missing symbol, a different equally-maddening reason.

This was the early phase of porting our sensor hub daemon to a Yocto-built minimal image — the same project where we eventually cut our OS footprint from 3.3 GB to 150 MB. That number gets attention when we talk about it. What doesn't get attention is what came before it: the month of recipe failures, missing libraries at runtime, and rebuild cycles so slow they made iteration feel impossible. This post is about those failures, and the three techniques that finally made the workflow manageable.

Problem

Yocto is not a package manager. It is a complete cross-compilation framework. When you write a recipe for your own application, you are describing not just where the source lives but how it should be configured, compiled, staged, and installed into a sysroot that is structurally different from your host machine. The error messages, when things go wrong, often point at paths that have no business being involved in a cross-build — because at some layer, the build system leaked back to the host environment.

Our application was the sensor hub daemon described in earlier posts — the same C++ process that runs the BLE bridge, manages the two-tier sensor queue, and forwards frames over CAN to the Linux host. On a full Raspbian image it just worked: Mosquitto, our ring buffer library, libgpiod, libserialport, all present. On our stripped Yocto image with a custom machine config, none of them were. Writing recipes for each exposed a different category of failure.

The three categories we kept hitting were: pkg-config contamination from the host environment, RDEPENDS vs DEPENDS confusion causing libraries to be missing at runtime even when the build succeeded, and iteration speed — a full bitbake rebuild cycle taking 30–50 minutes per attempt. All three are solvable. None of them are obvious from the documentation, and none of them announced themselves clearly in the error message.

Approach

The pkg-config contamination problem

The first failure looked like this. Our Mosquitto recipe pulled in OpenSSL as a dependency. CMake's FindPkgConfig module ran during the configure step, called pkg-config --cflags openssl, and got back a path that pointed at the host machine's OpenSSL headers. The build succeeded — because the headers were structurally compatible — but the binary was linked against the host's libssl.so path, which didn't exist on the target. The device would boot, the daemon would start, and then immediately segfault on the SSL handshake.

# The symptom: pkg-config resolving to host paths during cross-compile
$ bitbake -c compile mosquitto 2>&1 | grep "pkg-config"
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.8.1")
-- Checking for module 'openssl'
--   Found openssl, version 3.0.2
-- OpenSSL include dir: /usr/include/openssl   ← HOST path, not sysroot

The fix is to never let pkg-config see the host. Yocto's cross-compilation environment should be handling this automatically, but CMake's FindPkgConfig module bypasses the Yocto-injected wrapper if the recipe doesn't explicitly pass the right flags. The correct recipe pattern sets PKG_CONFIG_PATH to point only at the sysroot and exports PKG_CONFIG_SYSROOT_DIR so that any path the tool returns is automatically prefixed with the sysroot root:

# In your recipe (.bb file)
# recipe: mosquitto_2.0.18.bb

inherit cmake pkgconfig

# Tell CMake to use the cross-compile pkg-config wrapper
OECMAKE_C_FLAGS += "-I${STAGING_INCDIR}"
OECMAKE_LINK_FLAGS += "-L${STAGING_LIBDIR}"

# Export sysroot-aware pkg-config environment
export PKG_CONFIG_PATH = "${STAGING_LIBDIR}/pkgconfig:${STAGING_DATADIR}/pkgconfig"
export PKG_CONFIG_SYSROOT_DIR = "${STAGING_DIR_TARGET}"

DEPENDS += "openssl"

The rule we settled on after this: every recipe that calls CMake or autoconf and has any native library dependency needs explicit PKG_CONFIG_SYSROOT_DIR export. Don't assume the inheritance chain handles it. Check the configure log for any pkg-config line that resolves to a path not under ${STAGING_DIR_TARGET} and treat it as a bug.

RDEPENDS vs DEPENDS: the failure that builds succeed

The second category of failure is subtler and takes longer to find because the build completes successfully. DEPENDS in a Yocto recipe declares a compile-time dependency — a package that must be built and staged before this recipe runs. RDEPENDS declares a runtime dependency — a package that must be present on the target image when this recipe's output is installed. Confusing them causes the build to work perfectly and the device to fail at runtime.

We hit this with libgpiod. Our recipe listed it under DEPENDS (correct — we need its headers and .a during the build) but omitted it from RDEPENDS (wrong — we also need its .so on the target at runtime). The build produced a clean binary. bitbake world finished without warnings. The image was generated and flashed. The daemon started, tried to open /dev/gpiochip0 through libgpiod, and failed with:

sensor-hub: error while loading shared libraries: libgpiod.so.2: 
cannot open shared object file: No such file or directory

The binary was linked. The library was staged in the sysroot during the build. But it never made it into the final rootfs because nothing told Yocto the image needed it at runtime. The fix was straightforward once understood:

# sensor-hub.bb — correct DEPENDS and RDEPENDS

DEPENDS += "libgpiod libserialport mosquitto jansson"

# Runtime dependencies: everything the binary links against dynamically
RDEPENDS:${PN} += "libgpiod libserialport mosquitto jansson"

# For libraries we only need at build time (static linking), 
# DEPENDS alone is correct — RDEPENDS is not needed for those

The pattern that caught most of our RDEPENDS omissions: after flashing a new image, run ldd on every binary from your custom recipes and check each listed .so against what's actually in /usr/lib on the target. Any .so not found = a missing RDEPENDS entry. We made this part of the image validation step before any real hardware testing.

Iteration speed: devshell and externalsrc

The third problem was the feedback loop. A full bitbake rebuild to test a recipe change takes 30 to 50 minutes on the machines we used for embedded builds. Most of that time is rebuilding dependencies that haven't changed. The sstate-cache helps significantly for CI, but even with a warm cache, waiting on bitbake to re-run tasks for every small recipe edit is slow enough to make debugging feel adversarial.

Two tools fixed this.

The first is devshell. Running bitbake -c devshell your-recipe drops you into a shell inside the exact cross-compilation environment Yocto would use — correct sysroot, correct cross-compiler on PATH, correct CFLAGS and LDFLAGS exported. From inside that shell you can run cmake, make, or your build system directly and iterate in seconds instead of minutes. This is how we debugged the pkg-config contamination: we opened a devshell for the Mosquitto recipe, ran cmake manually, and watched exactly what pkg-config resolved to.

# Open a devshell for your recipe — iterates in seconds, not minutes
$ bitbake -c devshell sensor-hub

# You're now inside the cross-compilation environment:
$ echo $CC
aarch64-poky-linux-gcc --sysroot=/path/to/sysroot

$ echo $PKG_CONFIG_PATH
/path/to/sysroot/usr/lib/pkgconfig

# Run cmake manually to test pkg-config resolution:
$ cmake .. -DCMAKE_TOOLCHAIN_FILE=$TOOLCHAIN_FILE
-- Found PkgConfig: /usr/bin/aarch64-poky-linux-pkg-config  ← correct wrapper
-- Found openssl, version 3.0.7
-- OpenSSL include dir: /path/to/sysroot/usr/include  ← correct sysroot path

The second tool is externalsrc. Once you're done debugging in devshell and want to iterate on your own application's code without running the full bitbake fetch-unpack-patch cycle every time, externalsrc lets you point the recipe directly at your local source tree. Bitbake will use it as the source, skip the fetch step, and rebuild only the compile and install tasks when you change files.

# In your local.conf or a conf/local.conf override:
# Point the recipe at your live source tree
EXTERNALSRC:pn-sensor-hub = "/home/dev/hoomanely/sensor-hub/src"
EXTERNALSRC_BUILD:pn-sensor-hub = "/home/dev/hoomanely/sensor-hub/build-yocto"

# Now editing any source file and running:
#   bitbake -c compile sensor-hub && bitbake -c install sensor-hub
# skips fetch/unpack/patch and only recompiles what changed.
# A clean rebuild of just your application takes ~90 seconds 
# instead of 45 minutes.

The combination of externalsrc and devshell changed the development workflow from 'make a change, wait an hour to see if it compiles' to 'edit in your normal IDE, run a 90-second bitbake compile, test on hardware.' The sstate-cache underneath means the rest of the dependency tree is untouched.

Process

The recipe structure we landed on

After working through the failures above, every recipe for our own application code and its first-party dependencies follows the same pattern. The structure is intentional — it encodes all three lessons above:

# sensor-hub.bb — full recipe structure after lessons learned

SUMMARY = "Hoomanely sensor hub daemon"
DESCRIPTION = "BLE bridge, two-tier sensor queue, CAN forwarder"
LICENSE = "CLOSED"

# Build-time deps: headers + static/import libs needed during compile
DEPENDS = "libgpiod libserialport mosquitto jansson openssl"

# Runtime deps: every .so the binary will dlopen or link against at runtime
RDEPENDS:${PN} = "libgpiod libserialport mosquitto jansson openssl"

inherit cmake

# Sysroot-aware pkg-config — never let host paths leak into the sysroot build
export PKG_CONFIG_PATH = "${STAGING_LIBDIR}/pkgconfig:${STAGING_DATADIR}/pkgconfig"
export PKG_CONFIG_SYSROOT_DIR = "${STAGING_DIR_TARGET}"

# Explicit sysroot for CMake so find_package() also resolves correctly
OECMAKE_C_FLAGS += "-I${STAGING_INCDIR}"
OECMAKE_LINK_FLAGS += "-L${STAGING_LIBDIR}"

do_install() {
    install -d ${D}${bindir}
    install -m 0755 ${B}/sensor-hub ${D}${bindir}/sensor-hub
    install -d ${D}${sysconfdir}/sensor-hub
    install -m 0644 ${S}/config/default.conf \
        ${D}${sysconfdir}/sensor-hub/sensor-hub.conf
}

FILES:${PN} += "${sysconfdir}/sensor-hub"

Making sstate-cache work across team members

One iteration-speed win that paid off when we moved from a single developer working on the image to the full firmware team using it: a shared sstate-cache. The sstate-cache is Yocto's build artifact cache — when a recipe task hasn't changed, bitbake can restore the output from cache instead of rerunning the task. By default it lives on the local machine. Pointing it at a shared NFS or S3 bucket means any developer on the team starts with a warm cache and a first build that takes minutes rather than hours.

# In site.conf (shared across all team members via version control):
SSTATE_MIRRORS = "file://.* https://sstate.hoomanely.internal/yocto/PATH;downloadfilename=PATH"

# Or for a local NFS share:
SSTATE_MIRRORS = "file://.* file:///mnt/yocto-cache/sstate/PATH"

# To populate the cache after a successful build (run on the build server):
# bitbake world  — builds everything; populates the cache
# Subsequent developers get most tasks as cache hits

Our build server runs a nightly bitbake world against a clean checkout. Every developer's first build after pulling the latest image layer changes hits the sstate-cache for everything except the recipes they've modified. The 45-minute first build dropped to under 8 minutes once the cache was warm.

Results

After implementing the three fixes — correct PKG_CONFIG_SYSROOT_DIR exports, explicit RDEPENDS for every dynamically linked library, and externalsrc plus devshell for iteration — the recipe workflow changed measurably.

Before: a typical recipe debugging session took 3 to 4 hours. Write the recipe, run bitbake, wait 35–50 minutes for a build, find the error, fix one thing, rebuild. Three to four iterations to get a new recipe working from scratch.

After: the same session takes 45 minutes. devshell lets us verify CMake configuration in under 2 minutes. externalsrc reduces application rebuild cycles to ~90 seconds. The recipe structure above catches RDEPENDS issues before flashing — we added a post-install check that runs ldd on all installed binaries and cross-references the .so list against the image manifest.

The final image size — 150 MB from a starting point of 3.3 GB — was downstream of this work. You cannot aggressively strip an image if you don't have precise control over what each recipe actually pulls in. The recipe discipline we built to solve the compilation failures turned out to be the same discipline that made the image minimization tractable.

Why It Matters at Hoomanely

Our home hubs run a stripped Linux image because the alternative costs us in three ways: OTA update size (a 3.3 GB image over a home broadband connection is a different problem than a 150 MB one), flash storage on the device, and attack surface. The OTA post described how we built a dual-path firmware update system for the wireless SoC on the hub; that whole effort becomes easier to reason about when the Linux host image it lives on is minimal and under tight control.

The Yocto recipe discipline also connects directly to the reliability stories in the other hardware posts. The sleep-mode coordination described in the sleep modes post, the reset event observability in the reset architecture post — both of those require exact control over which kernel modules, udev rules, and systemd services are present on the image. A bloated image with uncontrolled package dependencies means you cannot be sure which daemon is holding a GPIO line, which kernel module loaded a conflicting driver, or which init script changed a peripheral's default state at boot. A minimal image, built from recipes you control, removes that uncertainty.

It is also a maintenance discipline. Every package in a Yocto image is a package someone has to keep building, keep patching for CVEs, and keep testing across kernel and toolchain upgrades. Fewer packages = fewer surprises eighteen months after the first hub ships.

Key Takeaways

Host paths in a cross-build are always a bug. Any pkg-config resolution, CMake find_package call, or linker path that doesn't resolve under STAGING_DIR_TARGET is wrong. Export PKG_CONFIG_SYSROOT_DIR in every recipe that has external library dependencies and never assume the inheritance chain handles it.

RDEPENDS and DEPENDS are not interchangeable. DEPENDS is for the build. RDEPENDS is for the runtime image. If a library is dynamically linked, it needs RDEPENDS. Run ldd on your installed binaries and compare against the image rootfs as part of the build validation — don't find out on the device.

devshell is the correct debugging environment for recipe failures. It gives you the exact cross-compilation shell bitbake uses, in seconds, without a full rebuild. If your recipe is broken, open a devshell before touching the .bb file.

externalsrc changes the development model. A bitbake world rebuild is for CI. For active development on your own application code, externalsrc plus a targeted bitbake -c compile reduces iteration time from 40 minutes to 90 seconds. Use it from the start of a porting effort, not after you've already suffered through ten full rebuilds.

A shared sstate-cache is team infrastructure, not a personal optimization. Set it up before a second person joins the image development work. A warm cache means the first build is fast; without it, every new developer runs the full multi-hour cold build.

Recipe discipline enables image minimization. You cannot confidently strip a package from an image if you don't know why it's there. The same attention to DEPENDS and RDEPENDS that fixes runtime failures is what makes it safe to remove things.

Author's Note

The first version of this porting effort had none of the structure above. The recipe for our application was a copy-pasted template with the wrong DEPENDS, no PKG_CONFIG_SYSROOT_DIR export, and no understanding of how Yocto's staging directory differed from a normal sysroot. It built. It ran on the bench. It broke on the target in ways that took days to trace back to the recipe.

The right framing isn't 'write the recipe and see if bitbake succeeds.' bitbake succeeding tells you the build was structurally valid. It does not tell you the resulting binary will run on the target, that every shared library it needs is in the image, or that the pkg-config paths it resolved during the build were from the sysroot rather than the host. Those are three separate things to verify, and none of them are automatic.