Nomad on Alpine Linux with MUSL: lightweight but not trivial

Introduction

Nomad by HashiCorp is a lightweight orchestration and scheduling tool for containers, virtual machines, and simple binaries. Unlike Kubernetes, Nomad does not require extensive infrastructure. It runs smoothly on a single node and impresses with its simplicity - especially in smaller setups, home labs or edge environments.

At its core, Nomad consists of a single binary that can act as both a server and a client. This makes it ideal for compact systems – such as an Alpine Linux server.

Alpine Linux & MUSL – minimalism with pitfalls

Alpine Linux stands for minimalism. The distribution is based on:

• musl libc instead of glibc
• busybox instead of GNU Coreutils

The result: an extremely slim, fast and secure system – ideal for container environments and resource-poor hardware.

But this minimalism has side effects. Many applications built against glibc will not work easily with musl. It becomes particularly problematic if you compile on a glibc-based system and then run the binary on Alpine.

The problem: built on Xubuntu, failed on Alpine

This is exactly the trap I fell into.

I wanted to build Nomad from source on my Xubuntu workstation (glibc) - and then run it on my Alpine-based home server.

The first attempt

git clone https://github.com/hashicorp/nomad.git
cd nomad
make release TARGETS=linux_amd64

The build went through without errors. The finished binary was in the pkg/linux_amd64/ directory:

pkg/linux_amd64/nomad

But after copying it to the Alpine server, disillusionment followed:

./nomad
-sh: ./nomad: not found

The file existed but was not executable. A classic indication of a linking problem.

Diagnosis with ldd

A short test brought certainty:

ldd nomad

Result:

        /lib64/ld-linux-x86-64.so.2 (0x7fa9060cf000)
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fa9060cf000)
Error relocating nomad: __snprintf_chk: symbol not found
Error relocating nomad: __vfprintf_chk: symbol not found
Error relocating nomad: __pthread_unregister_cancel: symbol not found
Error relocating nomad: __register_atfork: symbol not found
Error relocating nomad: __asprintf_chk: symbol not found
Error relocating nomad: __vdprintf_chk: symbol not found
Error relocating nomad: __isoc23_strtol: symbol not found
Error relocating nomad: __vasprintf_chk: symbol not found
Error relocating nomad: __vsnprintf_chk: symbol not found
Error relocating nomad: __strncpy_chk: symbol not found
Error relocating nomad: __longjmp_chk: symbol not found
Error relocating nomad: __dprintf_chk: symbol not found
Error relocating nomad: __fprintf_chk: symbol not found
Error relocating nomad: __pthread_register_cancel: symbol not found

The output clearly showed: The binary was linked against glibc - and musl simply couldn’t provide the required symbols.

The solution: Build directly under Alpine

The most elegant approach: Build Nomad natively with musl - i.e. directly in an Alpine environment.

A compact container that contains all the necessary tools is sufficient. The following container file lays the foundation:

# syntax=docker/dockerfile:1
FROM alpine:3.22 AS builder

# --- Build dependencies ---
RUN apk add --no-cache \
    bash \
    build-base \
    git \
    curl \
    ca-certificates \
    make \
    unzip \
    tar \
    linux-headers \
    zip

# --- Install Go ---
ENV GOLANG_VERSION=1.25.3
RUN curl -fsSL https://go.dev/dl/go${GOLANG_VERSION}.linux-amd64.tar.gz | tar -C /usr/local -xz
ENV PATH="/usr/local/go/bin:${PATH}"

# --- Build environment ---
WORKDIR /src
COPY . .

ENV CGO_ENABLED=0 \
    GOOS=linux \
    GOARCH=amd64 \
    GOPATH=/root/go \
    PATH="/root/go/bin:/usr/local/go/bin:${PATH}"

# --- Build Nomad ---
RUN make release TARGETS=linux_amd64

# --- Post-build export step ---
# Wenn /out gemountet ist, wird das Ergebnis direkt dort abgelegt
RUN mkdir -p /out && \
    cp pkg/linux_amd64/nomad /out/nomad && \
    cp pkg/linux_amd64.zip /out/linux_amd64.zip

# --- Runtime behavior: export results and exit ---
CMD ["/bin/sh", "-c", "echo '✅ Build complete. Files are in /out'; ls -lh /out"]

Compile and export

mkdir dist
podman build -v "$(pwd)/dist:/out" -t nomad-musl-builder -f Containerfile

After successful build the files are ready:

ls dist
linux_amd64.zip  nomad

A quick check on the target system confirms success:

ldd nomad

Output:

        /lib/ld-musl-x86_64.so.1 (0x7fc7a84ff000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fc7a84ff000)

The binary is now musl-native – and runs without any problems.

Test run

./nomad agent -dev -bind=0.0.0.0

In the browser under http://<ip-des-servers>:4646 The usual Nomad interface appears - this time without error messages.

Conclusion

Alpine Linux is an excellent foundation for lean systems - but its musl-based architecture demands attention when building software. Nomad itself is written in Go and can therefore be compiled natively for musl with little effort.

A dedicated container build ensures reproducible results and avoids typical pitfalls.

Finding: If you want to deploy on Alpine, you should build there too.

“Minimalism only works if you understand what’s missing.”

Image credit: The Nomad logo is a registered trademark of HashiCorp, Inc. Use in editorial reporting in accordance with HashiCorp Trademark Guidelines. The Alpine Linux logo is the property of the Alpine Linux Development Team. Use within the context of editorial reporting. Tux, the Linux mascot, was created by Larry Ewing (using GIMP, 1996). Use within the context of editorial reporting.