Proof of Concept: Porting Nomad to NetBSD

Background and motivation

HashiCorp Nomad is a lightweight orchestrator based on Remarkably easy way to manage a wide variety of workloads under one shared interface - from the classic process Containers to virtual machines. His strength lies in Modularity: thanks to different drivers (exec, raw_exec, qemu, docker and many more m.) very different environments and Operate technologies under a unified scheduler.

At its core, Nomad follows a clear and compact architectural principle: An agent program runs on every machine. In Server Mode it handles planning, cluster management and API access, in Client mode it executes tasks. The server nodes form together a Raft-based consensus cluster that manages the state replicates and makes scheduling decisions - without one A large number of separate control components are required.

This means that Nomad fulfills the essential control functions of a Control Plane, but comes without a classic, multi-layered one Control plane architecture like Kubernetes. The clients are working Largely autonomous, running tasks are carried out even during short periods of time Server unavailability further and thus reduce systemic Dependencies.

This design makes Nomad particularly attractive for critical or distributed environments where stability, visibility and Determinism is more important than maximum feature density.

Comparison: Kubernetes vs. Nomad (control plane architecture)

Why NetBSD?

The idea of ​​this proof of concept came from the combination of two values: Nomad’s simplicity and NetBSD’s clarity. NetBSD has stood for portability, consistency and transparency for decades. It’s a system that you understand - right down to the kernel. No unnecessary ballast, no non-transparent automatism.

This is exactly what makes it an exciting basis for Nomad:

• high portability and clean system design
• stable and predictable runtime environment
• easy to understand structure

The following drivers are particularly interesting for the port:

• exec – classic process model
• raw_exec – direct process start without additional isolation
• qemu – Virtualization as a workload
• perspective: a dedicated xen driver to enable native hypervisor integration

Nomad and NetBSD share the same core architectural idea: clarity over complexity.

The path to the prototype

Nomad is written in Go - a stroke of luck, as Go has excellent support on NetBSD. The ecosystem works smoothly, pkgsrc delivers up-to-date compilers and libraries, and Go’s cross-platform idea harmonizes with NetBSD’s design philosophy.

Step 1: Preparation

As is usual with third-party software on NetBSD, the route via pkgsrc would be ideal. However, since there is no port for Nomad there (yet), we start manually:

git clone https://github.com/hashicorp/nomad.git
pkgin install gmake go124

NetBSD comes with BSD Make out of the box, but Nomad relies on a GNU Makefile - so we install gmake and start the first attempt.

First build attempt

A gmake release starts the build, but the build aborts with an error message. Cause: the Makefile does not (yet) recognize NetBSD as a platform.

A quick look at GNUmakefile shows that the variable THIS_OS is being checked. A small patch is enough:

ifeq (NetBSD,$(THIS_OS))
ALL_TARGETS = netbsd_amd64
endif

Another attempt runs a little further - this time the build fails due to a Go dependency: github.com/mitchellh/go-ps.

The error:

undefined: processes
undefined: findProcess

This means: The library has no implementation for NetBSD.

Investigation of the dependency

A look at the repository shows: go-ps has been archived since 2024, the last commit was five years ago. Nomad apparently uses it for simple process queries - stable, but old. We fork the project at github.com/MatthiasPetermann/go-ps and start customizing.

Goal:

A NetBSD implementation that uses /proc - analogous to the existing process_unix.go that is active for Linux and Solaris.

To do this, it is sufficient to first adjust the build tag:

// +build linux solaris netbsd

When compiling, the message appears that UnixProcess.Refresh() is missing - the method that parses /proc/<pid>/stat. So we copy the Linux version (process_linux.go), create it as process_netbsd.go and adapt the parser.

Digression: /proc/<pid>/stat on NetBSD

A comparison shows:

NetBSD:

1016 (xinit) S 856 1016 850 ...

Linux:

1042 (systemd-logind) S 1 1042 1042 ...

The first fields (pid, comm, state, ppid, pgrp, sid) are identical - exactly the ones the parser uses. This means that the function can be adopted almost unchanged.

The resulting code:

// +build netbsd

package ps

import (
    "fmt"
    "io/ioutil"
    "strings"
)

func (p *UnixProcess) Refresh() error {
    statPath := fmt.Sprintf("/proc/%d/stat", p.pid)
    dataBytes, err := ioutil.ReadFile(statPath)
    if err != nil {
        return err
    }

    data := string(dataBytes)
    binStart := strings.IndexRune(data, '(') + 1
    binEnd := strings.IndexRune(data[binStart:], ')')
    p.binary = data[binStart : binStart+binEnd]

    data = data[binStart+binEnd+2:]
    _, err = fmt.Sscanf(data, "%c %d %d %d", &p.state, &p.ppid, &p.pgrp, &p.sid)
    return err
}

This means that go-ps is expanded to include NetBSD.

Integration with Nomad

So that Nomad uses our fork instead of the original library, we add the Replace block in go.mod:

replace (
    ...
    ...
    github.com/mitchellh/go-ps => github.com/MatthiasPetermann/go-ps v1.1.1
)

Then:

go mod tidy
gmake release

And lo and behold:

==> Building pkg/netbsd_amd64/nomad with tags ui codegen_generated release...
==> Packaging for netbsd_amd64...
  adding: LICENSE.txt (deflated 62%)
  adding: nomad (deflated 60%)
==> Results:
/h/mpeterma/Projects/hashcorp/nomad/pkg
├── netbsd_amd64
│   ├── LICENSE.txt
│   └── nomad
└── netbsd_amd64.zip

1 directory, 3 files

A complete, compressed release package for NetBSD - built with no obvious errors and without deep modifications to Nomad’s core.

The conclusion is a test as to whether the generated binary is executable:

mkdir nomad
cd nomad
unzip ~/Projects/hashcorp/nomad/pkg/netbsd_amd64.zip
./nomad
Usage: nomad [-version] [-help] [-autocomplete-(un)install] <command> [args]

Common commands:
    run         Run a new job or update an existing job
    stop        Stop a running job
    status      Display the status output for a resource
    alloc       Interact with allocations
    job         Interact with jobs
    node        Interact with nodes
    agent       Runs a Nomad agent

Other commands:
    acl                 Interact with ACL policies and tokens
    action              Run a pre-defined command from a given context
    agent-info          Display status information about the local agent
    config              Interact with configurations
...

Conclusion

Conclusion: The proof of concept shows that Nomad can be successfully built on NetBSD with a few targeted adjustments. The build process is stable and the resulting release package for netbsd_amd64 contains a startable binary.

The biggest hurdle was a lack of platform support in the go-ps library, which was solved by a custom NetBSD implementation. With this addition and a small extension in the Makefile, Nomad correctly recognizes NetBSD as the target platform and compiles without any further intervention.

This lays the foundation for practical testing of Nomad under NetBSD. In an upcoming article, the first runtime tests, functional tests of the task drivers (raw_exec, exec, qemu) and an assessment of how Nomad behaves in the NetBSD operating system environment will follow.

Image credit: The Nomad logo is a registered trademark of HashiCorp, Inc.. Use in editorial reporting in accordance with HashiCorp Trademark and Brand Guidelines. The NetBSD logo is a registered trademark of the NetBSD Foundation, Inc. Use in the context of editorial reporting in accordance with the Logo Use Guidelines.