Motivation
Distributed systems grow quickly – and with every new component complexity and the risk of uncontrolled interactions increase. This is particularly evident with messaging platforms like Apache Pulsar particularly noticeable when multiple locations via geo-replication are connected.
If a connection fails or a location gets out of sync, there is a risk one of the most critical situations in distributed systems: a split brain. Two cluster halves continue to accept write operations without each other to know – with potentially contradictory data and more complex Post-synchronization.
Pulsar Sentinel was born out of exactly this scenario: a small tool written in Go, the Pulsar Cluster monitored, detects quorum losses and targets them if necessary Self-fencing triggers – automatically, traceably and without Dependencies on external components.
The impulse for Pulsar Sentinel arose from a specific operational scenario, where geo-replication is not part of classic data distribution, but is used for 24/7 provision of a hot standby cluster.
In this architecture, message order, Request/Reply semantics and state consistency even in the event of a site failure or network segmentation is maintained.
This is where split brain conditions can have serious consequences – duplicate processing, conflicting events and incorrect recovery. Sentinel addresses exactly this point: targeted self-limitation, before serious inconsistencies arise.
Architecture and basic principle
Pulsar Sentinel follows a clear, deterministic structure:
- Communication via UDP heartbeats between nodes
- periodic HTTP health checks of the local broker
- Quorum monitoring by priority groups
- Fencing mechanism in case of quorum loss or split-brain risk
Each node periodically sends heartbeats with its status
(Healthy, Priority, Timestamp) and manages the incoming
Status information of the other nodes locally.
On this basis, the tool decides whether the cluster is still in balance.
Self-fencing refers to the controlled withdrawal of a node from the Cluster as soon as he realizes that his view of the overall system is not is more consistent.
The goal is not maximum availability at any price, but coherence and protection against divergence – for example, if two locations believe at the same time to be a “leader”.
Pulsar Sentinel performs self-fencing after a configurable Grace Period and with Cooldown between actions. This means the behavior remains predictable and reliably reproducible.
Components at a glance
1. Broker health check
The local Pulsar broker will regularly communicate with the endpoint
/admin/v2/brokers/health checked.
The result flows directly into the heartbeat status.
Optionally, authentication via the header
Authorization: Bearer <token>.
2. Heartbeat mechanism
Each node sends UDP packets with its status at fixed intervals
to a broadcast or multicast destination.
Incoming packets are parsed as JSON structures and stored in Maps
(peers, peersTS), which represent all known cluster nodes.
The messages contain:
- Dude
- Node ID
- Timestamp
- Status data (priority, health)
3. Quorum logic
Sentinel assesses the health of the cluster approximately every ten seconds:
- Does your priority group still have a quorum?
- Does a higher priority group with a stable quorum already exist?
If any of these conditions are not met, the fencing timer starts (Grace Period + Cooldown) to avoid inconsistencies.
4. HTTP Health Endpoint
An integrated HTTP server (default port 3000) combines local broker status
and cluster quorum for a consistent evaluation:
GET / → 200 OK # healthy
GET / → 503 FAIL # unhealthy
This endpoint is ideal as a health check for upstream load balancers.
Control and parameterization
The behavior can be controlled via CLI flags or environment variables.
| Flag | Description |
|---|---|
--priority |
Node priority (1 = highest) |
--timeout |
Time until a node is considered “lost” |
--grace-period |
Waiting time before fencing action |
--fencing-cooldown |
Pause between fencing operations |
--multicast |
Optional multicast group instead of broadcast |
--healthcheck |
Broker Health Check URL |
Example:
./pulsar-sentinel --priority 2 --multicast 239.0.0.1 --grace-period 30s
Fencing mechanism
If a node is recognized as critical by the quorum or health check logic,
Sentinel executes the fenceNode() function.
By default there is only logging - no destructive intervention.
In productive setups you can use your own command via the environment variable
FENCING_COMMAND can be defined, e.g. E.g.:
export FENCING_COMMAND="systemctl restart pulsar-proxy"
In this way, a node can be removed from the network in a targeted and controlled manner.
Integration and extensibility
Pulsar Sentinel currently runs fully self-sufficient and does not require a central database or control plane. All states are managed locally in memory and updated regularly – This means that the tool works stable and comprehensible even in isolated environments.
Currently Sentinel is designed for productive use in closed cluster setups, for example on dedicated nodes or in private cloud environments.
Expansions are planned for:
- signed health checks to secure states cryptographically,
- a Prometheus-compatible metrics endpoint,
- as well as additional message types (alerts, sync) for finer control.
These points are deliberately intended as the next evolutionary steps – The basis is in place, the expansion is clearly defined.
The source code is publicly available at: 🔗 pulsar-sentinel (Go version)
A functionally equivalent proof-of-concept version in Rust also exists: 🔗 pulsar-sentinel-rs
The Rust variant demonstrates the same concept with ownership and concurrency security, However, it has been deferred because the build process in Go is much more robust and has proven to be more portable. The main reason was the complexity of creating statically linked binaries: Rust pulled in the OpenSSL-C library across numerous dependencies, which made completely independent distribution difficult.
Conclusion
Pulsar Sentinel is not a framework, but a operational safety net – a tool that detects if a cluster gets out of sync and acts in a timely manner, before inconsistencies arise.
Finding: Stability in distributed systems does not come about through always new layers, but through simple, verifiable mechanisms, who reliably do the right thing in an emergency.