Pulsar Sentinel - Cluster Health and Fencing

Challenge

Critical Apache Pulsar environments needed additional safeguards beyond built-in mechanisms, including cross-boundary heartbeat checks, priority-based quorum validation, automated fencing, and transparent observability.

Approach

I developed Pulsar Sentinel as an independent tool combining heartbeat messaging, health checks, quorum logic, and configurable fencing actions.

Core capabilities:

• UDP heartbeat and broker-health monitoring
• priority-group quorum checks with escalation
• grace/cooldown handling to avoid unnecessary fencing
• configurable shell-based fencing actions
• Prometheus metrics endpoint with Grafana visualization
• full CLI-based configuration for timeouts, quorum sizes, ports, and logging

Snapshot

• Industry: Internal use
• Time frame: 2025
• Technologies
  • Go
  • Prometheus / Grafana
  • UDP / multicast heartbeats
  • TLS / JWT checks against Pulsar Admin API

Pulsar Sentinel runs continuously in 24/7 operation and significantly improves recovery behavior and overall messaging-platform availability.