Pulsar Data Pump for Isolated Environments

Challenge

In segmented infrastructures with multiple isolated Pulsar instances, selected topics had to be replicated unidirectionally across strict network boundaries where native geo-replication was not feasible.

Approach

I developed a custom Go-based Pulsar data pump for reliable high-throughput transport:

• selective unidirectional replication
• batching for high throughput
• resilient processing under latency and transient outages
• Prometheus endpoint for monitoring and alerting
• production-ready, low-overhead operation

Snapshot

• Industry: Industrial enterprise
• Time frame: 2024
• Technologies
  • Apache Pulsar
  • Go (custom data pump)
  • Prometheus / monitoring
  • TLS / firewall-separated environments

The solution enables controlled propagation of production streams into staging and test landscapes while remaining reliable in 24/7 operations.