Pulsar Data Pump for Isolated Environments

Challenge

In segmented infrastructures with multiple isolated Pulsar instances, selected topics had to be replicated unidirectionally across strict network boundaries where native geo-replication was not feasible.

Approach

I developed a custom Go-based Pulsar data pump for reliable high-throughput transport:

  • selective unidirectional replication
  • batching for high throughput
  • resilient processing under latency and transient outages
  • Prometheus endpoint for monitoring and alerting
  • production-ready, low-overhead operation

Snapshot

  • Industry: Industrial enterprise
  • Time frame: 2024
  • Technologies
    • Apache Pulsar
    • Go (custom data pump)
    • Prometheus / monitoring
    • TLS / firewall-separated environments

The solution enables controlled propagation of production streams into staging and test landscapes while remaining reliable in 24/7 operations.

Do you have a similar project in mind?

With custom tooling, I build reliable bridges across segmented system landscapes - robust, performant, and controllable.

Get in touch