MQTT meets Pulsar: A pragmatic bridge for modern IoT architectures

From the device to the event backbone

In many IoT landscapes you come across a familiar starting point: Countless sensors and edge devices speak MQTT – slim, simple, efficient. In addition, there are established infrastructures with established brokers such as Mosquitto, EMQX, HiveMQ or the MQTT stacks from various industrial and gateway manufacturers. These brokers are deeply integrated into existing edge topologies, are often operated with high availability and are part of security-critical communication paths.

At the same time, there is an increasing desire in the backend for streaming, persistence, scaling, Traceability, multi-tenancy and modern evaluation options – something that classic MQTT brokers only partially or not at all.

The question therefore arises:

Why not stream MQTT messages directly into Apache Pulsar - without rebuilding the existing MQTT infrastructure?

It was precisely from this idea that pulsar-iotbridge was born: A compact Go tool that attaches to an existing MQTT broker, subscribed to its topics and all incoming messages reflected into a clean, consistent Pulsar naming structure.

The existing MQTT landscape remains unchanged, while a modern, scalable event system is created in the backend.

Motivation

MQTT is great for device communication, but only to a limited extent Backend processing and analysis. In contrast, Apache Pulsar offers:

• Topics with retention
• horizontal scaling
• Subscription models for real-time and replays
• a clean multi-tenant model
• Backlogs, DLQs, compaction, geo-replication…

Short: MQTT for the edge, Pulsar for the backend - two worlds that complement each other perfectly.

Idea

The basic idea of ​​pulsar-iotbridge is deliberately minimalist:

1. Connect to an MQTT broker
2. Subscribe to all desired topics (by default: #)
3. Assign each incoming message to a naming structure
4. Publish the payload unchanged in the associated Pulsar topic

From an MQTT topic like:

dt/device123/temperature

becomes a pulsar topic like:

persistent://public/mqtt_ingest/dt/device123/temperature

Without transformation, without magic – simply an event transport between two worlds.

Implementation

The focus of the project is not on as many features as possible, but on a comprehensible, maintainable and secure architecture.

Pulsar-iotbridge relies on two proven building blocks:

1. MQTT client (Paho)

The bridge uses Eclipse Paho’s established Go implementation for MQTT. It connects as a client, subscribes to the desired topics and receives every message via a callback.

2. Apache Pulsar Go client

For the backend, the bridge uses the native Pulsar Go client. This offers:

• efficient batching
• LZ4 compression
• asynchronous publishing
• stable connection handling

A publisher is created and reused per Pulsar topic. This creates a clearly defined path per topic - structured and neatly isolated.

Architecture from a bird’s eye view

[ IoT-Device ] --MQTT--> [ MQTT-Broker ] --> [ pulsar-iotbridge ]
                                                          |
                                                          | Pulsar-Event
                                                          v
                                                [ Apache Pulsar ]
                                                          |
                                                          v
                                               [ Analytics / Backend ]

Local testing with Podman

You can start a Pulsar standalone with just a few lines:

podman run -it --rm \
  -p 6650:6650 \
  -p 8080:8080 \
  docker.io/apachepulsar/pulsar:4.0.7 \
  bin/pulsar standalone

An MQTT broker can run in parallel, e.g. E.g. Mosquito.

Starting the bridge

./pulsar-iotbridge run \
  --mqtt-url tcp://localhost:1883 \
  --pulsar-url pulsar://localhost:6650 \
  --tenant public \
  --namespace mqtt_ingest \
  --workers 8

Now all MQTT messages from all topics are intercepted and mirrored to Pulsar.

Verify

Receiving page in Pulsar:

./pulsar-cli consumer --regex -t 'persistent://public/mqtt_ingest/.*' -s debug

MQTT test:

mosquitto_pub -t dt/dev1/temp -m "22.4"

A few milliseconds later, the message appears in the Pulsar consumer.

Warnings and limitations

Alternative: Pulsar MOP (MQTT-on-Pulsar)

Instead of running an external bridge, Pulsar-MOP extends the Pulsar broker itself with a complete MQTT server. Devices still speak MQTT — but end up directly in the Pulsar ecosystem.

Advantages:

• no separate bridge or additional infrastructure required
• Devices send directly to Pulsar → no media disruption
• full native MQTT compatibility (QoS, Sessions, Retain etc.)
• Data lands immediately in Pulsar topics and benefits from retention, backlog & replays

Disadvantages:

• completely replaces the existing MQTT broker
• higher operational footprint (Pulsar also has to handle MQTT workloads)
• often too heavyweight for resource-limited edge scenarios
• less decoupled than an external bridge (monolithization tendency)

Conclusion

pulsar-iotbridge is a small but effective tool for existing MQTT landscapes into the event world of Apache Pulsar - without converting devices, without major architectural changes, without heavy connector infrastructure.

Image credit: The Pulsar logo is a registered trademark of the Apache Software Foundation. Use in editorial reporting in accordance with official guidelines.