minipulsar – current status of the Pulsar-compatible broker prototype for edge scenarios

Starting point: the protocol experiment

In the run-up to Christmas I described an experiment that arose from a rather banal but recurring situation: In conversations and during troubleshooting, the question kept coming up as to what was really happening on the line at Pulsar.

In everyday life, you rely on client libraries, APIs and configurations. This works well - but hides exactly the part in which many assumptions, states and implicit rules are hidden. For me too, some of this knowledge was felt rather than understood.

The most obvious way to change that was a radical one: build a minimal broker that implements just enough of the Pulsar binary protocol for real clients to talk to. Not as a product, not as a replacement for Apache Pulsar, but as an analysis tool.

This goal was achieved. The broker accepted connections, understood the relevant commands and behaved correctly from the perspective of real Pulsar clients. Above all, he made the protocol tangible: conditions, sequences and expectations were no longer theoretical, but concretely implemented and observable.

Actually, that was the end of everything.

From experiment to targeted prototype

The broker has remained in my toolbox because it has a quality that the original does not: it is small, stable and completely manageable. This raised a new, practical question:

Can a pure protocol experiment be used to specifically develop a broker that can be used sensibly where a complete Apache Pulsar stack is not technically or operationally suitable?

From my daily work with Apache Pulsar, I have developed a very clear feeling for which properties are actually critical in business scenarios - and which, above all, create complexity without providing proportional added value for certain areas of application.

This experience made it possible to make a conscious decision: minipulsar should not be a reduced Apache Pulsar, but rather an independent broker prototype that consistently focuses on a small, clearly defined functional core - and does without a large part of the distributed infrastructure.

A central motive for this further development was platform independence. Consistent use of native Go creates a single static binary with no runtime dependencies.

The broker is already running unchanged today:

• Linux -NetBSD
• FreeBSD
• macOS
• Windows

Especially on long-lasting systems, in isolated networks or on edge nodes, these properties are often more important than horizontal scaling or feature completeness.

Comparison: Apache Pulsar vs. minipulsar

Architecture and runtime model

minipulsar consists of a single static Go binary. The central building blocks are:

• TCP listener for Pulsar (:6650) and optionally TLS (:6651)
• Protocol layer for framing, checksums and Protobuf parsing
• Broker core for state management and delivery
• Storage layer based on SQLite
• a small control plane via declarative HCL configuration

Network and TLS

The broker can serve Plain TCP and TLS at the same time. TLS is implemented as a separate listener and does not change the protocol behavior or the internal state models.

LOOKUP responses can be explicitly configured so that clients receive consistent broker addresses - regardless of the transport used.

Supported Pulsar protocol scope

minipulsar implements the protocol paths necessary for classic publish/subscribe scenarios:

• CONNECT / CONNECTED
• LOOKUP / LOOKUP_RESPONSE
• PARTITIONED_METADATA (0 partitions)
• PRODUCER / PRODUCER_SUCCESS
• SEND / SEND_RECEIPT
• SUBSCRIBE / SUCCESS
• FLOW / MESSAGE
• ACK (individual)
• PING / PONG

Standard Pulsar clients work with this without any adjustments.

Quick start

The quick start shows two deliberately simple variants: minimal operation without a configuration file and a start with explicit HCL configuration.

Minimal startup (without configuration)

./minipulsar -addr :6650 -db ./minipulsar.db

The broker then accepts Pulsar clients at:

pulsar://localhost:6650

Start with HCL configuration

./minipulsar -addr :6650 -db ./minipulsar.db -messaging-config ./minipulsar.hcl

The structure and semantics of the HCL configuration is explained in the following sections (Namespaces, Policies, Functions and Bindings).

Topics and persistence model

minipulsar differentiates between persistent and non-persistent topics.

Use persistent topics:

• an append-only SQLite log
• one cursor per subscription
• a pending table for delivered, not yet confirmed messages

Non-persistent topics do not store messages and only deliver to active consumers.

Delivery and back pressure

Delivery is completely client-driven:

• Consumers control permits via FLOW
• the broker only delivers if permits are available
• There is no broker-side prioritization

Shared subscriptions are served via round robin. ACKs remove pending entries without changing the cursor.

Retention and cleanup

Retention and subscription lifecycles are defined via the control plane:

namespace "persistent://public/default" {
  retention_seconds = 10
  subscription_timeout_seconds = 30
}

This behavior limits memory requirements and amount of state, especially on resource-poor systems.

Security and access control

Security consists of two levels:

• Connection acceptance (security.mode)
• Namespace based produce/consume rules

Missing rules deterministically lead to access denied.

Operational aspects and observability

In addition to the protocol and runtime model, minipulsar also addresses basic operational requirements without requiring additional infrastructure.

Metrics

The broker provides Prometheus-compatible metrics at http://127.0.0.1:8080/metrics.
These are recorded at topic level and allow insights into connections, message flow and delivery - without an external agent or sidecar.

Logging

Logging is carried out consistently 12-factor-compliant via stdout.
There are no implicit log files, no hidden paths and no logger hierarchies with side effects.

This means minipulsar can be integrated directly into existing log pipelines or simply monitored via shell.

Optional TUI (Text User Interface)

Optionally, minipulsar can be started with an interactive TUI.
This is explicitly not intended for productive operation, but rather as a tool for development and test scenarios.

The TUI shows at a glance:

• active client connections
• Producer and consumer activity
• Topic accesses
• current broker events

This is particularly helpful in local developer setups or protocol tests to immediately see how clients connect and how the broker behaves.

Local functions and bindings

A recurring problem in messaging systems is that even very simple transformations or redirects require a full consumer and producer process. This creates unnecessary complexity for trivial data paths: additional code, additional deployments and additional operating states.

minipulsar addresses this problem with local functions and bindings that allow simple processing steps to be defined directly in the broker and explicitly linked to topics.

Full example: Messaging configuration

security {
  mode = "strict"
}

namespace "persistent://public/default" {
  produce = ["tester"]
  consume = ["tester"]
  subscription_timeout_seconds = 30
  retention_seconds = 10
}

function "transform" {
  path = "transform.lua"
  max_runtime = "250ms"
}

binding {
  source   = "persistent://public/default/temperature.f"
  function = "transform"
  target   = "persistent://public/default/temperature.c"
}

The Lua function: temperature conversion

function handle(payload, ctx)
  local value = tonumber(payload)
  if value == nil then
    return payload
  end
  local celsius = (value - 32) * 5 / 9
  return string.format("%.2f", celsius)
end

The function works exclusively on the payload and remains completely decoupled from the Pulsar Wire format.

Bindings as local data paths

Process:

1. Message arrives on temperature.f
2. Broker calls the transform function
3. Return value is republished
4. Consumers read from temperature.c

Implementation approach

As with the original protocol experiment, this prototype was developed iteratively. Significant parts of the code were implemented with the support of an LLM; Architecture decisions, protocol validation, and testing were intentionally done manually.

Conclusion

minipulsar combines Pulsar protocol fidelity with a deliberately small, comprehensible runtime model. As a prototype, it shows that central properties of Pulsar - state models, flow control and client interaction - can be clearly mapped even without a distributed infrastructure.

Minipulsar thus positions itself not as a competitor, but as a complement: as a tool for edge scenarios, for isolated environments and for all cases in which understanding, portability and clarity are more important than maximum scaling.

For me, minipulsar is not a completed project, but rather an ongoing object of thought and work. As long as new questions arise about the protocol, conditions or sensible simplifications, I will continue to work on it, try things out and gradually sharpen them - not with the aim of building “more”, but to better understand what is really necessary.