Data hub instead of a hatch: Pulsar Functions in the data stream

Introduction: From the message bus to the data hub

Event streaming platforms are still considered necessary but largely passive infrastructure in many architectures. They transport messages from A to B, decouple systems temporally and, ideally, ensure reliability and scalability. However, this is often where their role ends.

Apache Pulsar allows – under clear technical and organizational assumptions – a different approach. The platform can not only transport messages, but also become a central data hub, where data access, simple decision logic and governance coincide.

The idea behind it is pragmatic: If data flows through a central system anyway, certain rules, selections and simple transformations can take place there - provided they affect the data flow (and do not represent core technical logic), and provided ownership, operation and boundaries are clearly defined.

This article shows how Pulsar Functions implement this approach, under what conditions it proves itself – and where conscious limits must be drawn.

Apache Pulsar: Architecture and basic concepts

In order to be able to classify Pulsar Functions in a meaningful way, it is worth taking a quick look at the central terms.

Brokers are the active nodes of the system. They receive messages, manage topics and subscriptions and – optionally – execute Pulsar functions. Importantly, brokers are deliberately designed to be low-state and can be scaled horizontally. Persistent state lies outside the broker (e.g. in BookKeeper).

Tenants form the highest organizational and security boundary. They clearly separate clients, organizational units or projects from each other.

Namespaces further structure tenants. They bundle topics, subscriptions, retention rules, quotas and ACLs - and are therefore a clearly defined data space. In practice, the namespace is often the central unit for organizing deployments, limits and responsibilities for functions.

Topics are persistent event streams. Messages are not simply passed on, but are stored permanently and made available for consumption in a time-decoupled manner.

Subscriptions define how topics are read. They are not a detail, but a core governance tool for access, semantics, repeatability and load balancing.

Pulsar Functions: Compute in the data stream – with clear boundaries

Pulsar Functions expands this model to include targeted compute. They allow messages to be sent directly in the data stream:

• filter
• to forward (routing)
• to transform
• to be technically enriched

This is not about moving any technical logic into the broker. Pulsar functions address exactly the logic that is directly connected to the data flow - i.e. where decisions at the data edge would otherwise be distributed and difficult to control.

Operationally: A function is code in the data path. It can improve the flow of data – or, if discipline is poor, destabilize it. This is not an argument against functions, but an argument for clear rules, ownership and guardrails.

Case study: Digital products in the ERP data stream

Let’s look at a typical scenario from practice.

A proprietary ERP system mediates all business transactions via a message bus. In order to execute an order, it expects a message with a shopping cart from the shop system. Thousands of such orders are received every second. There is also a delivery system for digital products such as eBooks or MP3 downloads. This system is only interested in orders that contain at least one digital product, in order to then provide the customer with a download link.

On a “dumb” message bus, the delivery system would have to consume all orders, parse each message and decide for itself whether it is relevant.

This typically leads to:

• unnecessary network traffic (all data everywhere)
• wasted computing power at the edge (every consumer makes the same selection)
• proprietary formats that migrate deep into consuming systems (coupling)
• Debugging and governance problems (“Who filters where – and why?”)

Adapting the ERP system is - which is not uncommon - often expensive, time-consuming or simply not possible.

The Pulsar solution: selection in flight – under governance

Apache Pulsar allows this logic to be moved to where it has the most impact: into the data stream itself – as long as it stays close to the data flow and is operated cleanly.

A Pulsar Function analyzes incoming orders, recognizes digital products and forwards only relevant messages to a dedicated topic. Optionally, a transformation into a neutral, manufacturer-independent specialist format can already take place at this point.

That is correct. The alternative, however, is not “no logic”, but rather a (usually less controllable) distribution:

• Selection in each consumer (diffuse, difficult to audit)
• additional bridge services (more infrastructure, own lifecycle)
• or pressure on the ERP manufacturer (often unrealistic)

Alternatives in comparison – fairly ranked

Function Development

The actual function is deliberately implemented in Java: It is established in many companies and fits well into existing build and operational processes.

The model is more important than the language: A Pulsar Function is a small, clearly defined unit that makes exactly one decision in the data stream. It implements the Function<I, O> interface and has no infrastructure of its own, no server of its own and no “service lifecycle” of its own.

The following function only checks incoming orders to see whether they contain digital products. The example is deliberately kept simple to focus on the principle - not on parsing or schema details.

package net.d2ux.stream.order.filter;

import org.apache.pulsar.functions.api.Context;
import org.apache.pulsar.functions.api.Function;

public final class DigitalOrderFilter implements Function<String, String> {

    private static final String KEYWORD = "digital";

    @Override
    public String process(String input, Context context) {

        if (input == null || input.isBlank()) {
            context.getLogger().debug("Skipping empty message");
            return null;
        }

        // echte Systeme nutzen Schema (JSON/Avro) + robuste Validierung.
        if (input.toLowerCase().contains(KEYWORD)) {
            context.getLogger().info("Digital order detected");
            return input;
        }

        return null;
    }
}

Build and deploy the function

The function is built using Maven. A complete project therefore includes a pom.xml, a clean package structure and (ideally) tests and CI definitions.

For the example, a simple build step that creates an executable JAR is sufficient:

mvn clean install

The result is a single JAR. Deployment takes place via the Pulsar Shell:

$ pulsar-shell
default(localhost)> admin functions create \
  --tenant public \
  --namespace default \
  --name order.filter.digital \
  --jar /abs/path/filter-order-digital-1.0.0.jar \
  --classname net.d2ux.stream.order.filter.DigitalOrderFilter \
  --inputs persistent://public/default/orders.raw \
  --output persistent://public/default/orders.filtered.digital

Governance specifically: Guardrails instead of uncontrolled growth

Compute in stream only becomes an advantage if it remains controlled. Typical guardrails that have proven themselves in practice:

• Deployment rights: Only defined roles/teams are allowed to create/change functions.
• Topic whitelist: Functions are only allowed to access shared topics (input/output).
• Naming and versioning rules: Clearly identify what is “prod” and what is “experiment”.
• Change Management: Review, CI, rollback plan.
• Limits: Quotas and resource limit per namespace to protect the broker.

This is the point at which “governance” becomes concrete: not as a buzzword, but as a set of rules that make operations possible.

Tests in the data stream with the Pulsar CLI

The best way to test streaming systems is where they work: in the ongoing data flow. Instead of complex test harnesses, a deliberately simple approach that still tests realistic behavior is often sufficient.

The Pulsar CLI allows you to specifically start producers and consumers and thus observe the behavior of the function in interaction with brokers, topics and subscriptions.

In the first step, a consumer is started on the target topic to which the function forwards relevant messages:

$ export PULSAR_URL=pulsar://127.0.0.1:6650
$ pulsar-cli consumer -t 'persistent://public/default/orders.filtered.digital' -s debug

Producer on the source topic:

$ echo "Bestellung analog"  | pulsar-cli producer -t 'persistent://public/default/orders.raw'
$ echo "Bestellung digital" | pulsar-cli producer -t 'persistent://public/default/orders.raw'

Expected behavior: Only the digital order appears in the target topic.

Monitoring and transparency

Pulsar exports comprehensive metrics for Functions. Latencies, throughput and errors are visible - not a black box.

Important: Debugging in stream is different than debugging in services. You need clear metrics, clean logs and defined SLOs. Then it becomes manageable.

Conclusion

Pulsar Functions are not a replacement for microservices or any serverless feature. They are a precise tool for making clearly defined decisions in the data stream.

Those who respect their boundaries, clarify ownership and take governance seriously, can place compute where data flows anyway – with high transparency, auditable rules and often significantly less wastage than with edge selection in every consumer.

Apache Pulsar shows: A message bus can be more than transport – a manageable data hub, if you operate it as an infrastructure product (and not as a random script collection point).

Image credit: The Pulsar logo is a registered trademark of the Apache Software Foundation. Use in editorial reporting in accordance with official guidelines.