From batch to event
Recently, during an integration task, I was once again faced with a familiar pattern: A legacy system regularly generated data extracts as files that were then retrieved via FTP.
The obvious question arose: Why actually wait for a cron job to collect them? Couldn’t you react in real time - at the exact moment the file is written and closed?
Filecast was born from this idea: A lightweight Go tool that detects file system events and automatically streams every newly written file into an Apache Pulsar topic.
Motivation
The “write file, collect later” principle is firmly anchored in many integration landscapes – robust but unresponsive. Data is created in batches, not in real time. Any delay in this process increases the latency before information is actually usable.
A batch process works at intervals – say every 10 minutes or once a night. An event process reacts immediately as soon as something happens.
Instead of “fetch all new files”, it says:
“A file has been written – react now!”
If a company already runs an event infrastructure like Apache Pulsar, this change is easy to make: You don’t have to rebuild the legacy systems - just eventize the “end of file”.
Idea
The basic idea is surprisingly simple - and that’s why it’s so charming:
As soon as a file has been completely written in a monitored directory, Filecast recognizes this, reads the file in blocks and transfers it as a chunked stream to a Pulsar topic:
persistent://<tenant>/<namespace>/files/<filename>.stream
This means that every newly created file becomes an event stream – traceable, organized and reliable. The whole thing happens immediately as you write, not in a nightly batch.
A billing system writes a new CSV file of bookings to /data/export/ every hour.
With Filecast, every file ends up in the topic as a stream immediately after completion
persistent://finance/exports/files/buchungen_2025-11-11.csv.stream.
A downstream service can then directly consume, validate or import this data into a database - without FTP or polling.
Implementation
The proof-of-concept filecast is developed in Go - for good reasons:
- Go programs start quickly, run stably and do not require an external runtime environment.
- Concurrency and channels make Go ideal for IO-intensive processes such as file monitoring and streaming.
- In addition, Go can be compiled platform-independently - a binary is enough, no installation necessary.
Under the hood, filecast combines two central technologies:
1. inotify – File system events on Linux
inotify is a kernel subsystem that notifies applications of file changes.
Filecast uses it to react to new or changed files in real time.
As soon as a file is created (CREATE) and later closed (CLOSE_WRITE) in the monitored directory, Filecast detects that the write process is complete.
Only then does it read the file and transfer it to Pulsar – at exactly the right moment, without race conditions or half-finished data.
inotify is Linux specific.
On other operating systems - such as macOS or Windows - comparable mechanisms such as FSEvents or ReadDirectoryChangesW exist.
Also: Monitoring only works reliably on local file systems, not on NFS or SMB mounts.
2. Apache Pulsar Go Client – Publish files as a stream
Filecast uses the native Apache Pulsar Go Client to send the files. This allows data to be published in so-called Chunked Messages: The file is broken down into manageable blocks, sent sequentially and reassembled into a stream by the Pulsar broker.
This means that even large files can be processed efficiently without having to load them completely into memory.
Local testing with Podman
If you want to try out the principle, you can start Apache Pulsar locally with Podman - quickly, isolated and without installation:
podman run -it --rm \
-p 6650:6650 \
-p 8080:8080 \
docker.io/apachepulsar/pulsar:4.0.7 \
bin/pulsar standalone
This container launches a complete standalone instance with Broker, BookKeeper and Zookeeper.
The ports 6650 (client) and 8080 (admin/API) are directly available.
Start filecast
A simple call is then sufficient:
export PULSAR_URL=pulsar://localhost:6650
./filecast run -c \
--tenant public \
--namespace default \
--topic-base files \
-w /home/mpeterma/Dokumente
What happens when that happens?
-wsets the watched directory--tenant,--namespace,--topic-basedetermine the topic namespace-cenables chunked upload
Filecast now runs in the background and reacts to every new file in the target directory.
echo "Hello Pulsar" > ~/Dokumente/test.txt
Shortly thereafter, Filecast should publish it in the appropriate topic.
Verify with pulsar-cli
You can use the Pulsar CLI to check whether the messages end up correctly in the Pulsar broker:
export PULSAR_URL=pulsar://localhost:6650
./pulsar-cli consumer --regex -t 'persistent://public/default/files/.*' -s debug-subscriber
This will subscribe to all topics that belong to Filecast. As soon as a file is written and closed, the subscriber receives the associated chunks.
Architecture sketch
[ Legacy-System ]
|
| (schreibt Datei)
v
[ Dateisystem-Folder ] <-- inotify --> [ Filecast ]
|
| (streamed chunks)
v
[ Apache Pulsar ]
|
v
[ Modernes Zielsystem ]
Conclusion
filecast is a small but powerful tool to transform existing file-based interfaces in real time.
It carefully brings old integration patterns into the new world of Event-Driven Architecture.
Instead of cron jobs and FTP, there is now streaming and persistence. Instead of polling, there is reaction. And most importantly: Instead of batch there is Event.
This isn’t a concept on slides - the code is open source and running:
Image credit: The Pulsar logo is a registered trademark of the Apache Software Foundation. Use in the context of editorial reporting in accordance with the ASF trademark and logo guidelines.