Apache Kafka dockerization is less complicated that in the case of Cassandra (take a look at post about Dockerize Cassandra troubleshooting). But even of that, there are some thinks to know, globally of the same type as in the case of Cassandra.
Data Engineering Design Patterns
Looking for a book that defines and solves most common data engineering problems? I wrote
one on that topic! You can read it online
on the O'Reilly platform,
or get a print copy on Amazon.
I also help solve your data engineering problems 👉 contact@waitingforcode.com 📩
This post won't follow usual organization in sections. It's because Kafka containerization concerns mostly network issues - and more specifically advertised.listeners property (by supposing that we want to keep ZooKeeper private to container).
The advertised.listeners property defines hostname and port exposed to producers and consumers. By default this value is null. It doesn't make any problem when Kafka is executed in standalone mode, as process of client's machine. The 9092 port is then reachable. The problem occurs when they're not on the same server. For such case, one of solutions is to change advertised.listeners in a way that it exposes broker's IP:
DOCKER_HOST="$(hostname --ip-address)" echo "advertised.listeners=PLAINTEXT://$DOCKER_HOST:9092" >> $kafkaConfig
Beside of that, in configured broker I didn't want to allow automatic topic creation neither block topic deletion. It's the reason why below properties were defined as simply as previous one:
echo "delete.topic.enable=true" >> $kafkaConfig echo "auto.create.topics.enable=false" >> $kafkaConfig
This short post describes some prerequisites to containerize Kafka. It explains which properties need to be configured to make broker available for client application as well some custom entries helping to control better topics manipulation.
Consulting
With nearly 16 years of experience, including 8 as data engineer, I offer expert consulting to design and optimize scalable data solutions.
As an O’Reilly author, Data+AI Summit speaker, and blogger, I bring cutting-edge insights to modernize infrastructure, build robust pipelines, and
drive data-driven decision-making. Let's transform your data challenges into opportunities—reach out to elevate your data engineering game today!
👉 contact@waitingforcode.com
đź”— past projects
