Dockerize Kafka


Apache Kafka dockerization is less complicated that in the case of Cassandra (take a look at post about Dockerize Cassandra troubleshooting). But even of that, there are some thinks to know, globally of the same type as in the case of Cassandra.

Data Engineering Design Patterns

Looking for a book that defines and solves most common data engineering problems? I'm currently writing one on that topic and the first chapters are already available in πŸ‘‰ Early Release on the O'Reilly platform

I also help solve your data engineering problems πŸ‘‰ πŸ“©

This post won't follow usual organization in sections. It's because Kafka containerization concerns mostly network issues - and more specifically advertised.listeners property (by supposing that we want to keep ZooKeeper private to container).

The advertised.listeners property defines hostname and port exposed to producers and consumers. By default this value is null. It doesn't make any problem when Kafka is executed in standalone mode, as process of client's machine. The 9092 port is then reachable. The problem occurs when they're not on the same server. For such case, one of solutions is to change advertised.listeners in a way that it exposes broker's IP:

DOCKER_HOST="$(hostname --ip-address)" 
echo "advertised.listeners=PLAINTEXT://$DOCKER_HOST:9092"  >> $kafkaConfig 

Beside of that, in configured broker I didn't want to allow automatic topic creation neither block topic deletion. It's the reason why below properties were defined as simply as previous one:

echo "delete.topic.enable=true" >> $kafkaConfig
echo "auto.create.topics.enable=false" >> $kafkaConfig

This short post describes some prerequisites to containerize Kafka. It explains which properties need to be configured to make broker available for client application as well some custom entries helping to control better topics manipulation.

If you liked it, you should read:

πŸ“š Newsletter Get new posts, recommended reading and other exclusive information every week. SPAM free - no 3rd party ads, only the information about waitingforcode!