Messaging articles

Looking for something else? Check the categories of Messaging:

Apache Kafka Apache Pulsar RabbitMQ

If not, below you can find all articles belonging to Messaging.

Isolation level in Apache Kafka consumers

Who says transaction, automatically invokes isolation levels, so what can be viewed by the consumer from uncommitted transactions. Apache Kafka also implements this concept and I will take a closer look on it in this blog post.

Continue Reading β†’

Control messages in Apache Kafka

During my last exploration of logs compaction, I found a method called isControlBatch. At the time, I only had a rough idea about this category of batches and that's the reason why I decided to learn a little bit more about them.

Continue Reading β†’

Records writing in Apache Kafka

In my journey to understand transaction internals in Apache Kafka, I discovered another intriguing class that by the way led me to a few others ;) This class is RecordBatch but in this blog post you will also meet MemoryRecords and FileRecords.

Continue Reading β†’

Offset-based lookup in Apache Kafka

In March I published a blog post about timestamp-based lookup in Apache Kafka. But as you know, it's not the single lookup possibility. Another one uses indexes and it will be the topic of this article.

Continue Reading β†’

The role of Apache BookKeeper in Apache Pulsar - part 1

After recent introduction to Apache ZooKeeper in Apache Pulsar, it's time to see another component of this messaging system, namely Apache BookKeeper. In this first post of the series, I will introduce the key concepts of BookKeeper and also try to see how they're implemented.

Continue Reading β†’

Logs compaction in Apache Kafka - compact cleanup policy

Delete covered one of my previous posts is not a single clean up policy in Apache Kafka. Another one is compaction which reduces the size of the segments by keeping the last value for every key.

Continue Reading β†’

Logs compaction in Apache Kafka - delete and cleanup policy

Since my very first experiences with Apache Kafka, I was always amazed by the features handled by this tool. One of them, that I haven't had a chance to explore yet, is logs compaction. I will shed some light on it in this and next week's article.

Continue Reading β†’

Global and local Apache ZooKeeper in Apache Pulsar - part 2

In my last post about Apache pulsar, I introduced global and local ZooKeepers. In this one, which is the follow-up, I'll check what both of them contain.

Continue Reading β†’

Files synchronization - zoom at Kafka HDFS Connector

When I was playing with Kafka HDFS Connector, I saw that the generated files are suffixed by some numbers. It intrigued me and I decided to explore the topic in this article.

Continue Reading β†’

Global and local Apache ZooKeeper in Apache Pulsar - part 1

Before really exploring Apache ZooKeeper in Apache Pulsar, I was thinking that 1 post will be enough. But in fact it won't be, unless it would be really long which often is hard to read and understand. That's why you can find here the first part of my exploration about ZooKeeper in Pulsar.

Continue Reading β†’

Apache Pulsar - global architecture and local setup

I'm really happy to start a whole new chapter on the blog and include Apache Pulsar to my monitored topics! Even though I already wrote about this technology in December 2019, I still feel hungry because the topic was more an analysis of Apache Spark Structured Streaming connector than the analysis of the tool per se. That's why I'm starting right now with the presentation of the basic concepts, like global architecture and local setup, both needed to go further.

Continue Reading β†’

Timestamp-based lookup in Apache Kafka

The next thing I wanted to understand while still working on transactions was the lookup. I can imagine how to get the first or the last element of a partition but I had no idea how it can work for more fine-grained access, like the one using timestamp.

Continue Reading β†’

Apache Kafka idempotent producer

I wrote all posts published this year about Apache Kafka (NIO, max in flight requests) to better understand idempotent producers. In this post I'll try to do that before going further and analyze transactions support.

Continue Reading β†’

Apache Kafka and max.in.flight.requests.per.connection

I didn't plan to write this post at all. However, when I was analyzing the idempotent producer, it was hard to understand out-of-sequence policy for multiple in-flight requests without understanding what this in-flight requests parameter really means.

Continue Reading β†’

NIO Selector in Apache Kafka

It's rare when in order to write a blog post I need to cover more than 3 other topics. But that's what happens with Apache Kafka idempotent producer that I will publish soon. But before that, I need to understand and explain NIO Selector, its role in Apache Kafka, and finally the in flight requests. Since the first topic was already covered, I will move to the second one.

Continue Reading β†’

From Apache Spark connector to Apache Pulsar basic concepts

Some time ago I saw an interesting presentation about Apache Pulsar and it intrigued me. Compute separated from the storage in a streaming system? Sounds great! In this series of posts, I will try to understand how different challenges were solved but I will start by making an exercise of trying to figure out Apache Pulsar's architecture from its Structured Streaming connector.

Continue Reading β†’

Dockerize Kafka

Apache Kafka dockerization is less complicated that in the case of Cassandra (take a look at post about Dockerize Cassandra troubleshooting). But even of that, there are some thinks to know, globally of the same type as in the case of Cassandra.

Continue Reading β†’

Replication in Apache Kafka

Since Apache Kafka is distributed messaging system and we haven't described replication yet, it's a good moment to do so.

Continue Reading β†’

Requests in Apache Kafka

Kafka clients communicate with broker through dedicated TCP connection. They send a lot of different requests, mostly to handle eventual rebalancing.

Continue Reading β†’

Controller in Apache Kafka

Controller is a well-known concept for the ones who have worked with MVC paradigm. But regarding to Kafka, it shouldn't be thought in the same categories.

Continue Reading β†’