Messaging articles

Looking for something else? Check the categories of Messaging:

Apache Kafka Apache Pulsar RabbitMQ

If not, below you can find all articles belonging to Messaging.

Global and local Apache ZooKeeper in Apache Pulsar - part 2

In my last post about Apache pulsar, I introduced global and local ZooKeepers. In this one, which is the follow-up, I'll check what both of them contain.

Continue Reading →

Files synchronization - zoom at Kafka HDFS Connector

When I was playing with Kafka HDFS Connector, I saw that the generated files are suffixed by some numbers. It intrigued me and I decided to explore the topic in this article.

Continue Reading →

Global and local Apache ZooKeeper in Apache Pulsar - part 1

Before really exploring Apache ZooKeeper in Apache Pulsar, I was thinking that 1 post will be enough. But in fact it won't be, unless it would be really long which often is hard to read and understand. That's why you can find here the first part of my exploration about ZooKeeper in Pulsar.

Continue Reading →

Apache Pulsar - global architecture and local setup

I'm really happy to start a whole new chapter on the blog and include Apache Pulsar to my monitored topics! Even though I already wrote about this technology in December 2019, I still feel hungry because the topic was more an analysis of Apache Spark Structured Streaming connector than the analysis of the tool per se. That's why I'm starting right now with the presentation of the basic concepts, like global architecture and local setup, both needed to go further.

Continue Reading →

Timestamp-based lookup in Apache Kafka

The next thing I wanted to understand while still working on transactions was the lookup. I can imagine how to get the first or the last element of a partition but I had no idea how it can work for more fine-grained access, like the one using timestamp.

Continue Reading →

Apache Kafka idempotent producer

I wrote all posts published this year about Apache Kafka (NIO, max in flight requests) to better understand idempotent producers. In this post I'll try to do that before going further and analyze transactions support.

Continue Reading →

Apache Kafka and max.in.flight.requests.per.connection

I didn't plan to write this post at all. However, when I was analyzing the idempotent producer, it was hard to understand out-of-sequence policy for multiple in-flight requests without understanding what this in-flight requests parameter really means.

Continue Reading →

NIO Selector in Apache Kafka

It's rare when in order to write a blog post I need to cover more than 3 other topics. But that's what happens with Apache Kafka idempotent producer that I will publish soon. But before that, I need to understand and explain NIO Selector, its role in Apache Kafka, and finally the in flight requests. Since the first topic was already covered, I will move to the second one.

Continue Reading →

From Apache Spark connector to Apache Pulsar basic concepts

Some time ago I saw an interesting presentation about Apache Pulsar and it intrigued me. Compute separated from the storage in a streaming system? Sounds great! In this series of posts, I will try to understand how different challenges were solved but I will start by making an exercise of trying to figure out Apache Pulsar's architecture from its Structured Streaming connector.

Continue Reading →

Dockerize Kafka

Apache Kafka dockerization is less complicated that in the case of Cassandra (take a look at post about Dockerize Cassandra troubleshooting). But even of that, there are some thinks to know, globally of the same type as in the case of Cassandra.

Continue Reading →

Replication in Apache Kafka

Since Apache Kafka is distributed messaging system and we haven't described replication yet, it's a good moment to do so.

Continue Reading →

Requests in Apache Kafka

Kafka clients communicate with broker through dedicated TCP connection. They send a lot of different requests, mostly to handle eventual rebalancing.

Continue Reading →

Controller in Apache Kafka

Controller is a well-known concept for the ones who have worked with MVC paradigm. But regarding to Kafka, it shouldn't be thought in the same categories.

Continue Reading →

Rebalancing in Apache Kafka

If you're working with Katka, rebalancing is maybe a word the most commonly met. It's also an important word because it helps to ensure correct message consumption.

Continue Reading →

Coordinator in Apache Kafka

Since Kafka is a distributed system, it naturally has to coordinate its members somehow. The synchronization mechanism is ensured by coordinators.

Continue Reading →

Configuring Apache Kafka

Until now we've almost used only default configuration values. In this article we'll see some of configuration possibilities more in detail.

Continue Reading →

Messages in Apache Kafka

An intrinsic part of each messaging system are messages. After learnt previously about producing and consuming messages, it's good moment to see what these messages really are.

Continue Reading →

Producers in Apache Kafka

After presenting consumers in Kafka, it's a good moment to pass to producers.

Continue Reading →

Message queue in Apache Kafka

In one of previous articles we've discovered how to implement Kafka to publish/subscribe pattern, ie. one consumer per consumer group. This time we'll describe another aspect of Kafka consuming, messages queue.

Continue Reading →

Consumers in Apache Kafka

When you need to write about messaging, you'll certainly meet a dilemma about 'which part, consuming or producing, describe first ?'. I decided to start with simple consumers. By 'simple', I mean only 1 consumer per group. Thanks to that we can avoid the point of partition sharing.

Continue Reading →