RabbitMQ doesn't introduce very much new concepts to Entreprise Integration Patterns. However, it's good to know how it defines them before starting to write code.
What would it take for you to trust your Databricks pipelines in production?
A 3-day bug hunt on a 3-person team costs up to €7,200 in lost engineering time. This workshop teaches you to prevent that — unit tests, data tests, and integration tests for PySpark and Databricks Lakeflow, including Spark Declarative Pipelines.
Konieczny
This article introduces basic concepts of RabbitMQ. It doesn't focus on global Entreprise Integration Pattern components because they were described in the article about introduction to Spring Integration. Each RabbitMQ component is described in separate part. And it begins by exchanges.
Exchanges
Exchanges are middlemen between message producers and consumers. Messages are not transmitted directly to one or more queues. They are sent firstly to exchanges and then, routed to appropriated queues. Routing is made according to exchange type. We can distinguish 4 types of exchanges (for AMQP v0.9.1):
- direct exchange - the simplest one. The message is routed directly to corresponding queue. The queue is identified through routing key matching.
- fanout exchange - this type allows to send one message to two or more queues. This exchange doesn't contain a concept of identifying appropriated queues. Once one queue is subscribed to given exchange, it'll receive the message, point.
- topic exchange - it's like the mix of direct and fanout exchange. It routes the message to queues taking into account: the routing key and queue pattern used to bind to given exchange. Messages sent to this exchange can be routed to several queues.
- headers exchange - as the name indicates, this exchange is based on headers defined in transmitted message. The routing key is ignored and only headers are considered in message dispatching process. The routing is made in 2 configurations: for all matching headers and for any matching header. The configuration to apply is defined by the queue throught x-match attribute.
Queues
In previous part we saw a lot of places where queue word appears. The definition which could be associated to them is: the elements storing messages consumed by application. As observed previously, messages are first dispatched to exchanges. After that, according to exchange type, they are sent to queues, where they remain until consumption.
Two queue types exist: durable and transient. The first ones can survive RabbitMQ restarts because they are saved to persistent storage (as disk). The second ones is recreated from scratch in the case of reboot. But in the part describing messages we'll see that durable queue doesn't guarantee that message survives broker restart.
Bindings
This concepts describes the relationship between queues and exchanges. When one queue wants to receive messages from exchange, it must define some rules. According to these rules, exchange will chose appropriated queues and route messages to them. This rules are known as bindings.
Channels
To work with RabbitMQ, the application must connect to it. The connection is nothing more, nothing less than TCP connection between RabbitMQ broker and the application consuming or producing messages. After establishing the connection, a AMQP channel is created. The communication is not done directly through TCP connection but through AMQP channel defined inside it.
The mainspring of channels is the performance. TCP session management is quite expensive. Opening multiple TCP connections could lead to performance problems. It's more optimized to open one connection per application and define one or more channels inside it.
Messages
Another very important component of RabbitMQ is a message. A message is composed by payload and label. The first one contains transmitted data. The second is more linked to configuration because it contains routing key used to determine exchange which should receive the message.
Generally, messages aren't the most mysterious part for RabbitMQ beginners. At the begin, the single main problem can be provoked by the messages durability. As mentioned earlier, by default messages don't persistent. To make them persistent, ie. be able to recover them after RabbitMQ restart, 3 conditions must be respected:
- message delivery mode must be set to persistent
- message must be published into a durable exchange
- message must be stored in a durable queue
However, forcing messages persistence brings some performance drawbacks. Persistent messages are stored in a persistence log file. So they must be saved in disk with I/O operation. And this kind of operations is much slower than the same operation executed on RAM side.
To resume this introduction part, RabbitMQ consists on messages sent through a channel opened in connection between application and RabbitMQ broker. These messages are routed to appropriated queues. Routing process is made thanks to defined bindings. By default, messages don't persist and disappear after RabbitMQ restart. To bypass that, 3 parts: message, exchange and queue must be configured to be persistent.
Data Engineering Design Patterns
Looking for a book that defines and solves most common data engineering problems? I wrote
one on that topic! You can read it online
on the O'Reilly platform,
or get a print copy on Amazon.
I also help solve your data engineering problems contact@waitingforcode.com đź“©