Spring integration is a Spring-family project illustrating well the idea of enterprise integration patterns. To understand better this Spring project, we'll write some test cases to show how they work. This article will introduce the basic concepts of Spring Integration.
What would it take for you to trust your Databricks pipelines in production?
A 3-day bug hunt on a 3-person team costs up to €7,200 in lost engineering time. This workshop teaches you to prevent that — unit tests, data tests, and integration tests for PySpark and Databricks Lakeflow, including Spark Declarative Pipelines.
Konieczny
Presented basics are needed to, first, understand Spring Integration project, and second, to better take the test cases up. These basics are hidden under the concept of Entreprise Integration Patterns which will be described in the first part of the article. At the second part we'll present main components of Spring Integration project. At the end we'll describe shortly the method used to learn Spring Integration.
Entreprise Integration Patterns
Web people can think about integration as about putting together images and styles in the goal to construct HTML page. However, in enterprise IT world, integration means the way of put together different components of company services. For example, it can be translated by association of orders passed in website to global CRM system. This kind of interactions were described in "Enterprise Integration Patterns" reference book by Gregor Hohpe and Bobby Woolf. We, for now, we'll discuss only about the main concepts of EIP.
Let's iterate through main concepts of enterprise integration:
messages : it's an unit of information transmitted between different applications. It's also a kind of contract between both of them because sender commits to send the message respecting the expectation of receiver. In practice, it can by translated by, for example, sending a message correctly formatted (respecting XSD) or of correct type (XML, JSON etc.). A message should also be atomic, ie. could be transmitted entirely or not be transmitted at all.
In some cases, message can be also splitted to several small packets. These packets are after put together to reconstruct message initially sent. Message sent in this way is called Message Sequence.
Message can play different role. It can only provide an information to receiver but can also trigger an action in the receiver's side.
channels : you can think about it as about The Channel Tunnel which links France to Great Britain. In enterprise integration, channel allows to join two applications and pass a message from one to another. Two different types of channels exist: one-to-one and one-to-many. In the first case, the message is sent from one point to another. In the second the same message will be passed to more than one receivers.
- routes : in some situations messages must pass through different channels before arrive to the receiver. Because sender can do not know which routes follow, it can send a message to Message Router which will be in charge to transfer the data to the final destination. Router won't change anything in the message but only send it from one channel to another.
translators : applications can expect different message formats. For example, sender application can transmit XML message while the receivers expects to work with JSON. In this case the message, before coming to receiver, message will be translated to expected format by one filter.
endpoints : until now we're talking about applications which send and receive messages. To be more precise, these applications must be preceded and terminated by message endpoints. Message endpoints is, in the case of sender application, a service producing a message to send. In the case of receiver application, the endpoints is a service consuming a message.
Another types of message endpoints are: router, splitter (constructs message sequence), aggregator (put together splitted packets to one, final message), service activator (launches action in the receivers side after receiving the message).
To resume this important component, we'll consider that endpoint is all element which makes something with the channel. It can be as well final consumer as intermediary consumer (router, aggregator, splitter...). Application, as well in sender as in receiver side, doesn't know anything about transmitted message. It just knows about the expected message. To understand it, you can read again the sample of translator. Application expects JSON message while sender transmits a XML one. Translator's endpoint takes message formatted in XML, transforms it into JSON format and passes finally to receiver application.
Main concepts of Spring Integration
After previous, global, description, you can think about integration as about underground system helping to transmit data from one application to another. Spring Integration corresponds to this definition by implementing all of previously defined main concepts.
messages : Spring Integration represents message as an implementation of typed org.springframework.integration.Message
instance. It contains two methods: one which returns message's content (getPayload()) and the other to return message headers (getHeaders()). One of basic implementations of Message is immutable org.springframework.integration.message.GenericMessage class. channels : all messages are sent through org.springframework.integration.MessageChannel implementations. The basic interface defines two methods which helps to send a message: send(Message message). The second method is also called send but it contains a supplementary parameter, timeout. Thanks to it we can specify the timeout accepted for message delivery.
From all defined channels, we should pick up the most appropriate. For example, when we expect that our message should be deliver directly to one receiver, we could use DirectChannel. In the case when we want to transmit priorizable messages, we should think about PriorityChannel. But they are only two examples of all available channels.
- routes : messages are routed thanks to the implementations of org.springframework.integration.router.MappingMessageRouterManagement. Its implementation will check different information to decide which channel should treat the message. In this way, PayloadTypeRouter will redirect messages to corresponding channels by watching on payload type which is in fact Message's type. The other class, HeaderValueRouter, will redirect the message by analyzing message's headers. It's also possible to send each messages to a set of channels with RecipientListRouter.
translators : several default transformers are defined in Spring Integration. Among them, we can distinguish following transformers: Object <=> JSON, Object <=> Map or JavaMail message => Spring Message. All of them implement org.springframework.integration.transformer.Transformer which defines the single transform(Message message) method.
endpoints : default message consumers are the implementations of org.springframework.integration.core.MessageHandler interface. It defines handleMessage(Message message) method which treats transmitted message. If the message can't be treated, MessageRejectedException is thrown. When an error occurs on handling, it's MessageHandlingException which is thrown. If more general error occurs, MessagingException is thrown.
The basic class for the majority of message handlers is org.springframework.integration.handler.AbstractMessageHandler which defines internal method to handle message: handleMessageInternal(Message message). As we can suppose, the original handleMessage method is implemented:
@Override public final void handleMessage(Message message) { Assert.notNull(message, "Message must not be null"); Assert.notNull(message.getPayload(), "Message payload must not be null"); if (this.logger.isDebugEnabled()) { this.logger.debug(this + " received message: " + message); } try { if (message != null && this.shouldTrack) { message = MessageHistory.write(message, this, this.getMessageBuilderFactory()); } this.handleMessageInternal(message); } catch (Exception e) { if (e instanceof MessagingException) { throw (MessagingException) e; } throw new MessageHandlingException(message, "error occurred in message handler [" + this + "]", e); } }An example of AbstractMessageHandler's implementation is org.springframework.integration.mail.MailSendingMessageHandler, used to send mails from received Message.
Spring Integration learning method
To learn Spring Integration, we'll explore its features by writing several small applications and prove expected behavior through test cases. The majority of small applications will be based on separated configuration files. Each article will described concepts approached in given sample code.
In this article we discovered the basic elements needed to start working with application integration systems. At the first part we saw the main aspects, as message, channel and endpoints. After that we discovered how these aspects were implemented in Spring Integration. The last paragraph presented shortly a taxi booking system. We'll write it to learn Spring Integration in practice.
Data Engineering Design Patterns
Looking for a book that defines and solves most common data engineering problems? I wrote
one on that topic! You can read it online
on the O'Reilly platform,
or get a print copy on Amazon.
I also help solve your data engineering problems contact@waitingforcode.com đź“©