idempotent consumer articles

on waitingforcode.com
Articles tagged with idempotent consumer. There are 3 article(s) corresponding to the tag idempotent consumer. If you don't find what you're looking for, please check related tags: Cerberus + PySpark, Change Data Capture, hybrid orchestration and coordination, Spark AI Summit Europe 2019 articles, Spark SQL aggregations, Spark SQL and Avro, Structured Streaming Kafka integration.

Check out my new course on Data Engineering!

Are you a data scientist who wants to extend his data engineering skills? Or a software engineer who wants to work with Big Data? If not, maybe a BI developer who wants to evolve to engineering position? My course will help you to achieve your goal! Join the class →

Idempotent consumer with AWS DynamoDB streams

In my previous post I presented an implementation of idempotent consumer pattern with Apache Cassandra CDC. One of drawbacks of that solution was the necessity of producing the messages with slower lightweight transactions. In this post I will show you how to do the same with AWS DynamoDB streams and without that constraint. Continue Reading →

Idempotent consumer pattern

Idempotence is something I appreciate, maybe the most, in data engineering. If you write an idempotent logic you don't need to worry when your logic is reprocessed. You don't need to worry that it will generate duplicates or inconsistent results between runs. However, using it is not always easy and I'm actively looking for all related patterns to it. This time I will focus on idempotent consumer implementation in Apache Camel. Even though it may sounds old-school with modern streaming and messaging solutions, it's a good solution to know. Continue Reading →