Become a better Data Engineer with waitingforcode.com
Master stream processing
You have a first successful experience with batch data pipelines and were asked to implement your first stream processing jobs?
You shouldn't consider stream processing as a batch on the unbounded data. It's much more than that!
There are various stream-processing concepts. This 3-modules course will show you them from
lead you through them, from the data ingestion to the stateful stream processing!
What I will learn?
- Data ingestion - real-time
- Plan
- API Gateway vs. direct ingestion
- API Gateway π» (theory + demo)
- Change Data Capture π»
- Files streaming π»
- Delivery semantics π»
- Idempotency π»
- Batch layer
- Real-time ad-hoc querying π»
- Polyglot persistence
- Homework
- Data cleansing
- Plan
- Data enrichment π» (theory + demo)
- Data anonymization
- Deduplication π»
- Schema
- Schema registry π»
- Schema management π»
- Metadata π»
- Binary file formats π»
- Monitoring and alerting π»
- Homework
- Stream processing
- Plan
- Patterns π» (theory + demo)
- Architectures
- Transformations π»
- Event time vs. processing time π»
- Scalability π»
- Auto-scaling π»
- Reprocessing π»
- Messaging patterns
- Backpressure π»
- Debugging
- Homework
- Stateful stream processing
- Plan
- Stateless vs. stateful π» (theory + demo)
- State store π»
- Incremental and full state π»
- Triggers π»
- Watermarks and late data π»
- Fault-tolerance π»
- Idempotency π»
- Aggregations π»
- Arbitrary stateful processing π»
- Joins π»
- Windows π»
- Complex Event Processing π»
- Homework
Libraries and tools used in demos: Apache Flink, Apache Kafka, Apache Spark, Delta Lake, Debezium, Kafka Connect, KSQL, ScyllaDB
Enroll in
Watch 3 samples of Master stream processing module
Enroll in
What I will get?
- 10 hours of content
Value $600
Each lesson has a dedicated recording so that you can follow the course on your own pace.
- hands-on homework exercises using modern Open Source technologies
Value $200
Each section ends with a homework assignement where you will have to implement the concepts presented
in the lesson. If you get stuck, you can rely on the instructions document.
If you lack the inspiration,
you can also check the proposed solutions out.
- code snippets in Python and Scala
Value $600
Each demo lesson has dedicated Github repository with code snippets so that you can
play with the code samples on your own.
- lifetime access to the course and material updates
Priceless
Since the data engineering changes, the course will be updated and you'll have
an automatic access to the upgrades.
$1400 Only $98 during the first year
Enroll in
