What's new on the cloud for data engineers - part 11 (06-09.2023)

It's time for another part of "What's new on the cloud for data engineers". Let's see what happened in the last 4 months.

Looking for a better data engineering position and skills?

You have been working as a data engineer but feel stuck? You don't have any new challenges and are still writing the same jobs all over again? You have now different options. You can try to look for a new job, now or later, or learn from the others! "Become a Better Data Engineer" initiative is one of these places where you can find online learning resources where the theory meets the practice. They will help you prepare maybe for the next job, or at least, improve your current skillset without looking for something else.

👉 I'm interested in improving my data engineering skillset

See you there, Bartosz

This 11th part covers all that happened between 28.05.2023 and 16.09.2023. As previously, I highlighted the most interesting news.

tl;nr. The blog post includes all major changes on the data engineering-like services. If you don't have time passing through all of them, you can find a short list of my top picks for that period:

  1. AWS: Managed Apache Flink in EMR.
  2. AWS: Timestamp-based starting position for Lambda on Apache Kafka.
  3. AWS: Support for querying Apache Iceberg tables
  4. Azure: Entra ID is the new name for Azure Active Directory.
  5. Azure: General availability of the auto-scaling for Stream Analytics jobs.
  6. GCP: Data clean rooms for a simplified data sharing on BigQuery.
  7. GCP: Preview of BigQuery Studio for enhanced data discovery experience.
  8. GCP: Pub/Sub direct synchronization with GCS.
  9. GCP: Event-driven transfers for Storage Transfer Service.

AWS

Athena

Aurora

Batch

Data Sync

Database Migration Service

DocumentDB

DynamoDB

EMR

EventBridge

Timestream

QuickSight

Azure

Backup

Cache for Redis

Cosmos DB

Data Explorer

Database Migration

Event Grid

Functions

HDInsight

Monitor

SQL Database


If you liked it, you should read:

📚 Newsletter Get new posts, recommended reading and other exclusive information every week. SPAM free - no 3rd party ads, only the information about waitingforcode!