What's new on the cloud for data engineers - part 10 (03-05.2023)

It's time for another part of "What's new on the cloud for data engineers". Let's see what happened in the last 3 months.

Data Engineering Design Patterns

Looking for a book that defines and solves most common data engineering problems? I wrote one on that topic! You can read it online on the O'Reilly platform, or get a print copy on Amazon.

I also help solve your data engineering problems 👉 contact@waitingforcode.com 📩

This 10th part covers all that happened between 10.03.2023 and 27.05.2023. As previously, I highlighted the most interesting news.

AWS

Athena

Aurora

Backup

Batch

Data Sync

Database Migration Service

DocumentDB

DynamoDB

ElastiCache

EMR

Kubernetes:

Security:

Serverless:

Others:

Glue

Crawlers:

Studio:

Others:

Kendra

Keyspaces

Kinesis

Firehose:

Lake Formation

Lambda

Processing:

Ops/Others:

MSK

Others:

Neptune

MemoryDB

Neptune

OpenSearch

RDS

SQL Server:

MySQL:

PostgreSQL:

Global:

Redshift

Others:

S3

Security:

Other features:

SNS

Timestream

QuickSight

Azure

Backup

Batch

Cache for Redis

Containers apps

Cosmos DB

MongoDB:

PostgreSQL

NoSQL

Misc

Data Explorer

Database Migration

Databricks

Event Grid

Event Hubs

Fabric

It's a new end-to-end, unified analytics service on Azure. It integrates other Azure technologies, including Azure Data Factory, Azure Synapse Analytics, and Power BI, into a single unified product. It has 7 different workloads that you can use for various use cases, such as real-time analytics, data orchestration, or data science.

Functions

Monitor

Purview

SQL Database

Hyperscale:

PostgreSQL:

SQL Managed Instance:

MySQL:

SQL Server on VM:

Security:

Misc:

Storage Account

Security:

Misc:

Storage Mover

Stream Analytics

Synapse

GCP

BigQuery

Administration/OPS:

IO:

SQL:

Security:

Other features:

Streaming:

Machine Learning:

BigQuery Transfer Service

Cloud Composer

Cloud Composer 2:

Security:

Bug fixes:

Others:

Cloud Functions

Cloud SQL

SQL Server:

MySQL:

PostgreSQL:

PostgreSQL and MySQL:

Global:

Cloud Storage

Data Loss Protection

New detectors and connections:

Other changes:

Dataflow

Dataplex

Dataproc

Datastream

Firestore

IAM

Pub/Sub

Spanner

Other features:

Querying:

Storage Transfer Service

Data Fabric is the most impactful and changes from the list. Unified set of other services to simplify cloud data stack sounds great! Besides, there are other smaller but also interesting changes, such as Vertical auto scaling for EMR on Kubernetes, Kafka Connect GA in Event Hubs, or GA lineage and CDC support in BigQuery!

Consulting

With nearly 16 years of experience, including 8 as data engineer, I offer expert consulting to design and optimize scalable data solutions. As an O’Reilly author, Data+AI Summit speaker, and blogger, I bring cutting-edge insights to modernize infrastructure, build robust pipelines, and drive data-driven decision-making. Let's transform your data challenges into opportunities—reach out to elevate your data engineering game today!

👉 contact@waitingforcode.com
đź”— past projects


If you liked it, you should read:

📚 Newsletter Get new posts, recommended reading and other exclusive information every week. SPAM free - no 3rd party ads, only the information about waitingforcode!