Hush! It's a secret on the cloud

How to manage secrets is probably one of the first problems you may encounter while deploying some resources from a CI/CD pipeline. The simple answer is: not manage them at all! Let the cloud services do this.

4-day workshop · In-person or online

What would it take for you to trust your Databricks pipelines in production?

A 3-day bug hunt on a 3-person team costs up to €7,200 in lost engineering time. This workshop teaches you to prevent that — unit tests, data tests, and integration tests for PySpark and Databricks Lakeflow, including Spark Declarative Pipelines.

Unit, data & integration tests

Medallion architecture & Lakeflow SDP

Max 10 participants · production-ready templates

See the full curriculum → €7,000 flat fee · cohort of up to 10

Bartosz
Konieczny

Services

I'll start the presentation with a service mostly used to manage encryption/decryption keys. All of the 3 major cloud providers I'm currently focusing on - AWS, Azure and GCP - have that service. On AWS and GCP it's called KMS (Key Management Service) whereas in Azure you will find it under the name of Key Vault. And despite this different naming, all of them offer similar features:

• integration with other cloud services - secret key stores (common name I'll use for KMS and Key Vault) seamlessly integrate with other cloud services. A pretty common example of the integration is the provision of encryption keys used to data stored in object stores, streaming brokers, or databases.
• lifecycle management - you can manage keys expiration and their automatic rotation.
• versioning - in case of a key rotation, the service preserves older versions. It's very important if a service used the key to encrypt some data. Without the versioning, the key's value would be lost, hence the possibility of decrypting the data.
• authorization - key stores seamlessly integrate with Identity and Access Management (IAM) services. You have then a kind of 2-factor data authorization mechanism because to read the data, any user will need to have access to the encryption keys and the data itself.

Besides the secret key stores that can be useful for encryption purposes, there is a second category of services, this time to manage the secrets. AWS and GCP have dedicated services for that, whereas Azure uses the aforementioned Key Vault. Actually, AWS has even 2 services that can be used to manage the secrets. The first of them is Systems Manager. The service has a Parameter Store capability that integrates with AWS KMS to store secrets in an encrypted format. The second service is Secret Manager. The latter is available under the same name on GCP. Unlike the secret key stores, the secret managers manage real secret data like passwords or access tokens. Although they do rely on the secret key stores to encrypt the persisted secrets but are separate services.

All of them share a set of best practices:

• at least privilege - allow only to read the key(s) needed by the application. For example, if it needs a key to store some data encrypted on an object store, it's not required to give the permission to the keys used to encrypt the streaming broker's data.
• logging - beware of logging. Reading a secret and storing it in logs is not a good idea because it's not secret anymore.
• automatically updated users - if the application stores a reference to the secret and not the secret value itself, you can have an easier maintenance workflow in case of the secret rotation. For example, with the following code snippet you can change the database password at any moment because the consumer will reload the value if it fails with an invalid password error:

  try:
    client.connect_to_db(params)
  except InvalidPasswordError:
    params['password'] = key_store.fetch(params.password)
    client.connect_to_db(params)
  

• automatically revoked users - imagine that your only way to access a data store is the login/password mechanism. For whatever reason, you have to share a single database user credentials across 3 different applications (not a dream, but it happens, often as a "temporary solution" that transforms to the production code :P). You create then 3 separate IAM identities allowed to decrypt the credentials from the store. Now, if you need to revoke the access for one of these applications but keep it online, you can simply change the secret value and revoke the key secrets access permission of the app.

Data services integration

The presented 2 families of services are the most popular ones, but some cloud components provide their secrets management layer. Databricks has a component called secret scope where you can write any secret values and access them with the dbutils.secrets.get function. It works with a separated secrets database but can also be connected to already existing secret key store like Azure Key Vault.

Another service supporting native integration with secrets service is Cloud Composer. It connects with the Secret Manager service. The connection consists of the usage of airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend as the secrets backend. The configuration also has a variables_prefix and connections_prefix entries, both defining the variables and connections the Composer can access. Thanks to these 2 values you can separate the secrets used by other applications from the secrets read by Cloud Composer service.

Finally, on AWS, the Secret Manager natively integrates with the RDS, Redshift, and DocumentDB services to store and automatically rotate the connection credentials.

To complete the list, you can, of course, use KMS/Key Vault as the encryption keys provider for various data stores like object stores, streaming brokers, data warehouses or NoSQL databases.

Secrets are your secret, so do not reveal them to anyone unauthorized. To facilitate this task on the cloud, you can use keys and secrets management services alongside the IAM layer.

Data Engineering Design Patterns

Looking for a book that defines and solves most common data engineering problems? I wrote one on that topic! You can read it online on the O'Reilly platform, or get a print copy on Amazon.

I also help solve your data engineering problems contact@waitingforcode.com 📩