Azure Data Engineering Lecture Notes

Azure Key Vault + Databricks Secrets

How to store a secret securely in Azure Key Vault, expose it to Azure Databricks through a Key Vault-backed secret scope, and retrieve it with dbutils.secrets.get().

Contents

Architecture Overview

The goal is to keep sensitive credentials out of notebooks and source control while still allowing Databricks workloads to access them when needed.

Azure Key Vault
↓
Secret: my-client-secret
↓
Databricks Key Vault-backed Secret Scope
↓
Scope: azure-keyvault-scope
↓
Databricks Notebook
↓
dbutils.secrets.get(...)
Key idea: Azure Key Vault stores the actual secret. Databricks uses a secret scope to reference the vault, and notebook code retrieves the secret by scope name and key.

1. Create the Azure Key Vault

Using Azure CLI:

az keyvault create \
  --name my-data-keyvault \
  --resource-group my-resource-group \
  --location centralus

Then add a secret:

az keyvault secret set \
  --vault-name my-data-keyvault \
  --name my-client-secret \
  --value "YOUR_SECRET_VALUE"
Item Example
Key Vault my-data-keyvault
Secret name my-client-secret
Secret value YOUR_SECRET_VALUE
Security note: Never put the real secret value in Git, a notebook, screenshots, documentation, or other public content.

2. Configure the Key Vault Permission Model

For the traditional Azure Key Vault-backed Databricks secret scope, configure the Key Vault to use the Vault access policy permission model.

Azure Portal
→ Key Vault
→ Access configuration
→ Permission model
→ Vault access policy

3. Get Two Values from Key Vault

Open the Key Vault in Azure Portal and go to Properties.

Vault URI / DNS Name

https://my-data-keyvault.vault.azure.net/

Resource ID

/subscriptions/.../resourceGroups/my-resource-group/providers/Microsoft.KeyVault/vaults/my-data-keyvault

4. Create a Databricks Secret Scope

In the Azure Databricks workspace, open:

https://<your-databricks-workspace-url>#secrets/createScope
Important: The createScope portion is case-sensitive.

Enter values similar to the following:

Scope Name:
azure-keyvault-scope

Manage Principal:
Creator

Azure Key Vault DNS Name:
https://my-data-keyvault.vault.azure.net/

Resource ID:
/subscriptions/.../providers/Microsoft.KeyVault/vaults/my-data-keyvault

Then select Create.

5. Verify the Secret Scope in Databricks

List the secret scopes available to the notebook:

dbutils.secrets.listScopes()

You should see a scope such as:

azure-keyvault-scope

Then list the secrets available through that scope:

dbutils.secrets.list("azure-keyvault-scope")

This lists the secret names or metadata, not their plaintext values.

For this example, you should find:

my-client-secret

6. Retrieve the Secret in a Databricks Notebook

Use dbutils.secrets.get():

client_secret = dbutils.secrets.get(
    scope="azure-keyvault-scope",
    key="my-client-secret"
)

The mapping is simple:

scope = Databricks secret scope name
key   = Azure Key Vault secret name

A slightly larger example:

client_id = "your-client-id"

client_secret = dbutils.secrets.get(
    scope="azure-keyvault-scope",
    key="my-client-secret"
)
Do not print secrets. Avoid commands such as print(client_secret). Databricks may redact secret values in some notebook output, but redaction should not be treated as a complete security boundary.

Common Use Cases

This pattern is useful when Databricks code needs credentials for:

  • Azure Data Lake Storage (ADLS)
  • Azure SQL databases and data warehouses
  • REST APIs
  • Service principals
  • JDBC connections
  • Other systems that require passwords, access tokens, or keys

End-to-End Flow to Remember

Actual secret
      ↓
Azure Key Vault
      ↓
Key Vault-backed Databricks secret scope
      ↓
dbutils.secrets.get()
      ↓
PySpark / Python notebook code

References