Azure Data Engineering Notes · Part II — Databricks & Unity Catalog Reference Guide
Security & Secrets Management

Azure Key Vault + Databricks Secrets

How to store a secret securely in Azure Key Vault, expose it to Azure Databricks through a Key Vault-backed secret scope, and retrieve it with dbutils.secrets.get().

The secret itself never leaves Key Vault — Databricks only ever holds a reference to it.

Architecture Overview

The goal is to keep sensitive credentials out of notebooks and source control while still allowing Databricks workloads to access them when needed.

Azure Key Vault
↓
Secret: my-client-secret
↓
Databricks Key Vault-backed Secret Scope
↓
Scope: azure-keyvault-scope
↓
Databricks Notebook
↓
dbutils.secrets.get(...)
Key idea

Azure Key Vault stores the actual secret. Databricks uses a secret scope to reference the vault, and notebook code retrieves the secret by scope name and key.

1. Create the Azure Key Vault

Using Azure CLI:

create the vaultAzure CLI
az keyvault create \
  --name my-data-keyvault \
  --resource-group my-resource-group \
  --location centralus

Then add a secret:

store the secret valueAzure CLI
az keyvault secret set \
  --vault-name my-data-keyvault \
  --name my-client-secret \
  --value "YOUR_SECRET_VALUE"
ItemExample
Key Vaultmy-data-keyvault
Secret namemy-client-secret
Secret valueYOUR_SECRET_VALUE
Security note

Never put the real secret value in Git, a notebook, screenshots, documentation, or other public content.

2. Configure the Key Vault Permission Model

For the traditional Azure Key Vault-backed Databricks secret scope, configure the Key Vault to use the Vault access policy permission model.

Azure Portal
→Key Vault
→Access configuration
→Permission model
→Vault access policy

3. Get Two Values from Key Vault

Open the Key Vault in Azure Portal and go to Properties.

Vault URI / DNS Name

copy this valueVault URI
https://my-data-keyvault.vault.azure.net/

Resource ID

copy this valueResource ID
/subscriptions/.../resourceGroups/my-resource-group/providers/Microsoft.KeyVault/vaults/my-data-keyvault

4. Create a Databricks Secret Scope

In the Azure Databricks workspace, open:

open in your workspaceURL
https://<your-databricks-workspace-url>#secrets/createScope
Important

The createScope portion is case-sensitive.

Enter values similar to the following:

fill in these fieldsDatabricks
Scope Name:
azure-keyvault-scope

Manage Principal:
Creator

Azure Key Vault DNS Name:
https://my-data-keyvault.vault.azure.net/

Resource ID:
/subscriptions/.../providers/Microsoft.KeyVault/vaults/my-data-keyvault

Then select Create.

5. Verify the Secret Scope in Databricks

List the secret scopes available to the notebook:

list available scopesPython
dbutils.secrets.listScopes()

You should see a scope such as:

expected resultOutput
azure-keyvault-scope

Then list the secrets available through that scope:

list secrets in this scopePython
dbutils.secrets.list("azure-keyvault-scope")

This lists the secret names or metadata, not their plaintext values.

For this example, you should find:

expected resultOutput
my-client-secret

6. Retrieve the Secret in a Databricks Notebook

Use dbutils.secrets.get():

retrieve the secretPython
client_secret = dbutils.secrets.get(
    scope="azure-keyvault-scope",
    key="my-client-secret"
)

The mapping is simple:

parameter mappingReference
scope = Databricks secret scope name
key   = Azure Key Vault secret name

A slightly larger example:

using both values togetherPython
client_id = "your-client-id"

client_secret = dbutils.secrets.get(
    scope="azure-keyvault-scope",
    key="my-client-secret"
)
Do not print secrets

Avoid commands such as print(client_secret). Databricks may redact secret values in some notebook output, but redaction should not be treated as a complete security boundary.

Common Use Cases

This pattern is useful when Databricks code needs credentials for:

End-to-End Flow to Remember

Actual secret
↓
Azure Key Vault
↓
Key Vault-backed Databricks secret scope
↓
dbutils.secrets.get()
↓
PySpark / Python notebook code

References