The problem it solves
Before Delta Lake, working with a data lake meant writing raw files — Parquet, CSV, JSON — straight into cloud storage and hoping nothing went wrong in between.
That approach carries real risk once more than one job, or more than one person, touches the same folder.
No transactions
A write job that fails halfway through can leave corrupted or partial data behind, with no way to roll back.
No schema enforcement
Anyone can write a file with a different shape into the same folder, silently breaking every job downstream.
No concurrency safety
A read that overlaps a write can return inconsistent or incomplete results, with nothing to prevent it.
No row-level updates
Plain Parquet files are immutable. Changing one row means rewriting the entire file by hand.
No history
Overwrite the data and the previous version is simply gone — no audit trail, no way back.
How it works underneath
A Delta table is nothing exotic — it's a folder of ordinary Parquet files, sitting next to a log that records every change ever made to them.
Every write — insert, update, delete, merge — adds a new entry to that log rather than touching existing files in place. Readers consult the log to work out exactly which files currently represent the true state of the table.
Nothing here requires a special database engine. Spark — or any engine that understands the Delta protocol — reads the JSON log first, then reads only the Parquet files the log says are still valid.
What the log buys you
Once every change is recorded rather than applied blindly, a handful of guarantees fall out for free.
Every write is atomic — it fully succeeds or fully fails, with no partial state left behind. Enforced through the log using optimistic concurrency control.
& evolution
Writes that don't match the table's schema are rejected by default. Deliberate changes are still allowed, e.g. via the mergeSchema option when adding a column.
Every change is logged, so any earlier version of the table can be queried directly — useful for auditing, debugging, or recovering from a bad write.
A single MERGE INTO statement can insert, update, and delete in one pass — the backbone of CDC pipelines, slowly changing dimensions, and deduplication.
& updates
DELETE and UPDATE rewrite only the files that are actually affected, not the whole table.
The same table can be a streaming sink and a batch source at once, with exactly-once write guarantees either way.
OPTIMIZE compacts small files into larger ones, ZORDER co-locates related data for faster filtering, and VACUUM clears out old files once their history is no longer needed.
Where this sits inside Unity Catalog
Delta Lake is the format. Unity Catalog is the layer that keeps track of where each table's data lives and who is allowed to touch it — the two are not the same thing, even though they're easy to conflate.
| Table type | Data location | On DROP TABLE |
|---|---|---|
| Managed | Default storage tied to the metastore or schema | Metadata and underlying files are both deleted |
| External | A location you specify, via an External Location | Only the metadata entry is removed; files are untouched |
Either way, the files themselves are still Delta — same transaction log, same guarantees, same time travel. The Metastore only tracks where the _delta_log lives; the reliability itself comes from Delta Lake, not from Unity Catalog.
Choosing Managed vs. External is a decision about who owns the file lifecycle. It has no bearing on whether ACID transactions or time travel are available — those come with Delta Lake either way.
Reading, writing, and travelling back
The same three operations cover most day-to-day work with a Delta table.
# Write as Delta — the default format in Databricks df.write.format("delta").mode("overwrite").save( "abfss://raw@<storage-account>.dfs.core.windows.net/sales/" )
df = spark.read.format("delta").load( "abfss://raw@<storage-account>.dfs.core.windows.net/sales/" )
# Time travel — read the table as it looked at version 3 df_old = spark.read.format("delta") \ .option("versionAsOf", 3) \ .load("abfss://raw@<storage-account>.dfs.core.windows.net/sales/")
MERGE INTO target_table t USING source_table s ON t.id = s.id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *;
Delta Lake is a set of Parquet files with a transaction log sitting quietly beside them — and that log is doing all the work.
- Reliability comes from the log, not from any special file format for the data itself.
- Time travel, schema enforcement, and merges are only possible because every change is recorded before it's applied.
- Managed vs. External tables decide who owns the files — both are Delta underneath, with identical guarantees.
OPTIMIZE,ZORDER, andVACUUMexist to keep that reliability from costing you performance over time.