Zarr: Lifecycle, Retention and Optimize
Since: v0.6.0 · (Python level; no JAR needed)
Use this page to keep a store tidy and fast. Writes never change a file. They add new ones, so old
files pile up and a shard can end up spread over many small objects. gbx_zarr_vacuum deletes the files
no readable version needs. gbx_zarr_optimize folds small objects back into whole shards. This page also
explains how long old versions stay readable.
The examples use the names and the sample cube from Zarr Stores.
Lifecycle and retention
Immutable files mean old files pile up. Every update, overwrite and failed write leaves packs that no
current row needs. gbx_zarr_vacuum deletes the files that no readable version needs.

A file is kept while any of these is true:
- live: a current row points at it;
- tagged: a tag row points at it;
- retained: a version within
retainpoints at it (retainis"7 days"unless you set it when you created or registered the store); - recent: it is under one hour old, because the write that made it may still be running. The files of
a write that never committed are kept for the store's
retain, too.
Everything else is superseded or uncommitted:
- superseded: a newer version replaced it, and only versions older than
retainused it; - uncommitted: its write failed, was retried or lost a conflict.
dry_run=True lists what would go and changes nothing. Two checks protect the retain value:
- A
retainunder one hour is refused unlessforce=True. - A
retainlonger than the store's is refused too, since versions older than the store's ownretainare not readable anyway.
Time travel needs two things: the files, and the manifest's own Delta history. When a store is created or registered, GeoBrix sets two properties on the manifest table:
delta.deletedFileRetentionDurationto at least the store'sretain;delta.logRetentionDurationto the larger of 30 days andretain.
Vacuum keeps them aligned. It never lowers a value you raised.
(
spark.sql(f"SHOW TBLPROPERTIES {table}")
.where("key LIKE 'delta.%Retention%' OR key = 'geobrix.zarr.retain'")
.orderBy("key")
.show(truncate=False)
)
+----------------------------------+----------------+
|key |value |
+----------------------------------+----------------+
|delta.deletedFileRetentionDuration|interval 7 days |
|delta.logRetentionDuration |interval 30 days|
|geobrix.zarr.retain |7 days |
+----------------------------------+----------------+
Time travel stops at retain
Time travel stops at retain, with or without a vacuum. GeoBrix promises a version only as far back
as the horizon. The horizon is the later of the horizon a vacuum recorded and the moment retain ago.
A read by version number or by timestamp of a version older than the one that was live at the horizon
raises ZarrVersionUnavailable with a Fix:. This holds even if no vacuum has ever run, because Delta's
own maintenance may already have removed the files. The latest version and every tagged version are
always readable.
A version that changes no data counts as the version before it. Such a version is an OPTIMIZE, a
property change, or the auto compaction Databricks runs after every MERGE. So a version a write or a
restore returned stays readable while only such versions follow it. GeoBrix reads it at the newest of
them, whose files Delta keeps. This store keeps its versions for two seconds so the example is quick (the default is
"7 days"):
import time
from databricks.labs.gbx.zarr import (
ZarrVersionUnavailable,
gbx_zarr_read,
gbx_zarr_tag,
gbx_zarr_write,
)
first = gbx_zarr_read(
spark,
cube,
variables=["t2m", "time", "latitude", "longitude"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T11:00")},
)
# retain is "7 days" unless you say otherwise; two seconds keeps this example short.
gbx_zarr_write(first, table, mode="create", root=root, retain="2 seconds")
gbx_zarr_tag(spark, table, "first-day") # a second version, and a name for it
time.sleep(3)
try:
gbx_zarr_read(spark, table, variables=["t2m"], version_as_of=1)
except ZarrVersionUnavailable as err:
print("Fix:", err.fix)
tagged = gbx_zarr_read(
spark,
table,
variables=["t2m"],
sel={"latitude": 48.0, "longitude": 2.0},
version_as_of="tag:first-day",
output="table",
)
print("tag first-day:", tagged.count(), "hours")
Fix: pick a later version or tag the version you need with gbx_zarr_tag
tag first-day: 12 hours
Predictive optimization can run VACUUM on a Unity Catalog managed table by itself. In GeoBrix's own
checks it kept a table's files for at least 7 days, even when the table asked for less. A store whose
retain is shorter than that can still find Delta history for a version whose files are gone. That is why
the horizon, and not Delta's history, decides what is readable. A vacuum that deletes records its horizon
on the table before it removes anything. To keep a version for good, tag it.
Run a vacuum
The vacuum below ran a week after the store was written. (In the documentation tests, the clock is moved on.) It lists first, then deletes. The tag keeps its files, and the live rows keep theirs. The packs that only older versions used go. So does the pack of the conflict example's losing write, which never committed:
from databricks.labs.gbx.zarr import gbx_zarr_vacuum
plan = gbx_zarr_vacuum(spark, table, dry_run=True) # lists; deletes nothing
plan.groupBy("action", "reason").count().orderBy("action", "reason").show()
gbx_zarr_vacuum(spark, table)
+------------+-----------+-----+
| action| reason|count|
+------------+-----------+-----+
| kept| live| 5|
| kept| tagged| 8|
|would_delete| superseded| 7|
|would_delete|uncommitted| 1|
+------------+-----------+-----+
A vacuum only touches the pack files GeoBrix wrote under the store's own root. It never deletes the objects of a registered store, anything else in a root, or anything in a directory that another store owns. A dry run changes nothing at all.
Chunk rows and shard rows
Zarr v3 groups many small inner chunks into one shard object. The manifest has a row for each of two things:
- a shard row points at a whole shard in one object: that is how a registered store of shards begins;
- a chunk row points at one inner chunk inside a pack file.

A write adds chunk rows, one for each chunk it changes, in packs it has just written. That keeps a small append or update cheap, because nothing already written is rewritten. A new managed store is chunk rows only. The price is on the read side. A shard is now spread over many small objects, and on a Volume every object costs a round trip.
spark.sql(f"""
SELECT kind, count(*) AS rows FROM {table}
WHERE array = 't2m' AND kind IN ('shard', 'chunk') AND tag IS NULL
GROUP BY kind ORDER BY kind""").show()
+-----+----+
| kind|rows|
+-----+----+
|chunk| 8|
+-----+----+
Fold chunk rows back with optimize
gbx_zarr_optimize folds chunk rows back into shard rows. For every shard that has chunk rows, it:
- copies the shard's bytes, without decoding them, into one new pack as a standard Zarr v3 shard;
- commits one shard row in place of the chunk rows, with the same compare-and-swap as a write;
- folds deleted chunks away.
A shard whose rows changed while the optimize ran is skipped, with a warning, and folded the next time.
gbx_zarr_write(..., optimize=True) folds the shards a write touched right after it commits. Run optimize
after many small appends, especially for reads from a Volume. It changes no cell.
from databricks.labs.gbx.zarr import gbx_zarr_optimize
version = gbx_zarr_optimize(spark, table)
spark.sql(f"""
SELECT kind, count(*) AS rows FROM {table}
WHERE array = 't2m' AND kind IN ('shard', 'chunk') AND tag IS NULL
GROUP BY kind ORDER BY kind""").show()
+-----+----+
| kind|rows|
+-----+----+
|shard| 1|
+-----+----+
Next Steps
- Versions, time travel and tags: tag a version so a vacuum keeps it.
- Performance and limits: what layout and compute mean for read and write times.
gbx_zarr_vacuumandgbx_zarr_optimize: every parameter.