Skip to main content

Zarr: Lifecycle, Retention and Optimize

Since: v0.6.0 · (Python level; no JAR needed)

Use this page to keep a store tidy and fast. Writes never change a file. They add new ones, so old files pile up and a shard can end up spread over many small objects. gbx_zarr_vacuum deletes the files no readable version needs. gbx_zarr_optimize folds small objects back into whole shards. This page also explains how long old versions stay readable.

The examples use the names and the sample cube from Zarr Stores.

Lifecycle and retention​

Immutable files mean old files pile up. Every update, overwrite and failed write leaves packs that no current row needs. gbx_zarr_vacuum deletes the files that no readable version needs.

The timeline of a vacuum. Versions v1 to v8 run left to right. The retain window, seven days by default, is shaded at the right. It starts at the horizon, which is the later of the last vacuum's horizon and the moment retain ago. Version v4 is live at the horizon and stays readable. Versions older than it raise ZarrVersionUnavailable. A pack replaced before the horizon is superseded and deleted. A pack replaced inside the window is retained, because a readable version, v4, still uses it. Packs that live rows point at are live. A pack that a tag row points at is tagged and kept until the tag is removed. The pack of a write that never committed is uncommitted. It is kept for the retain period and then deleted. A pack younger than one hour is recent and always kept. Below the timeline, the manifest table's own retention properties keep Delta history at least as long as the store's retain. Predictive optimization may vacuum the table itself.

A file is kept while any of these is true:

  • live: a current row points at it;
  • tagged: a tag row points at it;
  • retained: a version within retain points at it (retain is "7 days" unless you set it when you created or registered the store);
  • recent: it is under one hour old, because the write that made it may still be running. The files of a write that never committed are kept for the store's retain, too.

Everything else is superseded or uncommitted:

  • superseded: a newer version replaced it, and only versions older than retain used it;
  • uncommitted: its write failed, was retried or lost a conflict.

dry_run=True lists what would go and changes nothing. Two checks protect the retain value:

  • A retain under one hour is refused unless force=True.
  • A retain longer than the store's is refused too, since versions older than the store's own retain are not readable anyway.

Time travel needs two things: the files, and the manifest's own Delta history. When a store is created or registered, GeoBrix sets two properties on the manifest table:

  • delta.deletedFileRetentionDuration to at least the store's retain;
  • delta.logRetentionDuration to the larger of 30 days and retain.

Vacuum keeps them aligned. It never lowers a value you raised.

(
spark.sql(f"SHOW TBLPROPERTIES {table}")
.where("key LIKE 'delta.%Retention%' OR key = 'geobrix.zarr.retain'")
.orderBy("key")
.show(truncate=False)
)
Example output
+----------------------------------+----------------+
|key |value |
+----------------------------------+----------------+
|delta.deletedFileRetentionDuration|interval 7 days |
|delta.logRetentionDuration |interval 30 days|
|geobrix.zarr.retain |7 days |
+----------------------------------+----------------+

Time travel stops at retain​

Time travel stops at retain, with or without a vacuum. GeoBrix promises a version only as far back as the horizon. The horizon is the later of the horizon a vacuum recorded and the moment retain ago.

A read by version number or by timestamp of a version older than the one that was live at the horizon raises ZarrVersionUnavailable with a Fix:. This holds even if no vacuum has ever run, because Delta's own maintenance may already have removed the files. The latest version and every tagged version are always readable.

A version that changes no data counts as the version before it. Such a version is an OPTIMIZE, a property change, or the auto compaction Databricks runs after every MERGE. So a version a write or a restore returned stays readable while only such versions follow it. GeoBrix reads it at the newest of them, whose files Delta keeps. This store keeps its versions for two seconds so the example is quick (the default is "7 days"):

import time

from databricks.labs.gbx.zarr import (
ZarrVersionUnavailable,
gbx_zarr_read,
gbx_zarr_tag,
gbx_zarr_write,
)

first = gbx_zarr_read(
spark,
cube,
variables=["t2m", "time", "latitude", "longitude"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T11:00")},
)
# retain is "7 days" unless you say otherwise; two seconds keeps this example short.
gbx_zarr_write(first, table, mode="create", root=root, retain="2 seconds")
gbx_zarr_tag(spark, table, "first-day") # a second version, and a name for it
time.sleep(3)
try:
gbx_zarr_read(spark, table, variables=["t2m"], version_as_of=1)
except ZarrVersionUnavailable as err:
print("Fix:", err.fix)
tagged = gbx_zarr_read(
spark,
table,
variables=["t2m"],
sel={"latitude": 48.0, "longitude": 2.0},
version_as_of="tag:first-day",
output="table",
)
print("tag first-day:", tagged.count(), "hours")
Example output
Fix: pick a later version or tag the version you need with gbx_zarr_tag
tag first-day: 12 hours

Predictive optimization can run VACUUM on a Unity Catalog managed table by itself. In GeoBrix's own checks it kept a table's files for at least 7 days, even when the table asked for less. A store whose retain is shorter than that can still find Delta history for a version whose files are gone. That is why the horizon, and not Delta's history, decides what is readable. A vacuum that deletes records its horizon on the table before it removes anything. To keep a version for good, tag it.

Run a vacuum​

The vacuum below ran a week after the store was written. (In the documentation tests, the clock is moved on.) It lists first, then deletes. The tag keeps its files, and the live rows keep theirs. The packs that only older versions used go. So does the pack of the conflict example's losing write, which never committed:

from databricks.labs.gbx.zarr import gbx_zarr_vacuum

plan = gbx_zarr_vacuum(spark, table, dry_run=True) # lists; deletes nothing
plan.groupBy("action", "reason").count().orderBy("action", "reason").show()
gbx_zarr_vacuum(spark, table)
Example output
+------------+-----------+-----+
| action| reason|count|
+------------+-----------+-----+
| kept| live| 5|
| kept| tagged| 8|
|would_delete| superseded| 7|
|would_delete|uncommitted| 1|
+------------+-----------+-----+

A vacuum only touches the pack files GeoBrix wrote under the store's own root. It never deletes the objects of a registered store, anything else in a root, or anything in a directory that another store owns. A dry run changes nothing at all.

Chunk rows and shard rows​

Zarr v3 groups many small inner chunks into one shard object. The manifest has a row for each of two things:

  • a shard row points at a whole shard in one object: that is how a registered store of shards begins;
  • a chunk row points at one inner chunk inside a pack file.

Chunk rows over a shard row, in three stages. First, a store of shards has one shard row pointing at one shard object that holds eight inner chunks. After appends and updates, the shard row is still there. Chunk rows laid over it point at replacement chunks in small packs. One chunk is a deleted tombstone. A read lays the chunk rows over the shard's bytes. After optimize, a single shard row points at a new pack that holds all eight inner chunks again. The chunk rows and the deleted chunk are folded away.

A write adds chunk rows, one for each chunk it changes, in packs it has just written. That keeps a small append or update cheap, because nothing already written is rewritten. A new managed store is chunk rows only. The price is on the read side. A shard is now spread over many small objects, and on a Volume every object costs a round trip.

spark.sql(f"""
SELECT kind, count(*) AS rows FROM {table}
WHERE array = 't2m' AND kind IN ('shard', 'chunk') AND tag IS NULL
GROUP BY kind ORDER BY kind""").show()
Example output
+-----+----+
| kind|rows|
+-----+----+
|chunk| 8|
+-----+----+

Fold chunk rows back with optimize​

gbx_zarr_optimize folds chunk rows back into shard rows. For every shard that has chunk rows, it:

  1. copies the shard's bytes, without decoding them, into one new pack as a standard Zarr v3 shard;
  2. commits one shard row in place of the chunk rows, with the same compare-and-swap as a write;
  3. folds deleted chunks away.

A shard whose rows changed while the optimize ran is skipped, with a warning, and folded the next time. gbx_zarr_write(..., optimize=True) folds the shards a write touched right after it commits. Run optimize after many small appends, especially for reads from a Volume. It changes no cell.

from databricks.labs.gbx.zarr import gbx_zarr_optimize

version = gbx_zarr_optimize(spark, table)
spark.sql(f"""
SELECT kind, count(*) AS rows FROM {table}
WHERE array = 't2m' AND kind IN ('shard', 'chunk') AND tag IS NULL
GROUP BY kind ORDER BY kind""").show()
Example output
+-----+----+
| kind|rows|
+-----+----+
|shard| 1|
+-----+----+

Next Steps​