Skip to main content

Zarr Stores: Concepts and Store Kinds

Since: v0.6.0 · (Python level; no JAR needed)

Zarr stores N-dimensional arrays as many small objects in object storage. That makes it the format of choice for large weather, climate and earth-observation cubes. Plain Zarr has no transactions. Two jobs that write to one store can lose each other's updates, and a reader can see a half-written store.

GeoBrix adds the missing layer:

  • Read and sample Zarr stores of any origin.
  • Register an existing store in place, without copying it.
  • Write transactionally. Every write is all-or-nothing, and concurrent writers do not corrupt each other.
  • Keep every version readable for the store's retain period (7 days by default). Tag a version to keep it for as long as you like.

This page is the place to start. It explains the four kinds of store and how a store is kept as files and rows. The examples run against a small sample cube, and the documentation tests run every block of code.

See it end to end

The Weather Features with Zarr notebook runs these ideas on a small weather cube. It registers the cube in place and samples it at 100 sites. It writes a derived variable back as a new version, maps it as raster tiles, and tags the version for training.

Store kinds at a glance​

A store is plain, registered (read-only or writable) or managed. Store kinds explains each one.

The four kinds of Zarr store side by side. A plain store has only its own objects and no manifest table. A read-only registered store has a Delta manifest table that points at the store's objects without copying them. A writable registered store has a manifest that points at the store's objects and at an overlay root of immutable pack files. The store's objects are locked against writes. A managed store has a manifest that points at pack files under a data root. A banner explains that many registrations of one store are independent branches. It also explains that every root has exactly one owner.

Which page do you need?​

You want toGo to
Look up a function, its parameters and its errorsZarr Function Reference
Read a cube, or sample it at pointsZarr Reader
Write blocks to a storeZarr Writer
Point a table at a store you already have, and keep it in syncRegistering and syncing
Know what happens when several jobs write at onceTransactions and concurrent writers
Read an older version, name one, or take one outVersions, time travel and tags
Free storage, or make reads faster after many small writesLifecycle and retention
See what a store costs, and what is not supportedPerformance and limits

Why Zarr​

ZarrGeoTIFF / COGNetCDF / HDF5GRIB2Delta table, one row per cell
ShapeAny number of dimensionsBands of a 2-D gridAny number of dimensionsA sequence of 2-D fieldsRows and columns; the grid is lost
LayoutOne object per chunk, or per shard of chunksOne file; a COG is tiled insideOne file, chunked insideOne file of messagesParquet files
Reading a slice from object storageFetches only the chunks the slice touchesA COG fetches the tiles it needsPossible with range-read tooling; many readers copy the whole fileNeeds an index of the messagesScans and filters rows
Several writers at onceSafe only for different chunks, and only if none changes the metadataOne file, one writerOne file, one writerOne file, one writerYes (Delta transactions)
All-or-nothing writesNoNoNoNoYes
Versions and time travelNoNoNoNoYes
Best forCubes read and written at scaleImagery and single surfacesFiles from data providersWeather model outputTabular analytics on cells

Zarr's strength is the layout. A job reads only the chunks it needs, and many jobs read different chunks at once. Its weakness is that the layout has no commit. GeoBrix keeps the layout and adds the commit. A store then gets the guarantees of the last column without turning an array into rows.

What GeoBrix adds

  • Register in place. Point a table at an existing store; nothing is copied.
  • Three ways to read. Read N-dimensional blocks, a long table with a column per dimension, or GeoBrix raster tiles. With tiles, every rst_* function works on a cube without converting it to GeoTIFF. Select by label or by position. See the Zarr Reader.
  • Sample. Read the time series at points, with the same geometry inputs as rst_sample (sampling at points).
  • Write back transactionally. Create, append, update and overwrite, with history, time travel, tags, restore, export and lifecycle management. See the Zarr Writer.
  • Concurrent writers. Writers that touch different chunks both land. Writers that only change the store's shape or coordinates are rebased automatically. Writers that meet on one chunk get a clear error.

What you need​

Install the env6 wheel with its [zarr] extra. Zarr runs on Databricks Runtime 19 and Serverless environment 6, at the Python level, so there is no JAR to install.

%pip install "geobrix[zarr] @ file:///Volumes/<catalog>/<schema>/<volume>/geobrix-<version>+env6-py3-none-any.whl"

The examples use three names:

  • cube is the path of a Zarr store.
  • table is the Unity Catalog name of a store's manifest table.
  • root is a directory on a Volume that GeoBrix writes a store's files to.

The cube path below exists once the sample cube has been generated into your sample-data Volume (next). Until then, point cube at any Zarr store.

# The sample cube, a name for the store's manifest table, and a Volume directory for its files.
cube = "/Volumes/main/geobrix_samples/geobrix-examples/global/era5-weather/era5_like_v3.zarr"
table = "main.weather.era5"
root = "/Volumes/main/weather/stores/era5"

The sample cube is two days of made-up hourly weather on a one-degree global grid: time (48 steps) by latitude (181) by longitude (360). Its variables are t2m (2 m temperature), u10 and v10 (10 m wind) and msl (mean sea level pressure). It is generated, not downloaded, and takes seconds:

from databricks.labs.gbx.sample.zarr_era5 import write_era5_cube

written = write_era5_cube(out_dir)

It writes the cube twice: era5_like_v3.zarr (Zarr v3 with sharding) and era5_like_v2.zarr (Zarr v2).

Read and sample a cube​

A selection reads as a table, one row per cell, with a column per dimension and per variable:

from databricks.labs.gbx.zarr import gbx_zarr_read

europe = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={
"time": slice("2023-01-01T00:00", "2023-01-01T02:00"),
"latitude": slice(
60, 40
), # labels, both ends included; latitude runs north to south
"longitude": slice(0, 30),
},
output="table",
)
europe.orderBy("time", "latitude", "longitude").show(3)
Example output
+-------------------+--------+---------+-----+
| time|latitude|longitude| t2m|
+-------------------+--------+---------+-----+
|2023-01-01 00:00:00| 40.0| 0.0|299.5|
|2023-01-01 00:00:00| 40.0| 1.0|299.5|
|2023-01-01 00:00:00| 40.0| 2.0|299.5|
+-------------------+--------+---------+-----+
only showing top 3 rows

Label slices include both ends and follow the axis order, which is why latitude runs from 60 down to 40. Points are sampled the same way, at the nearest cell:

from databricks.labs.gbx.zarr import gbx_zarr_sample

points = spark.createDataFrame(
[("paris", "POINT(2.0 48.0)"), ("london", "POINT(0.0 51.0)")], ["name", "geom"]
)
series = gbx_zarr_sample(
spark,
points,
table,
variables=["t2m"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T02:00")},
)
series.orderBy("name", "time").show(truncate=False)
Example output
+------+---------------+-------------------+------+
|name |geom |time |t2m |
+------+---------------+-------------------+------+
|london|POINT(0.0 51.0)|2023-01-01 00:00:00|297.5 |
|london|POINT(0.0 51.0)|2023-01-01 01:00:00|298.5 |
|london|POINT(0.0 51.0)|2023-01-01 02:00:00|299.5 |
|paris |POINT(2.0 48.0)|2023-01-01 00:00:00|298.25|
|paris |POINT(2.0 48.0)|2023-01-01 01:00:00|299.25|
|paris |POINT(2.0 48.0)|2023-01-01 02:00:00|300.25|
+------+---------------+-------------------+------+

Store kinds​

A Zarr store GeoBrix works with is one of four kinds. They differ in where the bytes live, whether a manifest table exists, and who may write. The diagram in Store kinds at a glance shows the four side by side.

KindCreated byWhere the bytes liveVersions and transactionsWho may write
Plain storeAnyone: xarray, a vendor, any Zarr tool, or gbx_zarr_write(..., transactional=False)The store's own objectsNone. GeoBrix reads the current bytes.Anyone, one writer at a time. GeoBrix writes one only with the opt-in
Registered, read-onlygbx_zarr_register(spark, path, table)The store's own objects, not copiedYes, in the manifest tableNobody through GeoBrix
Registered, writablegbx_zarr_register(spark, path, table, writable=True, overlay_root=...)The store's own objects, plus an overlay of new pack filesYesAnyone with MODIFY on the table and write access to the overlay root's Volume; the store itself is never written
Managedgbx_zarr_write(..., mode="create")Pack files GeoBrix wrote under the store's rootYesAnyone with MODIFY on the table and write access to the root's Volume

A plain store has no manifest, so no versions and no guard against a second writer. GeoBrix writes one only when you ask: transactional=False creates or appends to it, and no table exists for it.

Use a managed store when GeoBrix creates the data. Use a registered store when the data already exists and other tools keep using it. Registering and syncing covers the registered kinds.

How a store is made of files and rows​

Every byte GeoBrix writes goes into a new pack file that is never modified again. Pack files sit under data/<array>/<write_id>/part-*.zpack in the store's root.

A manifest is a Delta table with:

  • one row per chunk or shard, and per array and group document, that says which file, offset and length hold it;
  • one commit row for every write.

A version of the store is a version of that Delta table, and one Delta transaction is one store transaction. Because files are immutable and rows are versioned, a reader that pins a version sees a consistent snapshot no matter what writers do meanwhile.

A shard row points at a whole shard. A chunk row points at one chunk inside a pack file. Appends and updates add chunk rows, and gbx_zarr_optimize folds them back into shard rows; see chunk rows and shard rows.

Create a managed store​

A managed store is made by writing blocks. This example writes the first 12 hours of two variables, together with the coordinates that label them:

from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write

first = gbx_zarr_read(
spark,
cube,
variables=["t2m", "msl", "time", "latitude", "longitude"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T11:00")},
)
# "create" makes a managed store: `table` is its manifest, `root` holds its files.
version = gbx_zarr_write(first, table, mode="create", root=root)

The new array layout defaults to chunks of about 4 MiB decoded, grouped into shards of up to 128 MiB. Its codecs are bytes, zstd (level 3) and crc32c. chunks takes a preset or a size per dimension:

  • "maps" for large spatial tiles;
  • "series" for long time series at small tiles;
  • "balanced" for both.

The next step is usually Transactions, which covers appends, updates and concurrent writers.

File modes​

file_mode decides how the manifest refers to a store's files, and so who deletes superseded files. It matches the file_mode of gbx_file_write.

file_modeThe manifest's file columnWho deletes superseded filesWhere it runs
fuse (default)Path strings, relative to the store's rootGeoBrix gbx_zarr_vacuumEverywhere, Serverless included
externalFILE EXTERNAL references, governed with the tableGeoBrix gbx_zarr_vacuumCompute that supports FILE values, in a workspace with the File type preview (preview)
managedFILE MANAGED valuesThe tableNot available in this release
autoexternal on compute that supports FILE values (Serverless included, in a workspace with the File type preview), otherwise fuse; it never fails

The default is fuse, so a store made on FILE-capable compute still reads on every compute. Pass file_mode="auto" or "external" to keep FILE references.

managed is not available because Delta's VACUUM does not reclaim the bytes of a managed file, so a managed store would only grow. A registered store is never managed: managed ingest would copy the store.

A FILE store needs FILE-capable compute. A store whose files are FILE values (external) must be written, maintained and read on compute that supports them. Other compute refuses up front with a Fix: that tells you what to do. You have three options:

  • Use a Databricks Runtime 19 dedicated cluster (single user, with fileReferenceCreationMode=MANAGED) in a workspace with the preview.
  • Copy the data out with gbx_zarr_export(method="copy"), and register the copy with file_mode="fuse".
  • Create stores that other compute also uses with file_mode="fuse".

Reads of a store return the same cells in every mode.

from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write

first = gbx_zarr_read(
spark,
cube,
variables=["t2m", "time", "latitude", "longitude"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T11:00")},
)
# "fuse" (the default): the manifest keeps plain path strings. It works on every compute.
gbx_zarr_write(first, table, mode="create", root=root, file_mode="fuse")

Asking for managed fails before anything is written:

from databricks.labs.gbx.zarr import (
ZarrUnsupported,
gbx_zarr_read,
gbx_zarr_write,
)

first = gbx_zarr_read(
spark,
cube,
variables=["t2m", "time", "latitude", "longitude"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T11:00")},
)
try:
gbx_zarr_write(first, table, mode="create", root=root, file_mode="managed")
except ZarrUnsupported as err:
print(err)
Example output
file_mode="managed" is not available in this release: Delta VACUUM does not reclaim the bytes of a managed file (nor does a deletion-vector delete or REORG PURGE), so a managed store would only grow Fix: use file_mode="fuse" (the default), "auto" or "external"

Next Steps​