Skip to main content

Zarr Writer

Since: gbx_zarr_write v0.6.0

gbx_zarr_write writes a Zarr store transactionally. One write is one Delta MERGE on the store's manifest table, so it is visible whole or not at all. Writers that touch different chunks both land. Writers that meet on one chunk get a clear error. This page is the how-to: the modes, the inputs, the layout of new arrays, what happens when writers meet, and publishing a registered store. Zarr Stores explains the model, and the Zarr Function Reference lists every parameter and error. To read a store back see the Zarr Reader.

It runs at the Python level, so there is no JAR to install: install the env6 wheel with its [zarr] extra, as Zarr Stores describes. The examples use these names:

# The sample cube, a name for the store's manifest table, and a Volume directory for its files.
cube = "/Volumes/main/geobrix_samples/geobrix-examples/global/era5-weather/era5_like_v3.zarr"
table = "main.weather.era5"
root = "/Volumes/main/weather/stores/era5"

table is the manifest table of the store, and root is a directory on a Volume that GeoBrix owns for that store's files. Every code block on this page is run by the documentation tests, against the same sample cube as the other Zarr pages.

Modes​

modeDoesThe input must
"create"Makes a new managed store: the table target and the data root rootCarry the blocks, Dataset or tiles of the new arrays. Name the coordinate arrays too, or the dimensions are labelled by position only
"append"Extends one dimension (append_dim, by default the first dimension labelled by CF times)Carry that dimension's labels, all after the store's last
"update"Writes into the existing extentSay where it goes: blocks by their start and stop, a Dataset by its labels, tiles by their grid
"overwrite"Replaces the named variablesThe variables to replace. Chunks the input does not cover read as the fill value afterwards

This page describes the managed (transactional) store, the default; Plain stores covers the opt-in plain write.

root, chunks, shards, codecs, file_mode and retain belong to mode="create": in the other modes they raise ValueError with a Fix: (a later write may repeat the store's own file_mode). A write into a registered store needs the store to be registered with writable=True. It goes to the store's overlay root, in the base arrays' own codecs and key encoding. The store itself is never written (see Registered stores).

Old versions stay readable until vacuumed: an overwrite does not destroy what it replaces.

Inputs​

data is one of four things.

Blocks: a DataFrame in the schema gbx_zarr_read returns (BLOCKS_SCHEMA). With decode=True values (the default of a read) the writer applies the array's CF attributes the other way. It converts times back to the array's units, packed values to their integer type, and NaN to the fill value. Blocks that still carry the array's packing attributes (a read with decode=False) are written as the stored values they are.

An xarray.Dataset, held on the driver, so keep it to what fits there. Pass spark= because a Dataset has no session. An update places it by its coordinate labels, which may include steps an out-of-order append left unlabelled.

GeoBrix tiles: a DataFrame with one tile column, found by its schema (other columns may stay). They have a section of their own, Tiles.

A table of cells: a DataFrame with one row per cell, a column per dimension (named in dim_cols) and a column per variable. See Write from a table of cells.

Layouts​

A new array is laid out by three arguments of mode="create":

ArgumentValuesDefault
chunks"balanced", "maps", "series" or a dict of dimension to size"balanced"
shardsNone computes them, a dict sets some dimensions, False means no shardsNone
codecsThe inner chunk's codec chain, as Zarr v3 codec documents (dicts) or zarr codec objects, starting with a serializer such as {"name": "bytes"}bytes, zstd (level 3), crc32c

An inner chunk is the unit of reading and writing, about 4 MiB decoded. A shard groups inner chunks into one object of at most 128 MiB decoded and 4096 chunks. On a Volume every object costs a round trip. The last two dimensions are the spatial ones; any before them are leading dimensions (time, level, ...).

PresetChunkBest for
"maps"One leading step, large spatial tilesA map at a time
"series"32 x 32 spatial tiles, a long run of leading stepsThe time series of a point
"balanced"24 leading steps, spatial tiles sized to fill the restBoth

Only the first leading dimension takes the preset's step; any other leading dimension has chunks of 1. A chunks dict sets the dimensions it names, as given (an append dimension may want chunks longer than today's data). It leaves the others at the "balanced" size. Presets are clipped to a dimension's length. A shard is always a whole number of chunks. The leading dimension's shard grows by whole chunks past the current length. A store created small and appended to along time therefore keeps filling the same shards. Coordinates are written as one unsharded chunk, long enough that appends keep landing in it.

from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write

first = gbx_zarr_read(
spark,
cube,
variables=["t2m", "msl", "time", "latitude", "longitude"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T11:00")},
)
version = gbx_zarr_write(
first,
table,
mode="create",
root=root,
chunks={"time": 12, "latitude": 91, "longitude": 180}, # inner chunks, in cells
shards={"time": 24}, # a shard of two chunks along time; the rest is computed
codecs=[ # the inner chunk's codec chain, as Zarr v3 codec documents
{"name": "bytes", "configuration": {"endian": "little"}},
{"name": "zstd", "configuration": {"level": 9, "checksum": False}},
{"name": "crc32c"},
],
)

The layout is in the array's document, which is a row of the manifest table:

import json

row = spark.sql(
f"SELECT meta FROM {table} WHERE kind = 'array' AND array = 't2m'"
).first()
document = json.loads(row["meta"])
sharding = document["codecs"][0]["configuration"]
print("shape: ", document["shape"])
print("shard: ", document["chunk_grid"]["configuration"]["chunk_shape"])
print("chunk: ", sharding["chunk_shape"])
print("codecs:", [codec["name"] for codec in sharding["codecs"]])
Example output
shape: [12, 181, 360]
shard: [24, 182, 360]
chunk: [12, 91, 180]
codecs: ['bytes', 'zstd', 'crc32c']

create needs root not to exist, or to be empty. It gets an ownership ledger and the _gbx_store.json marker. The ledger is the small Delta table _gbx_owner. It decides which table owns the root even when several claim it at once. See One owner per root.

Two settings are the store's from then on. file_mode is "fuse" by default; "auto" or "external" keep FILE references; "managed" is not available in this release. retain is how long superseded versions stay readable, "7 days" by default. See File modes. A store in external mode keeps FILE values in its manifest. It can be written, read and maintained only on compute that supports them. Other compute refuses before it writes anything.

If a create fails after it made the table but before anything committed, run it again with the same arguments. A create on a manifest you own that has no commit yet adopts it. Two create calls that claim one root at the same moment cannot both win, and the loser never releases the marker the winner relies on.

Creating from an xarray Dataset​

import xarray as xr

from databricks.labs.gbx.zarr import gbx_zarr_write

cube_ds = xr.open_zarr(cube)
first = cube_ds[["t2m"]].isel(time=slice(0, 12)).load()
version = gbx_zarr_write(first, table, mode="create", root=root, spark=spark)

Name the coordinates​

A create from blocks labels a dimension only if the blocks carry its coordinate array. Read the coordinates with the variables (variables=["t2m", "time", "latitude", "longitude"], as the examples above do). Without them the write still succeeds and logs a warning, "the blocks carry no coordinate for ...". The new store labels those dimensions by position only.

Appending, updating and overwriting​

An append labels the new steps. They must come after the store's last. On a regular axis they land at their step, so appends may arrive out of order. On an irregular axis they land at the current length. Zarr transactions shows an append, an update and a conflict.

An update writes into the existing extent. An inner chunk the input covers whole is encoded without being read. A chunk it covers only in part is read at the base version, patched and written again. A write_id (32 lowercase hex characters) makes a write idempotent: writing again with one that has already committed returns its version and writes nothing.

An overwrite replaces the variables it names:

from databricks.labs.gbx.zarr import (
BLOCKS_SCHEMA,
block_to_numpy,
gbx_zarr_read,
gbx_zarr_write,
)

rows = gbx_zarr_read(spark, table, variables=["t2m"]).collect()
corrected = [
{
**row.asDict(),
"data": (block_to_numpy(row) - 0.5).astype("float32").tobytes(),
}
for row in rows
]
version = gbx_zarr_write(
spark.createDataFrame(corrected, BLOCKS_SCHEMA),
table,
mode="overwrite",
variables=["t2m"], # the variables to replace
)

variables=: naming what you write​

variables takes a name or a list of names. In an update or overwrite from blocks it does two jobs:

  • It is required to create a variable the store lacks. A new variable is created on the store's dimensions, with the layout and codecs of a new array and the attributes of its first block. Without variables= the writer only knows the arrays the store already has, and refuses a block of any other.
  • It narrows the work. Without it every decodable array of the store is a candidate. The write reads the manifest rows of all of them and moves them through its shuffle. Only then does it keep the rows of the chunks its input touches. Naming the variables leaves the other arrays' rows out from the start, so on a large store name the variables you write.
import json

from databricks.labs.gbx.zarr import (
BLOCKS_SCHEMA,
block_to_numpy,
gbx_zarr_history,
gbx_zarr_read,
gbx_zarr_write,
)

rows = gbx_zarr_read(spark, table, variables=["t2m"]).collect()
celsius = [
{
**row.asDict(),
"variable": "t2m_c",
"data": (block_to_numpy(row) - 273.15).astype("float32").tobytes(),
"attrs": json.dumps({**json.loads(row["attrs"]), "units": "degC"}),
}
for row in rows
]
version = gbx_zarr_write(
spark.createDataFrame(celsius, BLOCKS_SCHEMA),
table,
mode="update",
variables=["t2m_c"], # required: the store has no such array yet
)
history = gbx_zarr_history(spark, table)
history.select("version", "mode", "variables").orderBy("version").show(
truncate=False
)
Example output
+-------+------+-------------------------------------+
|version|mode |variables |
+-------+------+-------------------------------------+
|1 |create|[latitude, longitude, msl, t2m, time]|
|2 |update|[t2m_c] |
+-------+------+-------------------------------------+

An input that is read more than once​

A table of cells needs no staging, whatever its cost to evaluate. The first pass reads its dimension columns only (not the values), and the write pass reads the table once. An input that changed between the passes is refused, as for blocks, so give it a deterministic one.

A blocks DataFrame is read once by an update or an overwrite. A create and an append need the input's extent (and an append its new labels) before they can lay the chunks out. So they make a first pass over the block positions, which never reads the data column, and then the write pass. An opaque input, such as a mapInArrow or a query with a join, is therefore evaluated twice by them. A pass that finds the input has changed is refused ("the input changed between planning and writing"). Give a create or an append an input that is deterministic. A table is best: stage the blocks in one first, and the table is read once for the data.

from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write

blocks = gbx_zarr_read(
spark,
cube,
variables=["t2m", "time", "latitude", "longitude"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T11:00")},
)
blocks.write.mode("overwrite").saveAsTable(staged) # evaluated once, here
version = gbx_zarr_write(spark.table(staged), table, mode="create", root=root)

Blocks on the store's grid​

A create writes blocks that each cover whole chunks (whole shards, in a sharded array) where they are made. The task that holds a block encodes it and writes it to a data file of its own. Nothing is shuffled. This is decided for the whole create, every variable at once. If any block cuts across a chunk or a shard, or two blocks overlap, the write cuts all of them into pieces. It regroups the pieces by chunk through a shuffle and assembles them. The store is the same either way: the same documents, the same chunk rows, the same bytes in every chunk. Only the data file a chunk sits in differs.

To write the faster way, start every block on a chunk boundary (a shard boundary, for a sharded array). End it on one or at the edge of the array. The tasks that hold the blocks write them. An input that keeps its blocks in few partitions (a single one, say) is first spread out by block, so many tasks write at once. A block of several shards writes their chunks in its own order. One shard's chunks then do not sit side by side in the data file. A read of that shard takes more, smaller reads. A block of one shard keeps them together. A 1 GB create of blocks of one shard each wrote its data files at a median 119 MB/s through the Volume on Runtime 19. Direct to storage the median was 80 MB/s. Direct runs ranged from 39 to 139 MB/s. On Serverless the median was 122 MB/s. See Zarr performance.

An xarray.Dataset and a DataFrame of tiles are planned on the driver from what they hold, so neither has a pass of this kind.

Values and the array's type​

Decoded values are encoded with the array's own attributes. A write is refused, as ZarrUnsupported with a Fix:, when the values cannot be held. This happens for a NaN into an integer array with no _FillValue. It also happens for values that leave the stored integer range once packed, and for complex values into a real array. It happens as well for times into an array with no CF time units.

Times must be whole multiples of the array's time units. An axis stored as integer hours since ... holds no half hours. A write whose times would need finer units than the array has is refused whole, and nothing is committed:

import numpy as np

from databricks.labs.gbx.zarr import (
BLOCKS_SCHEMA,
ZarrUnsupported,
block_to_numpy,
gbx_zarr_read,
gbx_zarr_write,
)

times = gbx_zarr_read(
spark, table, variables=["time"], isel={"time": slice(0, 2)}
).collect()
half_past = [
{
**row.asDict(),
"data": (block_to_numpy(row) + np.timedelta64(30, "m")).tobytes(),
}
for row in times
]
try:
gbx_zarr_write(
spark.createDataFrame(half_past, BLOCKS_SCHEMA),
table,
mode="update",
variables=["time"],
)
except ZarrUnsupported as err:
print(err)
Example output
values cannot be written to array 'time': the times are not whole multiples of its units 'hours since 2023-01-01 00:00:00' (they would need 'minutes since 2023-01-01'), and its stored type int64 has no fractions Fix: write values that fit the array's type and attributes, or write its stored values (blocks read with decode=False)

The unit names that are understood are days, hours, minutes, seconds, milliseconds, microseconds and nanoseconds, in any case and with or without the plural (Hour is hours). The UDUNITS abbreviations, such as h and hrs, are not understood. Reading such an axis raises ZarrUnsupported ("is not a CF time unit"), and writing times into it is refused too. Correct the units attribute, or read the raw numbers with decode=False.

Write from a table of cells​

A table with one row per cell, such as a query result, is written as it is: no packing into blocks and no staging table. Pass dim_cols, a dict of dimension to the column that holds the cell's label along it, in the order of the array's dimensions. Every other column is a variable (or name them with variables=).

from pyspark.sql import functions as F

from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write

cells = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
isel={"time": slice(0, 6), "latitude": slice(0, 45), "longitude": slice(0, 90)},
output="table",
).select("time", "latitude", "longitude", (F.col("t2m") - 273.15).alias("temp_c"))
version = gbx_zarr_write(
cells,
table,
mode="create",
root=root,
dim_cols={"time": "time", "latitude": "latitude", "longitude": "longitude"},
)

The same call with transactional=False writes a plain store. An append extends the store along append_dim. The table's new labels of that dimension are placed after the store's (on its step, as for any append). The other dimensions' values must be labels the store has.

from pyspark.sql import functions as F

from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write

cells = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
isel={"time": slice(6, 12), "latitude": slice(0, 45), "longitude": slice(0, 90)},
output="table",
).select("time", "latitude", "longitude", (F.col("t2m") - 273.15).alias("temp_c"))
version = gbx_zarr_write(
cells,
table,
mode="append",
append_dim="time",
dim_cols={"time": "time", "latitude": "latitude", "longitude": "longitude"},
)

How it is written.

  • The driver reads the dimension columns only first. From them it takes the distinct labels of each dimension that has none yet, sorted ascending (a new store's dimensions, an append's new steps). It also finds which shards the cells fall in. A table that fills its grid (as many rows as the product of its labels) needs no second pass. The values columns are read once, by the write itself.
  • The cells are shuffled once, by the shard they fall in (or by the chunk, see below). A task owns whole shards. There is one partition per shard touched, up to one per slot of the session's parallelism. The shards of a larger store share the partitions in equal rounds. Each task builds its shards and adds them to the store where they are made, as a create from blocks on the shard grid does. A manifest append goes on through the writer's usual read-modify-write of the shards it touches.
  • A small shard that fits the memory of a task is built and written whole. A large shard (over 64 MiB with its variables) is written one inner chunk at a time. The task then holds one chunk of every variable, plus a copy of one. A plain store spools the shard's chunks on local disk and writes the shard from there. The files are the same either way. A chunk over the task budget is refused before any work, with a Fix that names chunks=.

Rules.

  • Labels. A new store's labels are the distinct values, ascending. To keep the order of another source (latitude usually descends), pass coords={"latitude": values, ...} to create. A label the table lacks is then a missing cell, and a table value that matches no label is refused. Floating-point labels match to a relative 1e-9; times and integers exactly.
  • Planning scan. The writer reads the table once before the data pass, to find the labels (no coords) or the storage units the cells fall in (coords). When you know the whole grid, pass coords for every dimension and complete=True. Every storage unit is then planned without that scan, and the table is read once, in the data pass. A unit with no cell at all ends the write with "the write pass saw N block(s), not the M its plan was made from". Leave complete off for a table that skips whole units. Holes inside a unit are still fill values. An append takes coords for the append dimension only: the new labels, after the store's last. The other dimensions use the store's labels. coords and complete=True together plan an append with no scan. Without them, an append reads the table once for the new labels. It reads the table a second time for its units, unless the table has one row per cell of the grid. A scan reads only the dimension columns, but a Python data source such as gbx_zarr_read(output="table") cannot skip the other columns, so it still reads every variable.
  • Missing cells read as the array's fill value (NaN for floats, 0 for integers and booleans). A null value is a missing cell. A null label is refused.
  • Duplicate cells (two rows at one position) are refused, never merged. The task that meets them raises ZarrError ("share a position with another cell"), and nothing is committed. Aggregate or drop the repeats first.
  • Types. A variable keeps its column's type in a new store (float, double, the integer types and boolean). In an existing array the values are cast to the array's type, as for blocks. A decimal or string column is refused with a Fix to cast it.
  • Modes. create and append. An update or an overwrite of cells would replace stored cells with the fill value wherever the table has none, so they take blocks. An append writes arrays the store already has.

Zarr to Zarr: compute on cells​

To read a store, compute on it and write the result, read the cells and compute in Spark columns. Read with output="table", derive the new columns, and write the table with coords= and complete=True. Pass transactional=False for a plain store.

import xarray as xr
from pyspark.sql import functions as F

from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write

window = {"time": slice(0, 12), "latitude": slice(0, 45), "longitude": slice(0, 90)}
cells = gbx_zarr_read(
spark, cube, variables=["u10", "v10"], isel=window, output="table"
).select(
"time",
"latitude",
"longitude",
F.hypot("u10", "v10").cast("float").alias("wind_speed"),
)
labels = xr.open_zarr(cube).isel(window)
return gbx_zarr_write(
cells,
path,
transactional=False,
dim_cols={"time": "time", "latitude": "latitude", "longitude": "longitude"},
coords={d: labels[d].values for d in ("time", "latitude", "longitude")},
complete=True,
spark=spark,
)
  • coords= fixes the labels and keeps their order, so the new store is labelled like the source.
  • complete=True skips the planning scan. The table is read once, by the write.
  • This is the fastest Zarr to Zarr path on a cluster. A computed column costs nothing extra, and the write needs no pass over the input beyond the one that feeds it.

Compute on blocks​

Use blocks when the compute is shaped like a block, such as a pandas UDF over the bytes of each block. gbx_zarr_read returns blocks by default, and gbx_zarr_write takes the result. No row is made per cell.

import json

import numpy as np
import pandas as pd
from pyspark.sql import functions as F

from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write

@F.pandas_udf("binary")
def wind_speed_block(u: pd.Series, v: pd.Series, dtype: pd.Series) -> pd.Series:
"""The bytes of one wind speed block, from the bytes of its u10 and v10 blocks."""
return pd.Series(
[
np.hypot(np.frombuffer(a, k), np.frombuffer(b, k)).astype(k).tobytes()
for a, b, k in zip(u, v, dtype)
]
)

hours = {"time": slice("2023-01-01T00:00", "2023-01-01T11:00")}
u = gbx_zarr_read(spark, cube, variables=["u10"], sel=hours).alias("u")
v = gbx_zarr_read(spark, cube, variables=["v10"], sel=hours).alias("v")
attrs = json.dumps({"units": "m s-1", "long_name": "10 metre wind speed"})
wind = u.join(v, F.col("u.start") == F.col("v.start")).select(
F.lit("wind_speed").alias("variable"),
"u.dims",
"u.start",
"u.stop",
"u.coord_start",
"u.coord_stop",
"u.dtype",
"u.shape",
wind_speed_block("u.data", "v.data", "u.dtype").alias("data"),
"u.fill_value",
F.lit(attrs).alias("attrs"),
)
labels = gbx_zarr_read(
spark, cube, variables=["time", "latitude", "longitude"], sel=hours
)
summary = gbx_zarr_write(
wind.unionByName(labels), path, transactional=False, spark=spark
)
  • A create from blocks reads its input twice: once to plan, once to write. The Zarr reader cannot skip the data columns in the first pass. If the input is expensive, stage it first (see above).
  • Join the blocks of the inputs on start, so each pair covers the same cells. Keep dims, start, stop and shape from one side.
  • Read the coordinate arrays too (time, latitude, longitude) and add them to the result, or a new store is labelled by position only.
  • Keep the compute deterministic, since the input is evaluated twice.
  • Use cells instead when the compute needs the neighbours of a cell, or when the source is not a Zarr store.

Tiles​

A DataFrame with a tile column writes each tile's pixels into the array of its variable, in any mode. Tiles are placed by their decoded geotransform against the array's grid, so a clip or a change of NoData keeps its place. A null tile is skipped.

  • A tile cut from Zarr by gbx_zarr_read(output="tiles") still carries where it came from (zarr_array, sliceDims, window). It goes to the array it came from, whose grid (geotransform and CRS) must be the plane it was cut from, else ZarrGridMismatch. Its other positions are its sliceDims.
  • A tile with no provenance needs dim_cols for every non-spatial dimension. It is a dict of dimension to the column that holds its label (a number or a time column). It also needs variables="<name>" to name its variable (a new store defaults to "data"). Most pixel-producing rst_* functions return such a tile, and so does a stack of GeoTIFFs. dim_cols also overrides sliceDims.

The tile round trip​

A pixel-producing rst_* function (a map algebra, a clip, a resample) returns a fresh tile without provenance, so it cannot be written back as it is. There are two ways round. Put the provenance back with withField("metadata", ...):

from pyspark.sql import functions as F

from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write

tiles = gbx_zarr_read(
spark, table, variables=["t2m"], isel={"time": slice(0, 4)}, output="tiles"
)
# rst_mapalgebra returns a new tile with no record of where it came from: copy the
# metadata of the tile it was made from back onto it
corrected = (
tiles.select(
rx.rst_mapalgebra(F.array("tile"), F.lit("A-1")).alias("tile"),
F.col("tile.metadata").alias("provenance"),
)
.withColumn("tile", F.col("tile").withField("metadata", F.col("provenance")))
.select("tile")
)
version = gbx_zarr_write(corrected, table, mode="update")

or say where each tile goes with dim_cols and variables=:

from pyspark.sql import functions as F

from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write

tiles = gbx_zarr_read(
spark, table, variables=["t2m"], isel={"time": slice(0, 4)}, output="tiles"
)
corrected = tiles.select(
rx.rst_mapalgebra(F.array("tile"), F.lit("A-1")).alias("tile"),
F.to_timestamp(F.col("tile.metadata")["sliceCoord_time"]).alias("time"),
)
version = gbx_zarr_write(
corrected,
table,
mode="update",
dim_cols={"time": "time"}, # the dimension, and the column that labels it
variables="t2m", # the array the tiles go to
)

A new store from tiles​

create derives the grid from the tiles: their CRS, their cell size and the extent of all of them. A tile with another CRS, another cell size, or an origin more than 1e-6 of a cell off the cell edges raises ZarrGridMismatch. Each dim_cols column gives the sorted labels of its dimension; the latitude and longitude (or y and x) labels come from the grid. A CRS other than EPSG:4326 is carried by a spatial_ref grid-mapping variable, so gbx_zarr_read(output="tiles") reads the CRS back.

from pyspark.sql import functions as F

from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write

tiles = gbx_zarr_read(
spark, cube, variables=["t2m"], isel={"time": slice(0, 3)}, output="tiles"
)
warmer = tiles.select(
rx.rst_mapalgebra(F.array("tile"), F.lit("A+1")).alias("tile"),
F.to_timestamp(F.col("tile.metadata")["sliceCoord_time"]).alias("time"),
)
version = gbx_zarr_write(
warmer,
table,
mode="create",
root=root,
dim_cols={"time": "time"},
variables="t2m", # a new store names its variable "data" unless you say
)

An append takes the new labels of the append dimension from its dim_cols column; the other dim_cols values must be labels the store has. update and overwrite look the values up in the store's coordinates. Values keep the tile's type in a new store. In an existing array they are cast to the array's type (floats to integers round half to even), and the tile's NoData becomes the array's _FillValue. A tile that does not fit is refused on the driver, before any file is written.

When writers meet​

Zarr transactions explains the rules row by row. In short:

  • writers that touch different chunks both land;
  • writers that only change the shape or a shared coordinate chunk are rebased;
  • a write that meets another's chunk raises ZarrWriteConflict and commits nothing.

The limits of the rebase are worth knowing before many jobs append to one array:

  • A write retries up to 20 attempts, waiting between them with a jittered pause that doubles up to 16 seconds. Its data files are never rewritten.
  • Appends to one array land one at a time. They all change the array's shape and the time coordinate chunk, so Delta commits one per round. The last of N simultaneous appenders needs about N attempts. In measurements on Databricks one attempt, including its wait, took 16 to 28 seconds. A writer that keeps losing can therefore spend several minutes before it gives up.
  • 17 or more simultaneous appenders to one array can exhaust the bound. Sixteen simultaneous appenders all landed in the measurements; beyond that the last ones raise ZarrWriteConflict and nothing of their write is visible. The Fix: says to retry later or reduce the concurrent writers to the array. Batch the appends into fewer, larger writes, or run the failed ones again when the others are done.
  • A coordinate label set to two different values is a data conflict. If another writer already set the same coordinate cell to a different value, the write raises ZarrWriteConflict rather than keeping the last writer's. Equal values merge.

Registered stores and publish​

A store registered with writable=True takes the same writes. They land in its overlay root, and the store's own files are never touched. Every other registration of the store, and every plain Zarr reader, is therefore unaffected until you publish.

Deleting a chunk​

A chunk whose every cell is the fill value is not stored, in every mode, as zarr does by default. If the chunk had a row, the write deletes it from the manifest. If it lies under a store's shard, the row becomes a tombstone (a row with no file). A read sees the fill value there. A read of an older version, and a gbx_zarr_restore to one, still has the data. gbx_zarr_optimize folds tombstones away, and a publish deletes the object.

import numpy as np

from databricks.labs.gbx.zarr import BLOCKS_SCHEMA, gbx_zarr_read, gbx_zarr_write

chunk = gbx_zarr_read(
spark,
table,
variables=["t2m"],
isel={"time": slice(0, 12), "latitude": slice(0, 91), "longitude": slice(0, 180)},
).first() # exactly one inner chunk
empty = {
**chunk.asDict(),
"data": np.full(tuple(chunk["shape"]), np.nan, dtype="float32").tobytes(),
}
gbx_zarr_write(spark.createDataFrame([empty], BLOCKS_SCHEMA), table, mode="update")
spark.sql(
f"SELECT kind, array, key, file IS NULL AS deleted FROM {table} "
"WHERE origin = 'gbx' AND array = 't2m' AND kind IN ('chunk', 'shard') "
"AND tag IS NULL"
).show(truncate=False)
Example output
+-----+-----+-------+-------+
|kind |array|key |deleted|
+-----+-----+-------+-------+
|chunk|t2m |c/0/0/0|true |
+-----+-----+-------+-------+

Publishing​

gbx_zarr_export(spark, table, method="publish") writes the overlay into the store itself, so every Zarr reader sees it (Outside changes: drift, sync and publish describes what it does and guarantees). The limits:

  • A publish refuses if the store changed since the last sync, with ZarrStoreDrift. Run gbx_zarr_sync and publish again.
  • A write that lands on the table during a publish makes the publish fail. The publish commits with a compare-and-swap on the overlay rows it turns into store objects. It raises ZarrWriteConflict and the table does not move if a writer replaced one of them after the publish read it. The same happens if Delta rejects the commit because a writer really raced it. The objects the publish had already written are in the store. The Fix: says to run gbx_zarr_sync, which lists them (the change is both for a key the overlay also changed, and the overlay still wins). Then publish again. The lock is released.
from databricks.labs.gbx.zarr import (
ZarrWriteConflict,
gbx_zarr_export,
gbx_zarr_sync,
)

try:
gbx_zarr_export(spark, table, method="publish")
except ZarrWriteConflict as err:
print(err.fix)
gbx_zarr_sync(spark, table).show() # lists the objects the failed publish wrote
version = gbx_zarr_export(spark, table, method="publish")
Example output
the objects this publish already wrote are in the store, so run gbx_zarr_sync(spark, 'weather_conflict') to record them, then publish again
+------+-----+-------+
|change|array| key|
+------+-----+-------+
| both| t2m|c/0/0/0|
+------+-----+-------+
  • A publish that was killed leaves its lock. The table refuses the next publish or sync with ZarrWriteConflict, and the Fix: says how to clear it. A lock can be broken once it is at least one hour old (a younger one may belong to a publish that is still writing). Pass break_lock=True to gbx_zarr_sync, then publish again:
from databricks.labs.gbx.zarr import (
ZarrWriteConflict,
gbx_zarr_export,
gbx_zarr_sync,
)

try:
gbx_zarr_export(spark, table, method="publish")
except ZarrWriteConflict as err:
print(err.fix)
# no publish is running, and the lock is more than an hour old: clear it, then publish
gbx_zarr_sync(spark, table, break_lock=True)
version = gbx_zarr_export(spark, table, method="publish")
Example output
wait for the publish to finish. If it was killed and none is running, clear the lock with gbx_zarr_sync(spark, 'weather_lock', break_lock=True), then publish again with gbx_zarr_export(spark, 'weather_lock', method="publish"); a lock can be broken once it is at least 1 hour old

A publish records the identity of each object it wrote only after the Volume has settled the object's modification time. A Volume moves that time to the next whole second a moment after the write. A read right after a publish therefore does not report drift. After looking at each object once, it waits, up to 2 seconds in all, for the times still moving to settle. An object whose time has not settled by then is recorded as it is, with a warning. Then gbx_zarr_sync puts its identity right. The version gbx_zarr_export(..., method="publish") returns is the commit that turned the overlay rows into store objects.

Folding after a write​

Appends and updates add one manifest row per chunk they change. optimize=True runs gbx_zarr_optimize on exactly the shards the write touched, right after its commit. A few small writes into a big shard cost one fold of that shard, and the content is the same either way. The version the call returns is the write's own; the fold, if it had work to do, is the next version.

from databricks.labs.gbx.zarr import gbx_zarr_history, gbx_zarr_read, gbx_zarr_write

following = gbx_zarr_read(
spark,
cube,
variables=["t2m", "msl", "time", "latitude", "longitude"],
sel={"time": slice("2023-01-01T12:00", "2023-01-01T23:00")},
)
version = gbx_zarr_write(following, table, mode="append", optimize=True)
# `version` is the append's own; the fold, if it had work to do, is the next version
gbx_zarr_history(spark, table).select("version", "mode").orderBy("version").show()
Example output
+-------+--------+
|version| mode|
+-------+--------+
| 1| create|
| 2| append|
| 3|optimize|
+-------+--------+

Plain stores (opt-in)​

Pass transactional=False to write a plain Zarr store instead of a managed one. target is then a path (a Volume path, or any mounted path), not a table name. The default stays transactional=True.

import xarray as xr

from databricks.labs.gbx.zarr import gbx_zarr_write

first = xr.open_zarr(cube)[["t2m"]].isel(time=slice(0, 12)).load()
summary = gbx_zarr_write(first, path, transactional=False, spark=spark)

The call returns a summary, {"path", "mode", "arrays", "chunks", "bytes"}: the store's path, the mode, the arrays it holds, and the chunks and bytes this call wrote. It is not a Delta version.

What it writes. A Zarr v3 store that xarray.open_zarr, zarr-python and gbx_zarr_read(spark, path) all open: one file per chunk, or per shard for a sharded array, and the metadata documents. The inputs, variables, dim_cols, chunks, shards, codecs and their checks are the same as for a managed store. Tasks write the files where the blocks are made. When blocks start and stop on the shard grid, nothing is shuffled.

Plain stores and gbx_zarr_export copies carry consolidated metadata, kept current after an append, so xarray.open_zarr(path) opens them without a warning.

What it drops. No manifest table, ownership ledger or marker file, and no Delta commit. So there are no versions or time travel, no retention or vacuum, and no overlay. A write is not all-or-nothing, and two writers at once are not guarded: use one writer at a time, or keep a managed store.

Safety it keeps. The metadata documents are written last, after every data file, and the root group's document is the very last. A write that stops part way leaves files but no store, so no reader opens a half-written one. A failed create removes what it wrote. A create refuses a path that already holds a store (a zarr.json, .zgroup or .zarray). An append needs a plain store there:

import xarray as xr

from databricks.labs.gbx.zarr import gbx_zarr_write

later = xr.open_zarr(cube)[["t2m"]].isel(time=slice(12, 24)).load()
summary = gbx_zarr_write(
later, path, transactional=False, mode="append", append_dim="time", spark=spark
)

An append extends the array shape, and rewrites the documents last, so readers see the old store until it ends. A chunk or shard it shares with existing cells keeps those cells. It refuses variables on other dimensions, and values that would be cut to fit the stored type (floats into integers, say), each with a Fix:. A failed append can leave files past the store's extent; no document names them.

Refusals. ValueError with a Fix: for mode="update" or "overwrite", a table name as target, and file_mode, retain, write_id, optimize, root or access="direct".

When to use it. One writer, speed, and a store other tools read. A managed store pays for its manifest and commit; a plain write pays for neither. Choose a managed store for concurrent writers, versions, or all-or-nothing writes.

Access and cleanup​

access is how the tasks reach the store's files, as for a read. With "auto" (default), a task writes straight to cloud storage where Unity Catalog can issue credentials (dedicated compute). Everywhere else, including Serverless, it writes through the Volume mount. With "fuse", a task always uses the mount and sends no credentials. With "direct", a write raises ZarrAccessUnavailable where credentials cannot be had. See Access for the credentials and the local-disk encryption note.

A pack upload that is refused or fails part way can leave an incomplete multipart upload in the bucket. GeoBrix does not clean it up: add a bucket lifecycle rule that aborts incomplete multipart uploads (AbortIncompleteMultipartUpload) to the bucket of the Volume.

Files GeoBrix writes live at data/<array>/<write_id>/part-<id>.zpack under the root. Never put your own files in that layout: a vacuum deletes any file in exactly that shape that no readable version needs. Files anywhere else under the root, and any other store's root, are never touched.

Errors​

Every error GeoBrix defines is a ZarrError and ends with a Fix: that says how to resolve it. A bad argument is a ValueError that ends the same way. A mistake in the call is raised on the driver before any file is written. An error raised while Spark runs the write arrives from the action as a pyspark.errors.PythonException, with the GeoBrix error name and its Fix: line in the message. A failed write commits nothing. The full list, with the cause of each, is in the Zarr Function Reference.

Next Steps​

  • Weather Features with Zarr — an end-to-end example notebook. It registers a weather cube in place and samples it into a features table. It writes a derived variable back as a new version, maps it as raster tiles, and tags the version for training.
  • Zarr Reader — read a store as blocks, a table or tiles.
  • Zarr Function Reference — every function, parameter and error.
  • Zarr Stores — the transactional model and its topic pages: versions, tags, retention and file modes.
  • Writers Overview — all writers, with their tiers and options.