Skip to main content

Zarr Function Reference

Since: v0.6.0 · (Python level only; no JAR)

The Zarr functions read, write, version and maintain Zarr stores. They are Python functions in databricks.labs.gbx.zarr and have no SQL binding. A store's manifest is a Delta table, so SQL can query it. The GeoBrix tiles a read returns work with every rst_* function. Install the env6 wheel with its [zarr] extra, on Serverless environment 6 or Databricks Runtime 19. See Zarr Stores. Importing the package imports no large library; importing a name loads only the module that defines it.

from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write
See it end to end

The Weather Features with Zarr notebook uses these functions in order. It registers a weather cube in place and samples it at 100 sites. It writes a derived variable back as a new version, maps it as raster tiles, and tags the version for training.

For how-to guides see the Zarr Reader and the Zarr Writer. For the model, see the topic pages. Every example on this page is run by the documentation tests. They use these names, which stand for what is yours:

# The sample cube, a name for the store's manifest table, and a Volume directory for its files.
cube = "/Volumes/main/geobrix_samples/geobrix-examples/global/era5-weather/era5_like_v3.zarr"
table = "main.weather.era5"
root = "/Volumes/main/weather/stores/era5"
FunctionDoes
gbx_zarr_readReads a store, or a cut of one, as blocks, a table of cells or GeoBrix tiles
gbx_zarr_sampleReads a store at the points of a DataFrame
gbx_zarr_writeCreates, appends to, updates or overwrites a store, in one transaction
gbx_zarr_registerRegisters an existing store in place as a table
gbx_zarr_syncBrings a registered table up to date with changes made by other tools
gbx_zarr_optimizeFolds the chunk rows that appends and updates leave into whole shards
gbx_zarr_vacuumDeletes the files no readable version needs
gbx_zarr_tag, gbx_zarr_untagNames a version so it outlives vacuum; removes the name
gbx_zarr_historyLists the commits of a store
gbx_zarr_restoreMakes the store equal an older version again, as a new commit
gbx_zarr_exportTakes a version out as a plain Zarr store or a reference file, or publishes an overlay
BLOCKS_SCHEMA, block_to_numpy, block_to_xarrayThe schema of a blocks read, and the helpers that turn a block into an array

Every error GeoBrix defines is a subclass of ZarrError, and its message ends with a Fix: that says how to resolve it; see Errors. A mistake in an argument is a plain ValueError, which except ZarrError does not catch; the ones in the tables below end with a Fix: too. The tables list the errors you are most likely to meet. Each row gives the start of the fix (all of it where it is short, and <name> where the message fills in a value).

Learn the model first​

New to Zarr in GeoBrix? Read the topic pages in this order. Each opens with what it is for.

TopicRead it to
Zarr Stores: concepts and store kindsUnderstand managed, registered and plain stores, and how a store is files and rows
Registering, syncing and ownershipUse a store you already have without copying it, and handle outside changes
Transactions and concurrent writersKnow what a write guarantees and what happens when writers meet
Versions, time travel and tagsRead, name, restore and export versions
Lifecycle, retention and optimizeFree storage and keep reads fast
Performance and limitsSize a workload and check what is not supported

Read and sample​

gbx_zarr_read​

gbx_zarr_read(spark, source, *, source_type="auto", variables=None, sel=None, isel=None,
output="blocks", version_as_of=None, timestamp_as_of=None, access="auto",
decode=True, virtual_tiles=True, band_dim=None, tile_size=None,
use_zarr_fill_value_as_mask=None)

Reads a Zarr store, or a selection of it, into a DataFrame. The read is planned on the driver and run in tasks. A table read is pinned to one version of the manifest, so it is consistent whatever commits meanwhile. Returns a DataFrame (the schema depends on output).

ParameterTypeDescription
sparkSparkSessionThe session.
sourcestrThe manifest table of a managed or registered store, or the path of a plain store.
source_typestr"auto" (default) takes a string that starts with / or has a URI scheme as a path and anything else as a table. "path" or "table" says which.
variablesstr or list[str]The arrays to read. Default: every gridded variable; coordinate arrays and scalars are left out.
seldictSelection by label: dimension to a scalar, an inclusive slice(a, b) or a list. Times may be ISO strings. A longitude slice may cross the seam.
iseldictSelection by position: dimension to an int, a slice (step 1) or a list of ints.
outputstr"blocks" (default), "table" or "tiles".
version_as_ofint or strA version number or "tag:<name>". A plain store has no versions.
timestamp_as_ofstr or datetimeThe version live at that time. Not with version_as_of.
accessstr"auto" (default), "fuse" or "direct"; see Access.
decodeboolApply the CF conventions (scale and offset, missing values, times). Default True; False returns the stored values.
virtual_tilesbooloutput="tiles" only. True (default): tiles hold no pixels and read their chunks when opened. False: each tile holds GeoTIFF bytes.
band_dimstroutput="tiles" only. A dimension to stack as the bands of each tile.
tile_sizetuple[int, int]output="tiles" only. (width, height) in cells; default one inner chunk.
use_zarr_fill_value_as_maskbool or Nonexarray's option of the same name. None (default) masks on the array fill_value for Zarr v2 only.

Returns. output="blocks": BLOCKS_SCHEMA, one row per variable and touched chunk. output="table": one row per selected cell, with a column per dimension and per variable, typed as the Reader describes, NULL for missing data, in no defined order. output="tiles": a (source, tile) DataFrame of GeoBrix tiles.

from databricks.labs.gbx.zarr import gbx_zarr_read

cells = gbx_zarr_read(
spark,
table,
variables=["t2m"],
sel={"time": "2023-01-01T06:00", "latitude": slice(50, 48), "longitude": 2.0},
output="table",
)
cells.orderBy("latitude").show()
Example output
+-------------------+--------+---------+------+
| time|latitude|longitude| t2m|
+-------------------+--------+---------+------+
|2023-01-01 06:00:00| 48.0| 2.0| 302.0|
|2023-01-01 06:00:00| 49.0| 2.0|301.75|
|2023-01-01 06:00:00| 50.0| 2.0|301.75|
+-------------------+--------+---------+------+

Errors. A mistake in the call is raised on the driver. The DataFrame is lazy. An error raised while Spark runs the read arrives from the action as a pyspark.errors.PythonException. Its message keeps the GeoBrix error name and Fix: line.

ErrorRaised whenThe fix says
ValueErrorsource_type is not auto, path or tablepass 'auto', 'path' or 'table'
ValueErroroutput is not blocks, tiles or tablepass 'blocks', 'tiles' or 'table'
ValueErroraccess is not auto, fuse or directpass 'auto', 'fuse' or 'direct'
ValueErrorBoth version_as_of and timestamp_as_of are givenname the version one way.
ValueErrorversion_as_of is neither an int nor "tag:<name>"pass a version number (gbx_zarr_history lists them) or a tag as 'tag:<name>'.
ValueErrortimestamp_as_of is not a string or a datetime, or not a timestamppass a timestamp string such as '2023-01-01 06:00:00', or a datetime.
ValueErroruse_zarr_fill_value_as_mask is not True, False or Nonepass True, False or leave it None (xarray's default).
ValueErrorA name in variables is not an array of the storename arrays of the store
ValueErrorA table read names a coordinate in variablesleave the coordinate out of variables, or read it as blocks
ValueErrorA dimension is in both sel and iselselect each dimension one way
ValueErrorsel or isel names a dimension the variables lackselect only existing dimensions
ValueErrorband_dim is not a dimension of the variablesname a dimension of the variables
ZarrVersionUnavailableA plain store is read with version_as_of or timestamp_as_ofregister the store with gbx_zarr_register to read it as of a version, or drop version_as_of and timestamp_as_of
ZarrVersionUnavailableThe version does not exist, or is older than the store's retain horizonpick a later version or tag the version you need with gbx_zarr_tag
ZarrVersionUnavailableThe tag does not existpick an existing tag, or create it with gbx_zarr_tag
ZarrUnsupportedA variable you named cannot be decoded (strings, or codecs around a shard, say)read the other variables, or rewrite this array in a form GeoBrix reads
ZarrUnsupportedA table or tiles are asked for a complex arrayread it as blocks, or leave it out of variables
ZarrUnsupportedNo variable can be tiled (no regular latitude and longitude grid, or complex values)read it with output="blocks" or output="table"
ZarrUnsupportedThe store keeps its files as FILE values and the compute does not support themuse a Databricks Runtime 19 dedicated cluster (single-user, with fileReferenceCreationMode=MANAGED in its cluster config) in a workspace with the File type preview
ZarrAccessUnavailableaccess="direct" is asked of a plain storeregister the store with gbx_zarr_register to use direct access, or use access="auto" or "fuse"
ZarrAccessUnavailableaccess="direct" with virtual tilesuse virtual_tiles=False to read the tiles now with direct access, or access="auto" or "fuse"
ZarrStoreDriftA registered store's files were changed by another tool (raised in a task)run gbx_zarr_sync(spark, <table>) on the store's manifest table to record the change, then read again

gbx_zarr_sample​

gbx_zarr_sample(spark, points_df, source, *, geom_col="geom", crs=None, variables=None, sel=None,
method="nearest", version_as_of=None, timestamp_as_of=None, access="auto",
use_zarr_fill_value_as_mask=None)

Samples a store at the points of a DataFrame. Returns points_df with every column it had. Then it adds a column for each dimension other than latitude and longitude that sel leaves in (time as TIMESTAMP). Last it adds a column per variable, typed as in a table read. A point is repeated for each selected position of those dimensions, so one point's time series is its rows. Row order is not defined.

ParameterTypeDescription
sparkSparkSessionThe session.
points_dfDataFrameA POINT column geom_col of WKB, EWKB, WKT or EWKT, as rst_sample reads it. Above 100,000 points it is evaluated more than once, so it must be deterministic.
sourcestrThe manifest table of a managed or registered store, or the path of a plain store.
geom_colstrThe geometry column. Default "geom".
crsstr or intThe CRS of points that carry no SRID. An EWKB or EWKT SRID wins; a point with neither is taken to be in the store's CRS.
variablesstr or list[str]Default: every gridded variable, which must share a grid.
seldictSelects the dimensions the points do not choose (time, level), by label.
methodstr"nearest" (the only one). "bilinear" is planned.
version_as_of, timestamp_as_of, access, use_zarr_fill_value_as_maskAs in gbx_zarr_read.

Three cases give NULL. First, a point is outside the grid (more than half a cell beyond its first or last cell). Second, a geometry is missing or empty. Third, a cell equals the fill value.

from databricks.labs.gbx.zarr import gbx_zarr_sample

points = spark.createDataFrame([("POINT(2.0 48.0)",)], ["geom"])
sampled = gbx_zarr_sample(
spark, points, table, variables=["t2m"], sel={"time": "2023-01-01T06:00"}
)
sampled.show(truncate=False)
Example output
+---------------+-------------------+-----+
|geom |time |t2m |
+---------------+-------------------+-----+
|POINT(2.0 48.0)|2023-01-01 06:00:00|302.0|
+---------------+-------------------+-----+

Errors. The errors of gbx_zarr_read for the source, version and access, and:

ErrorRaised whenThe fix says
ValueErrorpoints_df has no column geom_colpass geom_col=<the geometry column>
ValueErrorThe column is not BINARY or STRINGpass WKB/EWKB (BINARY) or WKT/EWKT (STRING) points
ValueErrorA geometry is not a POINTpass points (use a centroid for other shapes)
ValueErrorpoints_df already has a column the sample addsrename them in points_df, or sample other variables
ValueErrorsel selects latitude or longitudeselect only the other dimensions (time, level) with sel
ValueErrorcrs cannot be readuse an EPSG or ESRI code, a CRS string or WKT
ZarrUnsupportedmethod is not "nearest"use method="nearest"; bilinear is planned
ZarrUnsupportedThe array has no latitude and longitude (or y and x) dimensionsname the dimensions latitude/longitude or y/x
ZarrUnsupportedThe coordinates neither rise nor fallsample a store whose latitude/longitude (or y/x) labels are monotonic
ZarrGridMismatchThe variables are on different gridssample variables with the same dimensions, or sample them separately
ZarrErrorpoints_df changed between planning and readingmake points_df deterministic (write it to a table first), then sample that

BLOCKS_SCHEMA​

The DDL schema string of a blocks read: variable STRING, dims ARRAY<STRING>, start ARRAY<BIGINT>, stop ARRAY<BIGINT>, coord_start ARRAY<STRING>, coord_stop ARRAY<STRING>, dtype STRING, shape ARRAY<BIGINT>, data BINARY, fill_value STRING, attrs STRING. Pass it to spark.createDataFrame(rows, BLOCKS_SCHEMA) to build a blocks DataFrame to write. The columns are described under Output: blocks.

block_to_numpy​

block_to_numpy(row)

Returns the cells of a block row (a Row, or a dict from a blocks read) as a writable NumPy array. The array has the block's shape and dtype, in C order.

block_to_xarray​

block_to_xarray(row)

Returns a block row as an xarray.DataArray named for the variable, with the block's dimensions and attributes. A block carries the first and last label of each dimension rather than every label, so no coordinates are attached.

from databricks.labs.gbx.zarr import (
BLOCKS_SCHEMA,
block_to_numpy,
block_to_xarray,
gbx_zarr_read,
)

blocks = gbx_zarr_read(spark, table, variables=["t2m"], isel={"time": slice(0, 3)})
row = blocks.orderBy("start").first() # the block at the top left of the grid
cells = block_to_numpy(row) # a numpy array of the block's shape and dtype
named = block_to_xarray(row) # a DataArray named for the variable, with its dimensions
print(cells.shape, cells.dtype)
print(named.name, named.dims)
print(BLOCKS_SCHEMA.split(",")[0]) # the schema, as a DDL string
Example output
(3, 181, 295) float32
t2m ('time', 'latitude', 'longitude')
variable STRING

Write​

gbx_zarr_write​

gbx_zarr_write(data, target, *, mode="create", root=None, variables=None, append_dim=None,
dim_cols=None, coords=None, complete=False, chunks="balanced", shards=None, codecs=None, file_mode=None,
retain="7 days", write_id=None, access="auto", optimize=False, transactional=True,
spark=None)

Writes data to the Zarr store whose manifest table is target, in one transaction. Returns the store version (the Delta version) the write committed. With transactional=False, target is a path and the write makes a plain Zarr store instead; it returns a summary dict. See the Zarr Writer for the modes, layouts, tiles and concurrency in detail.

ParameterTypeDescription
dataDataFrame or xarray.DatasetA blocks DataFrame (BLOCKS_SCHEMA), a DataFrame with one GeoBrix tile column, a table of cells (one row per cell, with dim_cols), or a Dataset held on the driver.
targetstrThe manifest table of the store.
modestr"create" (default), "append", "update" or "overwrite".
rootstrcreate only. The directory the store's files go to; it must not exist, or be empty.
variablesstr or list[str]The variables to write. Required in an update or overwrite from blocks to create a variable the store lacks, and it narrows the work. For tiles with no provenance, the single variable they belong to.
append_dimstrappend only. Default: the first dimension labelled by CF times.
dim_colsdict[str, str]Tiles and tables of cells. Dimension to the column that labels each tile or cell along it. For cells the dict's order is the array's dimension order, every other column is a variable (or the ones variables names), and the mode is create or append. See Write from a table of cells.
coordsdict[str, array]Cells only. In a create, the labels of each dimension, in array order, instead of the distinct values of its column sorted ascending. In an append, the new labels of the append dimension only. It saves the pass over that column.
completeboolCells with coords for every dimension of a create, or for the append dimension of an append. True says every storage unit of that grid holds at least one cell, so the writer skips the scan that finds the units: the table is read once. A unit with no cell ends the write; default False.
chunksstr or dictcreate only. "balanced" (default), "maps", "series" or a dict of dimension to size.
shardsNone, False or dictcreate only. None computes the shards, a dict sets some, False means none.
codecslistcreate only. The inner chunk's codec chain as Zarr v3 codec documents or zarr codec objects. Default bytes, zstd (level 3), crc32c.
file_modestrcreate only: None (default) is "fuse" (path strings, so the store reads on every compute), "auto" (FILE references where the compute has them, otherwise "fuse") or "external". "managed" is not available in this release. A later write follows the store's own mode and raises if given another.
retainstrcreate only. How long superseded versions stay readable, such as "7 days" (the default).
write_idstr32 lowercase hex characters. Writing again with one that has committed returns its version and writes nothing.
accessstr"auto" (default), "fuse" or "direct"; see Access.
optimizeboolFold the shards the write touched right after it commits. The version returned is the write's own.
transactionalboolTrue (default) writes a managed store in one Delta commit. False writes a plain Zarr store at the path target: no manifest table, ledger, marker or commit; only mode="create" or "append"; one writer at a time. Returns {"path", "mode", "arrays", "chunks", "bytes"}, not a version. See Plain stores.
sparkSparkSessionRequired for an xarray.Dataset, which has no session of its own.
from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write

first = gbx_zarr_read(
spark,
cube,
variables=["t2m", "msl", "time", "latitude", "longitude"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T11:00")},
)
version = gbx_zarr_write(first, table, mode="create", root=root) # the first version

Errors. Mistakes in the call are raised on the driver before any file is written. A failed write commits nothing.

ErrorRaised whenThe fix says
ValueErrormode is not one of the fourpass one
ValueErrorA table of cells with mode "update" or "overwrite"use one of those, or write blocks
ValueErrorcoords in an update or overwrite, or on data that is not a table of cellsdrop coords
ValueErrorcoords for a dimension other than the append dimension, in an append of cellsdrop coords for <dimensions>
ValueErrorA dim_cols column the table lacks, or no value columnname a column of the table
ZarrUnsupportedA cells column of a type GeoBrix does not write (a decimal, a string), or a label column that is not a number or a timecast <column> to a number or a timestamp
ZarrErrorTwo cells at one positionkeep one row per cell
ZarrErrorA null label in a table of cellsdrop or fill the cells without a position before writing
ZarrErrorA chunk of a cells write over the task budgetuse smaller chunks
ZarrGridMismatchA cell whose label the store (or coords) lackswrite values the store's <dimension> labels have, or use mode='append' for new steps of the append dimension
ValueErrorroot, chunks, shards, codecs, file_mode or retain is given outside createdrop them, or create a new store
ValueErrortransactional=False with a table name as target, a mode of "update" or "overwrite", or root, file_mode, retain, write_id, optimize or access="direct"pass the store's path, use create or append, or drop them
ZarrErrortransactional=False and mode="create" at a path that holds a storechoose a new path, add to the store with mode='append', or remove the store first
ZarrErrortransactional=False and mode="append" at a path that holds no storecreate it first with mode='create', or point at the store's path
ZarrGridMismatchtransactional=False append of values that would be cut to fit the stored typeconvert <variable> to the store's type before appending, or write it to a new store
ValueErrorwrite_id is not 32 lowercase hex characterspass uuid.uuid4().hex, or leave write_id unset
ValueErrorretain is not a number and a unitpass a number and a unit, such as retain='7 days'.
ValueErrordata is not a blocks DataFrame or a Datasetpass gbx_zarr_read(output='blocks') output or an xarray.Dataset
ValueErrorA Dataset is written without sparkpass spark=spark
ValueErrorThe DataFrame lacks the blocks columnspass the output of gbx_zarr_read(output='blocks'), or a DataFrame in BLOCKS_SCHEMA
ValueErrorshards is not None, False or a dictpass shards={'time': 48} or None
ValueErrorA shard size is not a multiple of the chunk sizepass a multiple of the chunk size
ValueErrorA shard of the given shards holds more than 4,096 inner chunkspass smaller shards= sizes, or larger chunks=.
ValueErrorAn append finds no CF time dimensionpass append_dim='<dimension>'
ValueErrorThe first label to append is not after the store's lastuse mode="update" to rewrite existing steps or fill a gap an earlier append left
ZarrErrorThe target table of an append, update or overwrite does not existcreate the store with mode='create' (or register an existing one)
ZarrOwnershipErrorcreate names a table that existschoose a new table name, or write to it with mode='append', 'update' or 'overwrite'
ZarrOwnershipErrorcreate names a non-empty rootchoose a new or empty root, or register an existing Zarr store there with gbx_zarr_register
ZarrReadOnlyThe target is a read-only registrationregister the store with writable=True and an overlay_root
ZarrUnsupportedcodecs is not a valid list of codecspass codecs as a list of Zarr v3 codec documents (dicts such as {'name': 'zstd', 'configuration': {'level': 3}}) or zarr codec objects, starting with a serializer such as {'name': 'bytes'}
ZarrUnsupportedA variable holds strings or objectsconvert it to a numeric dtype, or leave it out of variables
ZarrUnsupportedValues cannot be held by the array's type (a NaN into integers with no fill, times between the steps of the units)write values that fit the array's type and attributes, or write its stored values (blocks read with decode=False)
ZarrUnsupportedThe array's time units are not CF time units (h, hrs)store the coordinate with CF time units such as 'hours since 2023-01-01', or write its stored values (blocks read with decode=False)
ZarrGridMismatchThe labels to append are off the store's stepappend '<dimension>' labels on the store's step
ZarrGridMismatchA variable's dimensions differ from the store'swrite the variable on the store's dimensions, in their order
ZarrGridMismatchA block lies outside the array's shapewrite blocks within the store's extent
ZarrGridMismatchA tile is in another CRS than the gridreproject the tiles to the grid's CRS (rst_transform), or write to a store on the tiles' grid
ZarrGridMismatchA tile has another cell size than the gridresample the tiles to the grid's resolution, or write to a store on the tiles' grid
ZarrGridMismatchA tile's origin is off the cell edgesshift or resample the tiles onto the grid's cell edges
ZarrGridMismatchA tile is rotated or not north-upwarp the tiles north-up first (rst_transform), then write them
ZarrWriteConflictAnother writer changed a chunk this write also touches (keys, winner)re-read the store at its latest version and write again
ZarrWriteConflictThe commit lost 20 attempts to other writersretry later or reduce concurrent writers to this array
ZarrErrorThe input of a create or append changed between the plan and the writethe input changed between planning and writing; materialize it as a table first (write it to Delta, then pass spark.table(...))
ZarrErroroptimize=True committed the write, but the fold failedrun gbx_zarr_optimize(spark, '<table>'); the write itself is done.

Register and sync​

gbx_zarr_register​

gbx_zarr_register(spark, path, table, *, writable=False, overlay_root=None, file_mode="fuse",
retain="7 days")

Registers the Zarr store at path as the manifest table table, in place. Nothing is copied and nothing in the store is written. The table's rows point at the store's own objects. Each is recorded with its identity, so that a change made outside GeoBrix is detected on read. Many tables can register one store; each is its own branch with its own history. Returns the table's first store version, the one that holds the registered rows. Progress is printed, one line for each step.

ParameterTypeDescription
sparkSparkSessionThe session.
pathstrThe store's path on a Unity Catalog Volume (or any mounted file system).
tablestrThe manifest table to create. It must not exist.
writableboolDefault False: read-only. True needs overlay_root.
overlay_rootstrA directory outside the store, on a Volume, that this table owns and GeoBrix writes to.
file_modestr"fuse" (default: path strings), "auto" (FILE references where the compute has them, otherwise "fuse") or "external" (FILE EXTERNAL values, governed with the table; needs the File type preview). "managed" is not available.
retainstrHow long superseded versions stay readable. Default "7 days".

An array GeoBrix cannot decode (a string coordinate, say) is registered like any other and listed under unsupported in the commit's metadata. Reading it raises, and reading the rest works. Zero-byte objects and objects that are not chunks of any array are left out, with a warning. An array whose codecs are wrapped around sharding_indexed is registered as one GeoBrix cannot decode, so it is unreadable until it is rewritten. This means anything but sharding_indexed as the only outer codec. zarr-python never writes that.

from databricks.labs.gbx.zarr import gbx_zarr_register

version = gbx_zarr_register(spark, base, table) # the first version of the table

Errors.

ErrorRaised whenThe fix says
ValueErrorwritable=True without overlay_rootpass overlay_root='/Volumes/<catalog>/<schema>/<volume>/<dir>'
ValueErroroverlay_root without writable=Truepass writable=True with it, or drop overlay_root.
ValueErrorretain is not a number and a unitpass a number and a unit, such as retain='7 days'.
ZarrOwnershipErrorThe table already existschoose a new table name, or read or write the existing store
ZarrOwnershipErrorThe overlay root overlaps the storechoose an overlay_root outside the store and not containing it
ZarrOwnershipErrorAnother table owns the overlay rootchoose a different root for <table>, or drop <owner> and remove <marker> and <ledger> if that store is gone
ZarrUnsupportedpath is a DBFS path or not a mounted pathpass the store's /Volumes/<catalog>/<schema>/<volume>/... path
ZarrUnsupportedfile_mode="external" where FILE values are not availableuse file_mode="fuse" (the default)
ZarrUnsupportedAn array's metadata is malformedrepair or rewrite the array's metadata, then register the store again
ZarrErrorpath is not a Zarr storepass the root of a Zarr v2 or v3 store (the directory holding zarr.json or .zgroup)

gbx_zarr_sync​

gbx_zarr_sync(spark, table, *, break_lock=False)

Brings the manifest table of a registered store up to date with its files. It lists the store again and records what other writers changed as one table version, and nothing is committed when nothing changed. Returns a DataFrame with one row per change: change, array and key (a metadata document is zarr.json, .zarray, .zattrs or .zgroup).

ParameterTypeDescription
sparkSparkSessionThe session.
tablestrThe manifest table of a store registered with gbx_zarr_register.
break_lockboolClear a publish lock that is at least an hour old. A younger lock is never broken. Default False.

The change column holds one of:

changeMeaning
changedThe object has a new identity, or new contents.
newThe store gained an object.
deletedThe store lost an object.
bothThe store changed an object GeoBrix has written over. The overlay is kept and still wins on read; the store's change is recorded.

An array's extent becomes the larger of the store's and the overlay's. A sync refuses while a publish holds its lock on the table. A publish that was killed leaves the lock behind; once it is at least an hour old, break_lock=True clears it and the sync goes on. A younger lock is never broken.

from databricks.labs.gbx.zarr import gbx_zarr_sync

changes = gbx_zarr_sync(spark, table)
changes.show()
Example output
+-------+-----+-------+
| change|array| key|
+-------+-----+-------+
|changed| t2m|c/0/0/0|
+-------+-----+-------+

Errors.

ErrorRaised whenThe fix says
ZarrUnsupportedThe table is not a registered storepass the table of a store made by gbx_zarr_register
ZarrUnsupportedAn array changed its data type, chunk grid or codecs outside GeoBrixre-register the store
ZarrWriteConflictA publish holds its lock on the tablewait for the publish to finish. If it was killed and none is running, clear the lock with gbx_zarr_sync(spark, '<table>', break_lock=True), then publish again with gbx_zarr_export(spark, '<table>', method="publish")
ZarrWriteConflictThe table changed while it was being syncedrun gbx_zarr_sync again

Maintain​

gbx_zarr_optimize​

gbx_zarr_optimize(spark, table, *, variables=None, target_file_mb=128)

Folds the chunk rows of a store's sharded arrays into whole shards. Appends and updates add one manifest row per inner chunk they change. This keeps a write cheap but spreads a shard over many small pack files. This copies each shard that has chunk rows, without decoding, into a new pack file as a standard Zarr v3 shard. It writes one shard row in place of the chunk rows. Chunks an all-fill write deleted (tombstones) are folded away. Run it after many small appends, especially on a Volume; it changes no cell. Returns the version the fold committed, or the current version when there was nothing to fold. The current version is the last one that changed data, so an OPTIMIZE or an auto compaction after it does not change it.

ParameterTypeDescription
sparkSparkSessionThe session.
tablestrThe manifest table.
variableslist[str]Limit the fold to these arrays. An unsharded array has nothing to fold.
target_file_mbintThe size cap of a pack file, in MB; a shard larger than that gets its own.

The commit is compare-and-swap: a shard whose rows changed while it ran is skipped, logged as a warning, and folded next time. In a registered store the folded shard goes to the overlay.

from databricks.labs.gbx.zarr import gbx_zarr_optimize

version = gbx_zarr_optimize(spark, table) # the version the fold committed

Errors.

ErrorRaised whenThe fix says
ValueErrorvariables names an array the store lacksname arrays of the store
ValueErrortarget_file_mb is not a whole number of at least 1pass target_file_mb=128
ZarrReadOnlyThe store is a read-only registrationregister the store with writable=True and an overlay_root
ZarrWriteConflictOther writers kept changing the rowsretry once the other writers are done

gbx_zarr_vacuum​

gbx_zarr_vacuum(spark, table, *, retain=None, dry_run=False, force=False)

Deletes the pack files of a store that no readable version needs. Returns a DataFrame with one row per pack file under the store's data root: file (relative to the root), action and reason.

actionreasonMeaning
keptliveA live row points at it.
kepttaggedA tag row points at it.
keptretainedA version within retain points at it, or the write that made it is that recent.
keptrecentIt is under one hour old (a write in flight), or, for a write that never committed, younger than the store's retain.
deleted or would_deletesupersededOnly versions older than retain pointed at it.
deleted or would_deleteuncommittedIts write never committed: a failed, retried or losing write.
ParameterTypeDescription
sparkSparkSessionThe session.
tablestrThe manifest table.
retainstrHow long old versions stay readable. Default: the store's own. A longer value than the store's is refused; under one hour is refused unless force=True.
dry_runboolTrue lists what would go and writes nothing at all.
forceboolAllow a retain under one hour. A write still running can then lose its files and fails on commit.

A vacuum only touches files in the exact layout GeoBrix writes, data/<array>/<write_id>/part-<id>.zpack, under the store's own root. It never deletes the objects of a registered store, anything else in a root, or anything in a directory that another store owns. One store's vacuum can therefore never delete another store's files. A run that deletes records the instant before which versions are no longer promised (geobrix.zarr.vacuum_horizon) before it deletes anything. It raises Delta's retention properties to the store's retain if they are lower. It never deletes a commit row, so a replayed write_id is still recognised.

from databricks.labs.gbx.zarr import gbx_zarr_vacuum

plan = gbx_zarr_vacuum(spark, table, dry_run=True)
plan.groupBy("action", "reason").count().orderBy("action", "reason").show()
Example output
+------+--------+-----+
|action| reason|count|
+------+--------+-----+
| kept| live| 7|
| kept|retained| 3|
+------+--------+-----+

Errors.

ErrorRaised whenThe fix says
ValueErrorretain is under one hourpass retain='1 hour' or more, or force=True if no write is running
ValueErrorretain is longer than the store's ownraise the store's retain first (ALTER TABLE <table> SET TBLPROPERTIES ('<property prefix>retain' = '<retain>')), or tag the version you want to keep with gbx_zarr_tag.
ZarrOwnershipErrorThe root's owner marker is not this table'srun a write, optimize, publish or vacuum (not a dry run) from the table's owner, which claims <root> again; if another store owns the root, create or register this one at a new root
ZarrUnsupportedThe store keeps its files as FILE values and the compute does not support themuse a Databricks Runtime 19 dedicated cluster (single-user, with fileReferenceCreationMode=MANAGED in its cluster config) in a workspace with the File type preview

gbx_zarr_tag​

gbx_zarr_tag(spark, table, name, *, version_as_of=None)

Names a version of the store so it outlives vacuum. It copies the live rows (or those of version_as_of, an int or "tag:<name>") into the manifest as rows with tag = name, in one MERGE. Read it with version_as_of="tag:<name>". Vacuum, Delta's own VACUUM and history expiry never remove what a tag points at. A tag costs one manifest row per shard and chunk of the version. Returns None.

ParameterTypeDescription
sparkSparkSessionThe session.
tablestrThe manifest table.
namestrThe tag: 1 to 128 characters of letters, digits and . _ : / @ + = -, starting with a letter or digit.
version_as_ofint or strThe version to tag, an int or "tag:<name>" to copy another tag. Default: the live store.
from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_tag

gbx_zarr_tag(spark, table, "day-one") # version_as_of= tags an older version
hours = gbx_zarr_read(
spark,
table,
variables=["t2m"],
sel={"latitude": 48.0, "longitude": 2.0},
version_as_of="tag:day-one",
output="table",
).count()
print(hours, "hours at tag:day-one")
Example output
24 hours at tag:day-one

Errors.

ErrorRaised whenThe fix says
ValueErrorThe tag name existspick another name, or remove it first with gbx_zarr_untag
ValueErrorThe name is not 1 to 128 characters of letters, digits and . _ : / @ + = -, starting with a letter or digitpass a name like 'two-days' or 'release/1.0'.
ZarrVersionUnavailableThe version's files are gonethe files were vacuumed; tag a later version or an existing tag
ZarrAccessUnavailableThe version's files cannot be checked, for example while the Volume is not mountedretry when the Volume is reachable
ZarrAccessUnavailableThe version's files cannot be read for lack of permission on the Volumecheck that you have READ VOLUME (and WRITE VOLUME to write) on the Volume, then try again

gbx_zarr_untag​

gbx_zarr_untag(spark, table, name)

Removes a tag: it deletes the rows where tag = name. Its files become collectable by the next vacuum unless a live row or another tag still points at them. Returns None.

ParameterTypeDescription
sparkSparkSessionThe session.
tablestrThe manifest table.
namestrThe tag to remove.
from databricks.labs.gbx.zarr import gbx_zarr_untag

gbx_zarr_untag(spark, table, "day-one")

Errors.

ErrorRaised whenThe fix says
ZarrVersionUnavailableThe tag does not existpass an existing tag name; gbx_zarr_history shows the versions

gbx_zarr_history​

gbx_zarr_history(spark, table)

Returns one row per commit of the store, oldest first, as a DataFrame built from the commit rows and the Delta history. A Delta OPTIMIZE or a property change is not a GeoBrix commit and is not listed.

ColumnHolds
versionThe store version, which is the Delta version.
timestampWhen it committed.
userWho committed it.
modecreate, append, update, overwrite, optimize, tag, untag, restore, register, sync or publish.
write_idThe write's id.
variablesThe arrays it wrote.
regionPer dimension, the [start, stop) it wrote.
from databricks.labs.gbx.zarr import gbx_zarr_history

history = gbx_zarr_history(spark, table)
history.select("version", "mode", "variables").orderBy("version").show(
truncate=False
)
Example output
+-------+------+-------------------------------------+
|version|mode |variables |
+-------+------+-------------------------------------+
|1 |create|[latitude, longitude, msl, t2m, time]|
|2 |append|[msl, t2m, time] |
+-------+------+-------------------------------------+

gbx_zarr_restore​

gbx_zarr_restore(spark, table, *, version_as_of=None, timestamp_as_of=None)

Makes the live rows equal an older version's, as a new commit. Returns the version. Pass version_as_of (an int or "tag:<name>") or timestamp_as_of. The commit upserts the rows the version had and the live rows lack or have changed. The commit also deletes the live rows the version did not have, under compare-and-swap. Nothing is copied and nothing is removed, so older versions stay readable and the restore can itself be undone. A chunk hidden by a tombstone since comes back when the version predates the tombstone. A version already equal to the live rows commits nothing and returns the current version: the last one that changed data. A commit that meets a concurrent one is retried like a write's, up to twenty attempts.

ParameterTypeDescription
sparkSparkSessionThe session.
tablestrThe manifest table.
version_as_ofint or strThe version to restore: an int or "tag:<name>".
timestamp_as_ofstr or datetimeThe version live at that time. Give this or version_as_of.
from databricks.labs.gbx.zarr import gbx_zarr_restore

version = gbx_zarr_restore(spark, table, version_as_of="tag:day-one")

Errors.

ErrorRaised whenThe fix says
ValueErrorNeither version_as_of nor timestamp_as_of is givengive the version, tag or time to restore
ZarrReadOnlyThe store is a read-only registrationregister the store with writable=True and an overlay_root
ZarrVersionUnavailableVacuum has removed the version's filesthe files were vacuumed; restore a later version or a tag
ZarrAccessUnavailableThe version's files cannot be checked, for example while the Volume is not mountedretry when the Volume is reachable
ZarrAccessUnavailableThe version's files cannot be read for lack of permission on the Volumecheck that you have READ VOLUME (and WRITE VOLUME to write) on the Volume, then try again
ZarrVersionUnavailableThe version's base objects changed sincerun gbx_zarr_sync(spark, '<table>') to record the store as it is now, then restore a later version
ZarrWriteConflictOther writers kept changing the rowsretry once the other writers are done

Export and publish​

gbx_zarr_export​

gbx_zarr_export(spark, source, to=None, *, method="copy", version_as_of=None, access="auto")

Exports a version of the store whose manifest table is source. Returns the path to (a str) for "copy" and "reference", and the version of the base store's new commit (an int) for "publish".

ParameterTypeDescription
sparkSparkSessionThe session.
sourcestrThe manifest table of the store (not a path).
tostrThe target: a directory for "copy" (it must not exist, or be empty), a .json path for "reference". Not for "publish".
methodstr"copy" (default), "reference" or "publish".
version_as_ofint or strThe version to export, an int or "tag:<name>". Default: the latest. Not for "publish".
accessstr"reference" only. With "direct", the reference names the cloud URL of each file (s3://, abfss://, gs://) for readers outside Databricks.
  • copy writes a plain Zarr store at canonical keys, in the store's own format. A copy of a Zarr v2 store is Zarr v2, and of a v3 store is v3. It writes the metadata last, so a reader never sees a half-written store. Any tool that reads that format opens it. A store gbx_zarr_write made is Zarr v3, with sharding and the crc32c codec by default, and a GDAL before 3.13 cannot read those (see the Raster Reader). to must be a path every node sees, such as a /Volumes path. Shards that are already in standard form are copied as bytes. Deleted chunks are left out, so readers see the fill value. If the copy fails nothing is left in to.
  • reference writes a kerchunk (version 1) JSON file that maps each key to a file, offset and length in the store's own files; nothing is copied. Every shard must be one region, so run gbx_zarr_optimize first. A reference points at the pack files of one version, so tag that version with gbx_zarr_tag to keep a vacuum from deleting them. It is about 100 bytes per object. The compute that reads a reference needs fsspec 2024.12.0 or later: Databricks runtimes preinstall an older one, which fails with ReferenceNotReachable. GeoBrix does not pin it, because the runtime shares fsspec with s3fs, gcsfs and adlfs.
  • publish writes the overlay of a writable registered store into the store itself, so every Zarr reader sees it, and returns the version it committed. See the Zarr Writer.
from databricks.labs.gbx.zarr import gbx_zarr_export

path = gbx_zarr_export(spark, table, to) # `to` must not exist, or be empty
from databricks.labs.gbx.zarr import (
gbx_zarr_export,
gbx_zarr_optimize,
gbx_zarr_tag,
)

gbx_zarr_optimize(spark, table) # a reference names one range per shard
gbx_zarr_tag(spark, table, "for-reference") # so a vacuum keeps the files it points at
path = gbx_zarr_export(
spark, table, to, method="reference", version_as_of="tag:for-reference"
)
from databricks.labs.gbx.zarr import gbx_zarr_export

version = gbx_zarr_export(spark, table, method="publish")

Errors.

ErrorRaised whenThe fix says
ValueErrormethod is not copy, reference or publishpass 'copy', 'reference' or 'publish'
ValueErrorsource is a pathregister the store with gbx_zarr_register and export its table
ValueErrorpublish is given to or version_as_ofcall gbx_zarr_export(spark, table, method='publish')
ValueErrorA copy target is not a path every node seesexport to a /Volumes path, for example /Volumes/<catalog>/<schema>/<volume>/<dir>
ValueErrorA copy target is not emptypass a directory that does not exist yet, or an empty one
ValueErrorA reference target does not end in .jsonpass a path ending in .json
ValueErrorA reference target existspass a path that does not exist, or remove it
ZarrUnsupportedA reference meets a shard made of chunk rowsrun gbx_zarr_optimize first, or use method="copy"
ZarrReadOnlypublish on a store with no overlayregister the store with writable=True and an overlay_root
ZarrStoreDriftpublish finds the store changed since the last syncrun gbx_zarr_sync(spark, '<table>'), then publish again
ZarrWriteConflictA publish holds its lock on the tablewait for the publish to finish. If it was killed and none is running, clear the lock with gbx_zarr_sync(spark, '<table>', break_lock=True), then publish again with gbx_zarr_export(spark, '<table>', method="publish")
ZarrWriteConflictA write landed on the table during the publish, or Delta rejected the publish's committhe objects this publish already wrote are in the store, so run gbx_zarr_sync(spark, '<table>') to record them, then publish again
ZarrVersionUnavailableThe version was vacuumed or its files are gonethe files were vacuumed; export a later version or a tag
ZarrAccessUnavailableThe version's files cannot be checked, for example while the Volume is not mountedretry when the Volume is reachable
ZarrAccessUnavailableThe version's files cannot be read for lack of permission on the Volumecheck that you have READ VOLUME (and WRITE VOLUME to write) on the Volume, then try again

For a publish, the way back from either ZarrWriteConflict (a write landed, or Delta rejected the commit) or ZarrStoreDrift is gbx_zarr_sync(spark, table) and then the publish again. A lock left by a killed publish is cleared with break_lock=True once it is an hour old. See Publishing.

Errors​

Every error GeoBrix defines is a ZarrError, built as ZarrError(message, fix="..."). str(error) is "<message> Fix: <fix>", and error.fix holds the action on its own. The errors pickle and unpickle with their details, so one raised on a Spark worker reaches the driver intact. From an action, it arrives wrapped in a pyspark.errors.PythonException that keeps the name and the Fix: line. A bad argument is raised as a ValueError instead, with its Fix: at the end of the message in the same way.

from databricks.labs.gbx.zarr import ZarrError, gbx_zarr_read

try:
gbx_zarr_read(spark, table, variables=["t2m"], version_as_of=99)
except ZarrError as err:
print(type(err).__name__)
print(err.fix)
Example output
ZarrVersionUnavailable
pick a later version or tag the version you need with gbx_zarr_tag
ClassRaised when
ZarrErrorThe base class of every error below. Raised itself for a failure with no more specific kind, such as a store that is not a Zarr store or a write whose input changed.
ZarrWriteConflictA concurrent commit changed data this write also touches, or a commit gave up after its attempts. It carries keys (the conflicting store keys), winner (the write_id of the commit that got there first, when known) and conflicts (one dict per conflicting row: kind, array, key, expected_write_id and the live write_id, null when the row is gone).
ZarrStoreDriftA registered store's files no longer match what the manifest recorded.
ZarrReadOnlyThe store or manifest cannot be written from here: a read-only registration, or a table that holds its own data where a registered store is needed.
ZarrGridMismatchA write's array shape, chunking, labels or tile grid do not fit the store's grid.
ZarrVersionUnavailableThe requested version has been vacuumed, never existed, or is past the store's horizon; also a tag that does not exist.
ZarrAccessUnavailableThe credentials or storage access the call needs are not available, such as access="direct" on Serverless.
ZarrUnsupportedThe store, an array or a value uses a feature the call does not support: a data type, a calendar, a codec chain, a file mode the compute lacks.
ZarrOwnershipErrorThe table or root belongs to a different manifest, already exists, or is not owned here.

Next Steps​

  • Weather Features with Zarr — an end-to-end example notebook. It registers a weather cube in place and samples it into a features table. It writes a derived variable back as a new version, maps it as raster tiles, and tags the version for training.
  • Zarr Reader and Zarr Writer — the how-to guides.
  • Zarr Stores — the transactional model, with topic pages for each part of it.
  • Virtual Tiles — the tiles a Zarr read returns.
  • Error Handling — how GeoBrix errors reach you.