Zarr Function Reference
Since: v0.6.0 · (Python level only; no JAR)
The Zarr functions read, write, version and maintain Zarr stores. They are Python
functions in databricks.labs.gbx.zarr and have no SQL binding. A store's manifest is a Delta table, so
SQL can query it. The GeoBrix tiles a read returns work with every
rst_* function. Install the env6 wheel with its [zarr] extra, on Serverless environment 6 or Databricks Runtime 19. See Zarr Stores. Importing the package
imports no large library; importing a name loads only the module that defines it.
from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write
The Weather Features with Zarr notebook uses these functions in order. It registers a weather cube in place and samples it at 100 sites. It writes a derived variable back as a new version, maps it as raster tiles, and tags the version for training.
For how-to guides see the Zarr Reader and the Zarr Writer. For the model, see the topic pages. Every example on this page is run by the documentation tests. They use these names, which stand for what is yours:
# The sample cube, a name for the store's manifest table, and a Volume directory for its files.
cube = "/Volumes/main/geobrix_samples/geobrix-examples/global/era5-weather/era5_like_v3.zarr"
table = "main.weather.era5"
root = "/Volumes/main/weather/stores/era5"
| Function | Does |
|---|---|
gbx_zarr_read | Reads a store, or a cut of one, as blocks, a table of cells or GeoBrix tiles |
gbx_zarr_sample | Reads a store at the points of a DataFrame |
gbx_zarr_write | Creates, appends to, updates or overwrites a store, in one transaction |
gbx_zarr_register | Registers an existing store in place as a table |
gbx_zarr_sync | Brings a registered table up to date with changes made by other tools |
gbx_zarr_optimize | Folds the chunk rows that appends and updates leave into whole shards |
gbx_zarr_vacuum | Deletes the files no readable version needs |
gbx_zarr_tag, gbx_zarr_untag | Names a version so it outlives vacuum; removes the name |
gbx_zarr_history | Lists the commits of a store |
gbx_zarr_restore | Makes the store equal an older version again, as a new commit |
gbx_zarr_export | Takes a version out as a plain Zarr store or a reference file, or publishes an overlay |
BLOCKS_SCHEMA, block_to_numpy, block_to_xarray | The schema of a blocks read, and the helpers that turn a block into an array |
Every error GeoBrix defines is a subclass of ZarrError, and its message ends with a Fix: that says how
to resolve it; see Errors. A mistake in an argument is a plain ValueError, which
except ZarrError does not catch; the ones in the tables below end with a Fix: too. The tables list the
errors you are most likely to meet. Each row gives the start of the fix (all of it where it is short,
and <name> where the message fills in a value).
Learn the model first
New to Zarr in GeoBrix? Read the topic pages in this order. Each opens with what it is for.
| Topic | Read it to |
|---|---|
| Zarr Stores: concepts and store kinds | Understand managed, registered and plain stores, and how a store is files and rows |
| Registering, syncing and ownership | Use a store you already have without copying it, and handle outside changes |
| Transactions and concurrent writers | Know what a write guarantees and what happens when writers meet |
| Versions, time travel and tags | Read, name, restore and export versions |
| Lifecycle, retention and optimize | Free storage and keep reads fast |
| Performance and limits | Size a workload and check what is not supported |
Read and sample
gbx_zarr_read
gbx_zarr_read(spark, source, *, source_type="auto", variables=None, sel=None, isel=None,
output="blocks", version_as_of=None, timestamp_as_of=None, access="auto",
decode=True, virtual_tiles=True, band_dim=None, tile_size=None,
use_zarr_fill_value_as_mask=None)
Reads a Zarr store, or a selection of it, into a DataFrame. The read is planned on the driver and run in
tasks. A table read is pinned to one version of the manifest, so it is consistent whatever commits
meanwhile. Returns a DataFrame (the schema depends on output).
| Parameter | Type | Description |
|---|---|---|
spark | SparkSession | The session. |
source | str | The manifest table of a managed or registered store, or the path of a plain store. |
source_type | str | "auto" (default) takes a string that starts with / or has a URI scheme as a path and anything else as a table. "path" or "table" says which. |
variables | str or list[str] | The arrays to read. Default: every gridded variable; coordinate arrays and scalars are left out. |
sel | dict | Selection by label: dimension to a scalar, an inclusive slice(a, b) or a list. Times may be ISO strings. A longitude slice may cross the seam. |
isel | dict | Selection by position: dimension to an int, a slice (step 1) or a list of ints. |
output | str | "blocks" (default), "table" or "tiles". |
version_as_of | int or str | A version number or "tag:<name>". A plain store has no versions. |
timestamp_as_of | str or datetime | The version live at that time. Not with version_as_of. |
access | str | "auto" (default), "fuse" or "direct"; see Access. |
decode | bool | Apply the CF conventions (scale and offset, missing values, times). Default True; False returns the stored values. |
virtual_tiles | bool | output="tiles" only. True (default): tiles hold no pixels and read their chunks when opened. False: each tile holds GeoTIFF bytes. |
band_dim | str | output="tiles" only. A dimension to stack as the bands of each tile. |
tile_size | tuple[int, int] | output="tiles" only. (width, height) in cells; default one inner chunk. |
use_zarr_fill_value_as_mask | bool or None | xarray's option of the same name. None (default) masks on the array fill_value for Zarr v2 only. |
Returns. output="blocks": BLOCKS_SCHEMA, one row per variable and touched chunk.
output="table": one row per selected cell, with a column per dimension and per variable, typed as
the Reader describes, NULL for missing data, in no defined order.
output="tiles": a (source, tile) DataFrame of GeoBrix tiles.
from databricks.labs.gbx.zarr import gbx_zarr_read
cells = gbx_zarr_read(
spark,
table,
variables=["t2m"],
sel={"time": "2023-01-01T06:00", "latitude": slice(50, 48), "longitude": 2.0},
output="table",
)
cells.orderBy("latitude").show()
+-------------------+--------+---------+------+
| time|latitude|longitude| t2m|
+-------------------+--------+---------+------+
|2023-01-01 06:00:00| 48.0| 2.0| 302.0|
|2023-01-01 06:00:00| 49.0| 2.0|301.75|
|2023-01-01 06:00:00| 50.0| 2.0|301.75|
+-------------------+--------+---------+------+
Errors. A mistake in the call is raised on the driver. The DataFrame is lazy. An error raised while
Spark runs the read arrives from the action as a pyspark.errors.PythonException. Its message keeps
the GeoBrix error name and Fix: line.
| Error | Raised when | The fix says |
|---|---|---|
ValueError | source_type is not auto, path or table | pass 'auto', 'path' or 'table' |
ValueError | output is not blocks, tiles or table | pass 'blocks', 'tiles' or 'table' |
ValueError | access is not auto, fuse or direct | pass 'auto', 'fuse' or 'direct' |
ValueError | Both version_as_of and timestamp_as_of are given | name the version one way. |
ValueError | version_as_of is neither an int nor "tag:<name>" | pass a version number (gbx_zarr_history lists them) or a tag as 'tag:<name>'. |
ValueError | timestamp_as_of is not a string or a datetime, or not a timestamp | pass a timestamp string such as '2023-01-01 06:00:00', or a datetime. |
ValueError | use_zarr_fill_value_as_mask is not True, False or None | pass True, False or leave it None (xarray's default). |
ValueError | A name in variables is not an array of the store | name arrays of the store |
ValueError | A table read names a coordinate in variables | leave the coordinate out of variables, or read it as blocks |
ValueError | A dimension is in both sel and isel | select each dimension one way |
ValueError | sel or isel names a dimension the variables lack | select only existing dimensions |
ValueError | band_dim is not a dimension of the variables | name a dimension of the variables |
ZarrVersionUnavailable | A plain store is read with version_as_of or timestamp_as_of | register the store with gbx_zarr_register to read it as of a version, or drop version_as_of and timestamp_as_of |
ZarrVersionUnavailable | The version does not exist, or is older than the store's retain horizon | pick a later version or tag the version you need with gbx_zarr_tag |
ZarrVersionUnavailable | The tag does not exist | pick an existing tag, or create it with gbx_zarr_tag |
ZarrUnsupported | A variable you named cannot be decoded (strings, or codecs around a shard, say) | read the other variables, or rewrite this array in a form GeoBrix reads |
ZarrUnsupported | A table or tiles are asked for a complex array | read it as blocks, or leave it out of variables |
ZarrUnsupported | No variable can be tiled (no regular latitude and longitude grid, or complex values) | read it with output="blocks" or output="table" |
ZarrUnsupported | The store keeps its files as FILE values and the compute does not support them | use a Databricks Runtime 19 dedicated cluster (single-user, with fileReferenceCreationMode=MANAGED in its cluster config) in a workspace with the File type preview |
ZarrAccessUnavailable | access="direct" is asked of a plain store | register the store with gbx_zarr_register to use direct access, or use access="auto" or "fuse" |
ZarrAccessUnavailable | access="direct" with virtual tiles | use virtual_tiles=False to read the tiles now with direct access, or access="auto" or "fuse" |
ZarrStoreDrift | A registered store's files were changed by another tool (raised in a task) | run gbx_zarr_sync(spark, <table>) on the store's manifest table to record the change, then read again |
gbx_zarr_sample
gbx_zarr_sample(spark, points_df, source, *, geom_col="geom", crs=None, variables=None, sel=None,
method="nearest", version_as_of=None, timestamp_as_of=None, access="auto",
use_zarr_fill_value_as_mask=None)
Samples a store at the points of a DataFrame. Returns points_df with every column it had. Then it adds a column
for each dimension other than latitude and longitude that sel leaves in (time as TIMESTAMP). Last it adds a
column per variable, typed as in a table read. A point is repeated for each selected position of those
dimensions, so one point's time series is its rows. Row order is not defined.
| Parameter | Type | Description |
|---|---|---|
spark | SparkSession | The session. |
points_df | DataFrame | A POINT column geom_col of WKB, EWKB, WKT or EWKT, as rst_sample reads it. Above 100,000 points it is evaluated more than once, so it must be deterministic. |
source | str | The manifest table of a managed or registered store, or the path of a plain store. |
geom_col | str | The geometry column. Default "geom". |
crs | str or int | The CRS of points that carry no SRID. An EWKB or EWKT SRID wins; a point with neither is taken to be in the store's CRS. |
variables | str or list[str] | Default: every gridded variable, which must share a grid. |
sel | dict | Selects the dimensions the points do not choose (time, level), by label. |
method | str | "nearest" (the only one). "bilinear" is planned. |
version_as_of, timestamp_as_of, access, use_zarr_fill_value_as_mask | As in gbx_zarr_read. |
Three cases give NULL. First, a point is outside the grid (more than half a cell beyond its first or last cell).
Second, a geometry is missing or empty. Third, a cell equals the fill value.
from databricks.labs.gbx.zarr import gbx_zarr_sample
points = spark.createDataFrame([("POINT(2.0 48.0)",)], ["geom"])
sampled = gbx_zarr_sample(
spark, points, table, variables=["t2m"], sel={"time": "2023-01-01T06:00"}
)
sampled.show(truncate=False)
+---------------+-------------------+-----+
|geom |time |t2m |
+---------------+-------------------+-----+
|POINT(2.0 48.0)|2023-01-01 06:00:00|302.0|
+---------------+-------------------+-----+
Errors. The errors of gbx_zarr_read for the source, version and access, and:
| Error | Raised when | The fix says |
|---|---|---|
ValueError | points_df has no column geom_col | pass geom_col=<the geometry column> |
ValueError | The column is not BINARY or STRING | pass WKB/EWKB (BINARY) or WKT/EWKT (STRING) points |
ValueError | A geometry is not a POINT | pass points (use a centroid for other shapes) |
ValueError | points_df already has a column the sample adds | rename them in points_df, or sample other variables |
ValueError | sel selects latitude or longitude | select only the other dimensions (time, level) with sel |
ValueError | crs cannot be read | use an EPSG or ESRI code, a CRS string or WKT |
ZarrUnsupported | method is not "nearest" | use method="nearest"; bilinear is planned |
ZarrUnsupported | The array has no latitude and longitude (or y and x) dimensions | name the dimensions latitude/longitude or y/x |
ZarrUnsupported | The coordinates neither rise nor fall | sample a store whose latitude/longitude (or y/x) labels are monotonic |
ZarrGridMismatch | The variables are on different grids | sample variables with the same dimensions, or sample them separately |
ZarrError | points_df changed between planning and reading | make points_df deterministic (write it to a table first), then sample that |
BLOCKS_SCHEMA
The DDL schema string of a blocks read:
variable STRING, dims ARRAY<STRING>, start ARRAY<BIGINT>, stop ARRAY<BIGINT>, coord_start ARRAY<STRING>, coord_stop ARRAY<STRING>, dtype STRING, shape ARRAY<BIGINT>, data BINARY, fill_value STRING, attrs STRING.
Pass it to spark.createDataFrame(rows, BLOCKS_SCHEMA) to build a blocks DataFrame to write. The columns
are described under Output: blocks.
block_to_numpy
block_to_numpy(row)
Returns the cells of a block row (a Row, or a dict from a blocks read) as a writable NumPy array.
The array has the block's shape and dtype, in C order.
block_to_xarray
block_to_xarray(row)
Returns a block row as an xarray.DataArray named for the variable, with the block's dimensions and
attributes. A block carries the first and last label of each dimension rather than every label, so no
coordinates are attached.
from databricks.labs.gbx.zarr import (
BLOCKS_SCHEMA,
block_to_numpy,
block_to_xarray,
gbx_zarr_read,
)
blocks = gbx_zarr_read(spark, table, variables=["t2m"], isel={"time": slice(0, 3)})
row = blocks.orderBy("start").first() # the block at the top left of the grid
cells = block_to_numpy(row) # a numpy array of the block's shape and dtype
named = block_to_xarray(row) # a DataArray named for the variable, with its dimensions
print(cells.shape, cells.dtype)
print(named.name, named.dims)
print(BLOCKS_SCHEMA.split(",")[0]) # the schema, as a DDL string
(3, 181, 295) float32
t2m ('time', 'latitude', 'longitude')
variable STRING
Write
gbx_zarr_write
gbx_zarr_write(data, target, *, mode="create", root=None, variables=None, append_dim=None,
dim_cols=None, coords=None, complete=False, chunks="balanced", shards=None, codecs=None, file_mode=None,
retain="7 days", write_id=None, access="auto", optimize=False, transactional=True,
spark=None)
Writes data to the Zarr store whose manifest table is target, in one transaction. Returns the store
version (the Delta version) the write committed. With transactional=False, target is a path and the
write makes a plain Zarr store instead; it returns a summary dict. See the Zarr Writer for the modes,
layouts, tiles and concurrency in detail.
| Parameter | Type | Description |
|---|---|---|
data | DataFrame or xarray.Dataset | A blocks DataFrame (BLOCKS_SCHEMA), a DataFrame with one GeoBrix tile column, a table of cells (one row per cell, with dim_cols), or a Dataset held on the driver. |
target | str | The manifest table of the store. |
mode | str | "create" (default), "append", "update" or "overwrite". |
root | str | create only. The directory the store's files go to; it must not exist, or be empty. |
variables | str or list[str] | The variables to write. Required in an update or overwrite from blocks to create a variable the store lacks, and it narrows the work. For tiles with no provenance, the single variable they belong to. |
append_dim | str | append only. Default: the first dimension labelled by CF times. |
dim_cols | dict[str, str] | Tiles and tables of cells. Dimension to the column that labels each tile or cell along it. For cells the dict's order is the array's dimension order, every other column is a variable (or the ones variables names), and the mode is create or append. See Write from a table of cells. |
coords | dict[str, array] | Cells only. In a create, the labels of each dimension, in array order, instead of the distinct values of its column sorted ascending. In an append, the new labels of the append dimension only. It saves the pass over that column. |
complete | bool | Cells with coords for every dimension of a create, or for the append dimension of an append. True says every storage unit of that grid holds at least one cell, so the writer skips the scan that finds the units: the table is read once. A unit with no cell ends the write; default False. |
chunks | str or dict | create only. "balanced" (default), "maps", "series" or a dict of dimension to size. |
shards | None, False or dict | create only. None computes the shards, a dict sets some, False means none. |
codecs | list | create only. The inner chunk's codec chain as Zarr v3 codec documents or zarr codec objects. Default bytes, zstd (level 3), crc32c. |
file_mode | str | create only: None (default) is "fuse" (path strings, so the store reads on every compute), "auto" (FILE references where the compute has them, otherwise "fuse") or "external". "managed" is not available in this release. A later write follows the store's own mode and raises if given another. |
retain | str | create only. How long superseded versions stay readable, such as "7 days" (the default). |
write_id | str | 32 lowercase hex characters. Writing again with one that has committed returns its version and writes nothing. |
access | str | "auto" (default), "fuse" or "direct"; see Access. |
optimize | bool | Fold the shards the write touched right after it commits. The version returned is the write's own. |
transactional | bool | True (default) writes a managed store in one Delta commit. False writes a plain Zarr store at the path target: no manifest table, ledger, marker or commit; only mode="create" or "append"; one writer at a time. Returns {"path", "mode", "arrays", "chunks", "bytes"}, not a version. See Plain stores. |
spark | SparkSession | Required for an xarray.Dataset, which has no session of its own. |
from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write
first = gbx_zarr_read(
spark,
cube,
variables=["t2m", "msl", "time", "latitude", "longitude"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T11:00")},
)
version = gbx_zarr_write(first, table, mode="create", root=root) # the first version
Errors. Mistakes in the call are raised on the driver before any file is written. A failed write commits nothing.
| Error | Raised when | The fix says |
|---|---|---|
ValueError | mode is not one of the four | pass one |
ValueError | A table of cells with mode "update" or "overwrite" | use one of those, or write blocks |
ValueError | coords in an update or overwrite, or on data that is not a table of cells | drop coords |
ValueError | coords for a dimension other than the append dimension, in an append of cells | drop coords for <dimensions> |
ValueError | A dim_cols column the table lacks, or no value column | name a column of the table |
ZarrUnsupported | A cells column of a type GeoBrix does not write (a decimal, a string), or a label column that is not a number or a time | cast <column> to a number or a timestamp |
ZarrError | Two cells at one position | keep one row per cell |
ZarrError | A null label in a table of cells | drop or fill the cells without a position before writing |
ZarrError | A chunk of a cells write over the task budget | use smaller chunks |
ZarrGridMismatch | A cell whose label the store (or coords) lacks | write values the store's <dimension> labels have, or use mode='append' for new steps of the append dimension |
ValueError | root, chunks, shards, codecs, file_mode or retain is given outside create | drop them, or create a new store |
ValueError | transactional=False with a table name as target, a mode of "update" or "overwrite", or root, file_mode, retain, write_id, optimize or access="direct" | pass the store's path, use create or append, or drop them |
ZarrError | transactional=False and mode="create" at a path that holds a store | choose a new path, add to the store with mode='append', or remove the store first |
ZarrError | transactional=False and mode="append" at a path that holds no store | create it first with mode='create', or point at the store's path |
ZarrGridMismatch | transactional=False append of values that would be cut to fit the stored type | convert <variable> to the store's type before appending, or write it to a new store |
ValueError | write_id is not 32 lowercase hex characters | pass uuid.uuid4().hex, or leave write_id unset |
ValueError | retain is not a number and a unit | pass a number and a unit, such as retain='7 days'. |
ValueError | data is not a blocks DataFrame or a Dataset | pass gbx_zarr_read(output='blocks') output or an xarray.Dataset |
ValueError | A Dataset is written without spark | pass spark=spark |
ValueError | The DataFrame lacks the blocks columns | pass the output of gbx_zarr_read(output='blocks'), or a DataFrame in BLOCKS_SCHEMA |
ValueError | shards is not None, False or a dict | pass shards={'time': 48} or None |
ValueError | A shard size is not a multiple of the chunk size | pass a multiple of the chunk size |
ValueError | A shard of the given shards holds more than 4,096 inner chunks | pass smaller shards= sizes, or larger chunks=. |
ValueError | An append finds no CF time dimension | pass append_dim='<dimension>' |
ValueError | The first label to append is not after the store's last | use mode="update" to rewrite existing steps or fill a gap an earlier append left |
ZarrError | The target table of an append, update or overwrite does not exist | create the store with mode='create' (or register an existing one) |
ZarrOwnershipError | create names a table that exists | choose a new table name, or write to it with mode='append', 'update' or 'overwrite' |
ZarrOwnershipError | create names a non-empty root | choose a new or empty root, or register an existing Zarr store there with gbx_zarr_register |
ZarrReadOnly | The target is a read-only registration | register the store with writable=True and an overlay_root |
ZarrUnsupported | codecs is not a valid list of codecs | pass codecs as a list of Zarr v3 codec documents (dicts such as {'name': 'zstd', 'configuration': {'level': 3}}) or zarr codec objects, starting with a serializer such as {'name': 'bytes'} |
ZarrUnsupported | A variable holds strings or objects | convert it to a numeric dtype, or leave it out of variables |
ZarrUnsupported | Values cannot be held by the array's type (a NaN into integers with no fill, times between the steps of the units) | write values that fit the array's type and attributes, or write its stored values (blocks read with decode=False) |
ZarrUnsupported | The array's time units are not CF time units (h, hrs) | store the coordinate with CF time units such as 'hours since 2023-01-01', or write its stored values (blocks read with decode=False) |
ZarrGridMismatch | The labels to append are off the store's step | append '<dimension>' labels on the store's step |
ZarrGridMismatch | A variable's dimensions differ from the store's | write the variable on the store's dimensions, in their order |
ZarrGridMismatch | A block lies outside the array's shape | write blocks within the store's extent |
ZarrGridMismatch | A tile is in another CRS than the grid | reproject the tiles to the grid's CRS (rst_transform), or write to a store on the tiles' grid |
ZarrGridMismatch | A tile has another cell size than the grid | resample the tiles to the grid's resolution, or write to a store on the tiles' grid |
ZarrGridMismatch | A tile's origin is off the cell edges | shift or resample the tiles onto the grid's cell edges |
ZarrGridMismatch | A tile is rotated or not north-up | warp the tiles north-up first (rst_transform), then write them |
ZarrWriteConflict | Another writer changed a chunk this write also touches (keys, winner) | re-read the store at its latest version and write again |
ZarrWriteConflict | The commit lost 20 attempts to other writers | retry later or reduce concurrent writers to this array |
ZarrError | The input of a create or append changed between the plan and the write | the input changed between planning and writing; materialize it as a table first (write it to Delta, then pass spark.table(...)) |
ZarrError | optimize=True committed the write, but the fold failed | run gbx_zarr_optimize(spark, '<table>'); the write itself is done. |
Register and sync
gbx_zarr_register
gbx_zarr_register(spark, path, table, *, writable=False, overlay_root=None, file_mode="fuse",
retain="7 days")
Registers the Zarr store at path as the manifest table table, in place. Nothing is copied and
nothing in the store is written. The table's rows point at the store's own objects. Each is recorded with its
identity, so that a change made outside GeoBrix is detected on read. Many tables can register one store;
each is its own branch with its own history. Returns the table's first store version, the one that holds
the registered rows. Progress is printed, one line for each step.
| Parameter | Type | Description |
|---|---|---|
spark | SparkSession | The session. |
path | str | The store's path on a Unity Catalog Volume (or any mounted file system). |
table | str | The manifest table to create. It must not exist. |
writable | bool | Default False: read-only. True needs overlay_root. |
overlay_root | str | A directory outside the store, on a Volume, that this table owns and GeoBrix writes to. |
file_mode | str | "fuse" (default: path strings), "auto" (FILE references where the compute has them, otherwise "fuse") or "external" (FILE EXTERNAL values, governed with the table; needs the File type preview). "managed" is not available. |
retain | str | How long superseded versions stay readable. Default "7 days". |
An array GeoBrix cannot decode (a string coordinate, say) is registered like any other and listed under
unsupported in the commit's metadata. Reading it raises, and reading the rest works. Zero-byte objects
and objects that are not chunks of any array are left out, with a warning. An array whose codecs are
wrapped around sharding_indexed is registered as one GeoBrix cannot decode, so it is unreadable until it is rewritten.
This means anything but sharding_indexed as the only outer codec. zarr-python never writes that.
from databricks.labs.gbx.zarr import gbx_zarr_register
version = gbx_zarr_register(spark, base, table) # the first version of the table
Errors.
| Error | Raised when | The fix says |
|---|---|---|
ValueError | writable=True without overlay_root | pass overlay_root='/Volumes/<catalog>/<schema>/<volume>/<dir>' |
ValueError | overlay_root without writable=True | pass writable=True with it, or drop overlay_root. |
ValueError | retain is not a number and a unit | pass a number and a unit, such as retain='7 days'. |
ZarrOwnershipError | The table already exists | choose a new table name, or read or write the existing store |
ZarrOwnershipError | The overlay root overlaps the store | choose an overlay_root outside the store and not containing it |
ZarrOwnershipError | Another table owns the overlay root | choose a different root for <table>, or drop <owner> and remove <marker> and <ledger> if that store is gone |
ZarrUnsupported | path is a DBFS path or not a mounted path | pass the store's /Volumes/<catalog>/<schema>/<volume>/... path |
ZarrUnsupported | file_mode="external" where FILE values are not available | use file_mode="fuse" (the default) |
ZarrUnsupported | An array's metadata is malformed | repair or rewrite the array's metadata, then register the store again |
ZarrError | path is not a Zarr store | pass the root of a Zarr v2 or v3 store (the directory holding zarr.json or .zgroup) |
gbx_zarr_sync
gbx_zarr_sync(spark, table, *, break_lock=False)
Brings the manifest table of a registered store up to date with its files. It lists the store again and
records what other writers changed as one table version, and nothing is committed when nothing changed.
Returns a DataFrame with one row per change: change, array and key (a metadata document is
zarr.json, .zarray, .zattrs or .zgroup).
| Parameter | Type | Description |
|---|---|---|
spark | SparkSession | The session. |
table | str | The manifest table of a store registered with gbx_zarr_register. |
break_lock | bool | Clear a publish lock that is at least an hour old. A younger lock is never broken. Default False. |
The change column holds one of:
change | Meaning |
|---|---|
changed | The object has a new identity, or new contents. |
new | The store gained an object. |
deleted | The store lost an object. |
both | The store changed an object GeoBrix has written over. The overlay is kept and still wins on read; the store's change is recorded. |
An array's extent becomes the larger of the store's and the overlay's. A sync refuses while a publish holds
its lock on the table. A publish that was killed leaves the lock behind; once it is at least an hour old,
break_lock=True clears it and the sync goes on. A younger lock is never broken.
from databricks.labs.gbx.zarr import gbx_zarr_sync
changes = gbx_zarr_sync(spark, table)
changes.show()
+-------+-----+-------+
| change|array| key|
+-------+-----+-------+
|changed| t2m|c/0/0/0|
+-------+-----+-------+
Errors.
| Error | Raised when | The fix says |
|---|---|---|
ZarrUnsupported | The table is not a registered store | pass the table of a store made by gbx_zarr_register |
ZarrUnsupported | An array changed its data type, chunk grid or codecs outside GeoBrix | re-register the store |
ZarrWriteConflict | A publish holds its lock on the table | wait for the publish to finish. If it was killed and none is running, clear the lock with gbx_zarr_sync(spark, '<table>', break_lock=True), then publish again with gbx_zarr_export(spark, '<table>', method="publish") |
ZarrWriteConflict | The table changed while it was being synced | run gbx_zarr_sync again |
Maintain
gbx_zarr_optimize
gbx_zarr_optimize(spark, table, *, variables=None, target_file_mb=128)
Folds the chunk rows of a store's sharded arrays into whole shards. Appends and updates add one manifest
row per inner chunk they change. This keeps a write cheap but spreads a shard over many small pack
files. This copies each shard that has chunk rows, without decoding, into a new pack file as a standard
Zarr v3 shard. It writes one shard row in place of the chunk rows. Chunks an all-fill write deleted (tombstones) are
folded away. Run it after many small appends, especially on a Volume; it changes no cell. Returns the
version the fold committed, or the current version when there was nothing to fold. The current version is
the last one that changed data, so an OPTIMIZE or an auto compaction after it does not change it.
| Parameter | Type | Description |
|---|---|---|
spark | SparkSession | The session. |
table | str | The manifest table. |
variables | list[str] | Limit the fold to these arrays. An unsharded array has nothing to fold. |
target_file_mb | int | The size cap of a pack file, in MB; a shard larger than that gets its own. |
The commit is compare-and-swap: a shard whose rows changed while it ran is skipped, logged as a warning, and folded next time. In a registered store the folded shard goes to the overlay.
from databricks.labs.gbx.zarr import gbx_zarr_optimize
version = gbx_zarr_optimize(spark, table) # the version the fold committed
Errors.
| Error | Raised when | The fix says |
|---|---|---|
ValueError | variables names an array the store lacks | name arrays of the store |
ValueError | target_file_mb is not a whole number of at least 1 | pass target_file_mb=128 |
ZarrReadOnly | The store is a read-only registration | register the store with writable=True and an overlay_root |
ZarrWriteConflict | Other writers kept changing the rows | retry once the other writers are done |
gbx_zarr_vacuum
gbx_zarr_vacuum(spark, table, *, retain=None, dry_run=False, force=False)
Deletes the pack files of a store that no readable version needs. Returns a DataFrame with one row per pack
file under the store's data root: file (relative to the root), action and reason.
action | reason | Meaning |
|---|---|---|
kept | live | A live row points at it. |
kept | tagged | A tag row points at it. |
kept | retained | A version within retain points at it, or the write that made it is that recent. |
kept | recent | It is under one hour old (a write in flight), or, for a write that never committed, younger than the store's retain. |
deleted or would_delete | superseded | Only versions older than retain pointed at it. |
deleted or would_delete | uncommitted | Its write never committed: a failed, retried or losing write. |
| Parameter | Type | Description |
|---|---|---|
spark | SparkSession | The session. |
table | str | The manifest table. |
retain | str | How long old versions stay readable. Default: the store's own. A longer value than the store's is refused; under one hour is refused unless force=True. |
dry_run | bool | True lists what would go and writes nothing at all. |
force | bool | Allow a retain under one hour. A write still running can then lose its files and fails on commit. |
A vacuum only touches files in the exact layout GeoBrix writes, data/<array>/<write_id>/part-<id>.zpack,
under the store's own root. It never deletes the objects of a registered store, anything else in a root,
or anything in a directory that another store owns. One store's vacuum can therefore never delete another
store's files. A run that deletes records the instant before which versions are no longer promised
(geobrix.zarr.vacuum_horizon) before it deletes anything. It raises Delta's retention properties to the
store's retain if they are lower. It never deletes a commit row, so a replayed write_id is still
recognised.
from databricks.labs.gbx.zarr import gbx_zarr_vacuum
plan = gbx_zarr_vacuum(spark, table, dry_run=True)
plan.groupBy("action", "reason").count().orderBy("action", "reason").show()
+------+--------+-----+
|action| reason|count|
+------+--------+-----+
| kept| live| 7|
| kept|retained| 3|
+------+--------+-----+
Errors.
| Error | Raised when | The fix says |
|---|---|---|
ValueError | retain is under one hour | pass retain='1 hour' or more, or force=True if no write is running |
ValueError | retain is longer than the store's own | raise the store's retain first (ALTER TABLE <table> SET TBLPROPERTIES ('<property prefix>retain' = '<retain>')), or tag the version you want to keep with gbx_zarr_tag. |
ZarrOwnershipError | The root's owner marker is not this table's | run a write, optimize, publish or vacuum (not a dry run) from the table's owner, which claims <root> again; if another store owns the root, create or register this one at a new root |
ZarrUnsupported | The store keeps its files as FILE values and the compute does not support them | use a Databricks Runtime 19 dedicated cluster (single-user, with fileReferenceCreationMode=MANAGED in its cluster config) in a workspace with the File type preview |
gbx_zarr_tag
gbx_zarr_tag(spark, table, name, *, version_as_of=None)
Names a version of the store so it outlives vacuum. It copies the live rows (or those of version_as_of,
an int or "tag:<name>") into the manifest as rows with tag = name, in one MERGE. Read it with
version_as_of="tag:<name>". Vacuum, Delta's own VACUUM and history expiry never remove what a tag points
at. A tag costs one manifest row per shard and chunk of the version. Returns None.
| Parameter | Type | Description |
|---|---|---|
spark | SparkSession | The session. |
table | str | The manifest table. |
name | str | The tag: 1 to 128 characters of letters, digits and . _ : / @ + = -, starting with a letter or digit. |
version_as_of | int or str | The version to tag, an int or "tag:<name>" to copy another tag. Default: the live store. |
from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_tag
gbx_zarr_tag(spark, table, "day-one") # version_as_of= tags an older version
hours = gbx_zarr_read(
spark,
table,
variables=["t2m"],
sel={"latitude": 48.0, "longitude": 2.0},
version_as_of="tag:day-one",
output="table",
).count()
print(hours, "hours at tag:day-one")
24 hours at tag:day-one
Errors.
| Error | Raised when | The fix says |
|---|---|---|
ValueError | The tag name exists | pick another name, or remove it first with gbx_zarr_untag |
ValueError | The name is not 1 to 128 characters of letters, digits and . _ : / @ + = -, starting with a letter or digit | pass a name like 'two-days' or 'release/1.0'. |
ZarrVersionUnavailable | The version's files are gone | the files were vacuumed; tag a later version or an existing tag |
ZarrAccessUnavailable | The version's files cannot be checked, for example while the Volume is not mounted | retry when the Volume is reachable |
ZarrAccessUnavailable | The version's files cannot be read for lack of permission on the Volume | check that you have READ VOLUME (and WRITE VOLUME to write) on the Volume, then try again |
gbx_zarr_untag
gbx_zarr_untag(spark, table, name)
Removes a tag: it deletes the rows where tag = name. Its files become collectable by the next vacuum unless
a live row or another tag still points at them. Returns None.
| Parameter | Type | Description |
|---|---|---|
spark | SparkSession | The session. |
table | str | The manifest table. |
name | str | The tag to remove. |
from databricks.labs.gbx.zarr import gbx_zarr_untag
gbx_zarr_untag(spark, table, "day-one")
Errors.
| Error | Raised when | The fix says |
|---|---|---|
ZarrVersionUnavailable | The tag does not exist | pass an existing tag name; gbx_zarr_history shows the versions |
gbx_zarr_history
gbx_zarr_history(spark, table)
Returns one row per commit of the store, oldest first, as a DataFrame built from the commit rows and the Delta
history. A Delta OPTIMIZE or a property change is not a GeoBrix commit and is not listed.
| Column | Holds |
|---|---|
version | The store version, which is the Delta version. |
timestamp | When it committed. |
user | Who committed it. |
mode | create, append, update, overwrite, optimize, tag, untag, restore, register, sync or publish. |
write_id | The write's id. |
variables | The arrays it wrote. |
region | Per dimension, the [start, stop) it wrote. |
from databricks.labs.gbx.zarr import gbx_zarr_history
history = gbx_zarr_history(spark, table)
history.select("version", "mode", "variables").orderBy("version").show(
truncate=False
)
+-------+------+-------------------------------------+
|version|mode |variables |
+-------+------+-------------------------------------+
|1 |create|[latitude, longitude, msl, t2m, time]|
|2 |append|[msl, t2m, time] |
+-------+------+-------------------------------------+
gbx_zarr_restore
gbx_zarr_restore(spark, table, *, version_as_of=None, timestamp_as_of=None)
Makes the live rows equal an older version's, as a new commit. Returns the version. Pass version_as_of (an
int or "tag:<name>") or timestamp_as_of. The commit upserts the rows the version had and the live rows
lack or have changed. The commit also deletes the live rows the version did not have, under compare-and-swap. Nothing is
copied and nothing is removed, so older versions stay readable and the restore can itself be undone. A
chunk hidden by a tombstone since comes back when the version predates the tombstone. A version already
equal to the live rows commits nothing and returns the current version: the last one that changed data. A commit that meets a concurrent
one is retried like a write's, up to twenty attempts.
| Parameter | Type | Description |
|---|---|---|
spark | SparkSession | The session. |
table | str | The manifest table. |
version_as_of | int or str | The version to restore: an int or "tag:<name>". |
timestamp_as_of | str or datetime | The version live at that time. Give this or version_as_of. |
from databricks.labs.gbx.zarr import gbx_zarr_restore
version = gbx_zarr_restore(spark, table, version_as_of="tag:day-one")
Errors.
| Error | Raised when | The fix says |
|---|---|---|
ValueError | Neither version_as_of nor timestamp_as_of is given | give the version, tag or time to restore |
ZarrReadOnly | The store is a read-only registration | register the store with writable=True and an overlay_root |
ZarrVersionUnavailable | Vacuum has removed the version's files | the files were vacuumed; restore a later version or a tag |
ZarrAccessUnavailable | The version's files cannot be checked, for example while the Volume is not mounted | retry when the Volume is reachable |
ZarrAccessUnavailable | The version's files cannot be read for lack of permission on the Volume | check that you have READ VOLUME (and WRITE VOLUME to write) on the Volume, then try again |
ZarrVersionUnavailable | The version's base objects changed since | run gbx_zarr_sync(spark, '<table>') to record the store as it is now, then restore a later version |
ZarrWriteConflict | Other writers kept changing the rows | retry once the other writers are done |
Export and publish
gbx_zarr_export
gbx_zarr_export(spark, source, to=None, *, method="copy", version_as_of=None, access="auto")
Exports a version of the store whose manifest table is source. Returns the path to (a str) for
"copy" and "reference", and the version of the base store's new commit (an int) for "publish".
| Parameter | Type | Description |
|---|---|---|
spark | SparkSession | The session. |
source | str | The manifest table of the store (not a path). |
to | str | The target: a directory for "copy" (it must not exist, or be empty), a .json path for "reference". Not for "publish". |
method | str | "copy" (default), "reference" or "publish". |
version_as_of | int or str | The version to export, an int or "tag:<name>". Default: the latest. Not for "publish". |
access | str | "reference" only. With "direct", the reference names the cloud URL of each file (s3://, abfss://, gs://) for readers outside Databricks. |
copywrites a plain Zarr store at canonical keys, in the store's own format. A copy of a Zarr v2 store is Zarr v2, and of a v3 store is v3. It writes the metadata last, so a reader never sees a half-written store. Any tool that reads that format opens it. A storegbx_zarr_writemade is Zarr v3, with sharding and thecrc32ccodec by default, and a GDAL before 3.13 cannot read those (see the Raster Reader).tomust be a path every node sees, such as a/Volumespath. Shards that are already in standard form are copied as bytes. Deleted chunks are left out, so readers see the fill value. If the copy fails nothing is left into.referencewrites a kerchunk (version 1) JSON file that maps each key to a file, offset and length in the store's own files; nothing is copied. Every shard must be one region, so rungbx_zarr_optimizefirst. A reference points at the pack files of one version, so tag that version withgbx_zarr_tagto keep a vacuum from deleting them. It is about 100 bytes per object. The compute that reads a reference needsfsspec2024.12.0 or later: Databricks runtimes preinstall an older one, which fails withReferenceNotReachable. GeoBrix does not pin it, because the runtime sharesfsspecwiths3fs,gcsfsandadlfs.publishwrites the overlay of a writable registered store into the store itself, so every Zarr reader sees it, and returns the version it committed. See the Zarr Writer.
from databricks.labs.gbx.zarr import gbx_zarr_export
path = gbx_zarr_export(spark, table, to) # `to` must not exist, or be empty
from databricks.labs.gbx.zarr import (
gbx_zarr_export,
gbx_zarr_optimize,
gbx_zarr_tag,
)
gbx_zarr_optimize(spark, table) # a reference names one range per shard
gbx_zarr_tag(spark, table, "for-reference") # so a vacuum keeps the files it points at
path = gbx_zarr_export(
spark, table, to, method="reference", version_as_of="tag:for-reference"
)
from databricks.labs.gbx.zarr import gbx_zarr_export
version = gbx_zarr_export(spark, table, method="publish")
Errors.
| Error | Raised when | The fix says |
|---|---|---|
ValueError | method is not copy, reference or publish | pass 'copy', 'reference' or 'publish' |
ValueError | source is a path | register the store with gbx_zarr_register and export its table |
ValueError | publish is given to or version_as_of | call gbx_zarr_export(spark, table, method='publish') |
ValueError | A copy target is not a path every node sees | export to a /Volumes path, for example /Volumes/<catalog>/<schema>/<volume>/<dir> |
ValueError | A copy target is not empty | pass a directory that does not exist yet, or an empty one |
ValueError | A reference target does not end in .json | pass a path ending in .json |
ValueError | A reference target exists | pass a path that does not exist, or remove it |
ZarrUnsupported | A reference meets a shard made of chunk rows | run gbx_zarr_optimize first, or use method="copy" |
ZarrReadOnly | publish on a store with no overlay | register the store with writable=True and an overlay_root |
ZarrStoreDrift | publish finds the store changed since the last sync | run gbx_zarr_sync(spark, '<table>'), then publish again |
ZarrWriteConflict | A publish holds its lock on the table | wait for the publish to finish. If it was killed and none is running, clear the lock with gbx_zarr_sync(spark, '<table>', break_lock=True), then publish again with gbx_zarr_export(spark, '<table>', method="publish") |
ZarrWriteConflict | A write landed on the table during the publish, or Delta rejected the publish's commit | the objects this publish already wrote are in the store, so run gbx_zarr_sync(spark, '<table>') to record them, then publish again |
ZarrVersionUnavailable | The version was vacuumed or its files are gone | the files were vacuumed; export a later version or a tag |
ZarrAccessUnavailable | The version's files cannot be checked, for example while the Volume is not mounted | retry when the Volume is reachable |
ZarrAccessUnavailable | The version's files cannot be read for lack of permission on the Volume | check that you have READ VOLUME (and WRITE VOLUME to write) on the Volume, then try again |
For a publish, the way back from either ZarrWriteConflict (a write landed, or Delta rejected the commit) or
ZarrStoreDrift is gbx_zarr_sync(spark, table) and then the publish again. A lock left by a killed publish
is cleared with break_lock=True once it is an hour old. See Publishing.
Errors
Every error GeoBrix defines is a ZarrError, built as ZarrError(message, fix="..."). str(error) is
"<message> Fix: <fix>", and error.fix holds the action on its own. The errors pickle and unpickle with
their details, so one raised on a Spark worker reaches the driver intact. From an action, it arrives wrapped in a
pyspark.errors.PythonException that keeps the name and the Fix: line. A bad argument is raised as a
ValueError instead, with its Fix: at the end of the message in the same way.
from databricks.labs.gbx.zarr import ZarrError, gbx_zarr_read
try:
gbx_zarr_read(spark, table, variables=["t2m"], version_as_of=99)
except ZarrError as err:
print(type(err).__name__)
print(err.fix)
ZarrVersionUnavailable
pick a later version or tag the version you need with gbx_zarr_tag
| Class | Raised when |
|---|---|
ZarrError | The base class of every error below. Raised itself for a failure with no more specific kind, such as a store that is not a Zarr store or a write whose input changed. |
ZarrWriteConflict | A concurrent commit changed data this write also touches, or a commit gave up after its attempts. It carries keys (the conflicting store keys), winner (the write_id of the commit that got there first, when known) and conflicts (one dict per conflicting row: kind, array, key, expected_write_id and the live write_id, null when the row is gone). |
ZarrStoreDrift | A registered store's files no longer match what the manifest recorded. |
ZarrReadOnly | The store or manifest cannot be written from here: a read-only registration, or a table that holds its own data where a registered store is needed. |
ZarrGridMismatch | A write's array shape, chunking, labels or tile grid do not fit the store's grid. |
ZarrVersionUnavailable | The requested version has been vacuumed, never existed, or is past the store's horizon; also a tag that does not exist. |
ZarrAccessUnavailable | The credentials or storage access the call needs are not available, such as access="direct" on Serverless. |
ZarrUnsupported | The store, an array or a value uses a feature the call does not support: a data type, a calendar, a codec chain, a file mode the compute lacks. |
ZarrOwnershipError | The table or root belongs to a different manifest, already exists, or is not owned here. |
Next Steps
- Weather Features with Zarr — an end-to-end example notebook. It registers a weather cube in place and samples it into a features table. It writes a derived variable back as a new version, maps it as raster tiles, and tags the version for training.
- Zarr Reader and Zarr Writer — the how-to guides.
- Zarr Stores — the transactional model, with topic pages for each part of it.
- Virtual Tiles — the tiles a Zarr read returns.
- Error Handling — how GeoBrix errors reach you.