Skip to main content

Zarr Reader

Since: zarr_gbx v0.6.0

Read a Zarr store, or a cut of one, into Spark. Three entry points share one reader:

Entry pointReadsUse it for
gbx_zarr_readA plain store (a path), or the manifest table of a managed or registered storeEverything: versions, selections by label or position, and the three outputs (blocks, a table of cells, GeoBrix tiles)
spark.read.format("zarr_gbx")A plain store (a path)A read that is a Spark source, with no call to a function. A DataSource has no session to read a manifest with, so it reads plain stores only
gbx_zarr_sampleA plain store or a table, at the points of a DataFrameThe time series at a set of points

All three run at the Python level, so there is no JAR to install. Install the env6 wheel with its [zarr] extra, on Serverless environment 6 or Databricks Runtime 19. Zarr Stores describes this. To write a store back see the Zarr Writer; for every parameter, return value and error see the Zarr Function Reference. The examples use two names that stand for what is yours:

# A Zarr store is a path; a managed or registered store is the name of its manifest table.
cube = "/Volumes/main/geobrix_samples/geobrix-examples/global/era5-weather/era5_like_v3.zarr"
table = "main.weather.era5"

The sample cube is the one Zarr Stores generates. It is two days of made-up hourly weather on a one-degree global grid, time (48) by latitude (181) by longitude (360), with the variables t2m, u10, v10 and msl. Every code block on this page is run by the documentation tests.

Choosing the source​

gbx_zarr_read(spark, source, ...) takes the source as a string. source_type="auto" (the default) treats a string that starts with / or has a URI scheme (dbfs:/, file:/, s3://) as the path of a plain store. It treats anything else as the name of a manifest table. Pass source_type="path" or "table" to say which.

  • A plain store is read as it is on disk, with whatever bytes are there. It has no versions.
  • A table is a managed store, or a store registered in place. A read pins one version of the manifest when it is planned: the latest, or the one named by version_as_of (a version number or "tag:<name>") or timestamp_as_of. Pass one of the two, not both. Whatever commits meanwhile, the read sees one consistent snapshot.
from databricks.labs.gbx.zarr import gbx_zarr_read

def hours(**version):
return gbx_zarr_read(
spark,
table,
variables=["t2m"],
sel={"latitude": 48.0, "longitude": 2.0},
output="table",
**version,
).count()

print("latest:", hours(), "hours")
print("version 1:", hours(version_as_of=1), "hours")
print("tag first-day:", hours(version_as_of="tag:first-day"), "hours")
Example output
latest: 24 hours
version 1: 12 hours
tag first-day: 12 hours

A version older than the store's retain period raises ZarrVersionUnavailable; see Lifecycle and retention. A store made with file_mode="external" holds FILE values in its manifest. It must be read on compute that supports them. Other compute refuses with a ZarrUnsupported whose Fix: names the options (see File modes).

Choosing what to read​

variables is a name or a list of names. The default is every gridded variable: coordinate arrays, the arrays that CF coordinates and bounds attributes name, and scalars are left out. An array GeoBrix cannot decode (a string coordinate, say) does not fail the read. It is left out and named only if you ask for it, select by its labels, or need its labels for a table.

sel selects by label and isel by position. Each is a dict of dimension to a value:

Valuesel (labels)isel (positions)
A scalarThe exact label, else the nearest label within half the axis step. The dimension stays, with length oneAn index. The dimension stays, with length one
A slice(a, b)Both ends included, in the order of the axis, so a descending latitude takes slice(60, 50). A time takes ISO stringsstart:stop with the end excluded. A step other than 1 is refused
A listThose labels; the output follows the array's order, not the list'sThose positions

A longitude axis that covers the whole circle is periodic. A slice may cross its seam, so slice(-10, 10) on a 0 to 360 axis is the labels from 350 round to 10. A label that is not on the axis raises ValueError when it is a scalar, and selects nothing when it is a slice. Variables that lack a selected dimension are read whole along the others.

from databricks.labs.gbx.zarr import gbx_zarr_read

by_label = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={
"time": slice("2023-01-01T00:00", "2023-01-01T02:00"),
"latitude": slice(60, 50), # latitude runs north to south
"longitude": slice(0, 10), # both ends are included
},
output="table",
)
by_position = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
isel={
"time": slice(0, 3), # positions: the end is not included
"latitude": slice(30, 41),
"longitude": slice(0, 11),
},
output="table",
)
print("by label:", by_label.count(), "cells")
print("by position:", by_position.count(), "cells")
Example output
by label: 363 cells
by position: 363 cells

Output: blocks​

output="blocks" (the default) returns one row per variable and touched chunk, holding that chunk's selected cells. It is the form to write back with gbx_zarr_write, and the form that keeps every dtype, including complex ones.

ColumnTypeHolds
variableSTRINGThe array's name
dimsARRAY<STRING>Its dimension names, in order
start, stopARRAY<BIGINT>Where the block sits in the array, per dimension: [start, stop) in cells
coord_start, coord_stopARRAY<STRING>The first and last label of each dimension
dtypeSTRINGThe NumPy dtype of data, such as <f4
shapeARRAY<BIGINT>The block's shape
dataBINARYThe cells, C-order bytes
fill_valueSTRINGWhat a never-written cell holds
attrsSTRINGThe array's attributes, as JSON

block_to_numpy(row) turns a row into a NumPy array of the block's shape and dtype (a writable copy), and block_to_xarray(row) into a named xarray.DataArray. A block carries the first and last label of each dimension rather than every label, so the DataArray has no coordinates attached.

from databricks.labs.gbx.zarr import block_to_numpy, gbx_zarr_read

blocks = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T02:00")},
)
blocks.orderBy("start").select("variable", "dims", "start", "stop", "shape").show(
truncate=False
)
cells = block_to_numpy(blocks.orderBy("start").first()) # a numpy array
print(cells.shape, cells.dtype)
Example output
+--------+---------------------------+------------+-------------+------------+
|variable|dims |start |stop |shape |
+--------+---------------------------+------------+-------------+------------+
|t2m |[time, latitude, longitude]|[0, 0, 0] |[3, 91, 180] |[3, 91, 180]|
|t2m |[time, latitude, longitude]|[0, 0, 180] |[3, 91, 360] |[3, 91, 180]|
|t2m |[time, latitude, longitude]|[0, 91, 0] |[3, 181, 180]|[3, 90, 180]|
|t2m |[time, latitude, longitude]|[0, 91, 180]|[3, 181, 360]|[3, 90, 180]|
+--------+---------------------------+------------+-------------+------------+

(3, 91, 180) float32

decode=True (the default) applies the CF conventions the array declares: scale and offset, missing values to NaN, and times to datetime64[ns]. The attributes it used are dropped from the block's attrs. decode=False returns the stored values. A time axis that is not in the standard calendars, or outside the years 1678 to 2262, raises ZarrUnsupported; read it with decode=False.

Output: a table of cells​

output="table" returns one row per selected cell, with a column per dimension and a column per variable. A dimension's column holds its labels, or the position as BIGINT when the dimension has no coordinate array. The variables must share their dimensions and shape. Row order is not defined.

from databricks.labs.gbx.zarr import gbx_zarr_read

europe = gbx_zarr_read(
spark,
cube,
variables=["t2m", "msl"],
sel={
"time": slice("2023-01-01T00:00", "2023-01-01T01:00"),
"latitude": slice(60, 40),
"longitude": slice(0, 10),
},
output="table",
)
europe.printSchema()
europe.orderBy("time", "latitude", "longitude").show(3)
Example output
root
|-- time: timestamp (nullable = true)
|-- latitude: double (nullable = true)
|-- longitude: double (nullable = true)
|-- t2m: float (nullable = true)
|-- msl: float (nullable = true)

+-------------------+--------+---------+-----+--------+
| time|latitude|longitude| t2m| msl|
+-------------------+--------+---------+-----+--------+
|2023-01-01 00:00:00| 40.0| 0.0|299.5|100580.0|
|2023-01-01 00:00:00| 40.0| 1.0|299.5|100570.0|
|2023-01-01 00:00:00| 40.0| 2.0|299.5|100580.0|
+-------------------+--------+---------+-----+--------+
only showing top 3 rows

Column types​

A column keeps the type of the values it was made from, so a table never widens a count to a double:

StoredColumn
int8TINYINT
int16, uint8SMALLINT
int32, uint16INT
int64, uint32BIGINT
uint64DECIMAL(20,0)
boolBOOLEAN
float16, float32FLOAT
float64DOUBLE
CF timesTIMESTAMP, in UTC, truncated to the microsecond
A packed array (scale_factor, add_offset)FLOAT or DOUBLE, since it decodes to floating point

A time column is a UTC instant. It does not move with the session time zone, and a time finer than a microsecond is truncated. Complex arrays raise ZarrUnsupported as a table or as tiles: read them as blocks.

Missing data is NULL​

A cell equal to the array's _FillValue or missing_value is NULL. The comparison is made on the stored values, before any scaling or cast, so an integer array stays an integer array:

from databricks.labs.gbx.zarr import gbx_zarr_read

rain = gbx_zarr_read(spark, gauges, variables=["rain"], output="table")
rain.printSchema()
rain.orderBy("day", "station").show()
Example output
root
|-- day: timestamp (nullable = true)
|-- station: long (nullable = true)
|-- rain: short (nullable = true)

+-------------------+-------+----+
| day|station|rain|
+-------------------+-------+----+
|2023-06-01 00:00:00| 101| 0|
|2023-06-01 00:00:00| 102| 12|
|2023-06-01 00:00:00| 103|NULL|
|2023-06-02 00:00:00| 101| 3|
|2023-06-02 00:00:00| 102|NULL|
|2023-06-02 00:00:00| 103| 8|
|2023-06-03 00:00:00| 101| 1|
|2023-06-03 00:00:00| 102| 2|
|2023-06-03 00:00:00| 103| 0|
+-------------------+-------+----+

With decode=False the table holds the stored values and nothing is NULL. A few details decide which cells count as missing:

  • A NaN fill value gives NULL cells. Where the fill is NaN, every NaN cell is missing. A NaN that is data beside a numeric fill stays NaN.
  • The array's fill_value is only a marker where xarray's rule says so. Zarr v2 has no _FillValue attribute, so use_zarr_fill_value_as_mask=None (the default, xarray's own) masks on the array fill_value for Zarr v2 and not for Zarr v3. An integer v2 array whose fill_value is 0 therefore reads its zeros as NULL by default. use_zarr_fill_value_as_mask=False keeps them, and True masks on fill_value in either version. The zarr_gbx DataSource has no option for this and uses the default.
  • An integer array read as booleans (a dtype attribute of "bool") is masked only on an explicit _FillValue or missing_value, never on the array's fill_value. Zarr's default of 0 is a stored False, not a missing cell.
from databricks.labs.gbx.zarr import gbx_zarr_read

for flag in (None, False):
cells = gbx_zarr_read(
spark,
counts,
variables=["counts"],
output="table",
use_zarr_fill_value_as_mask=flag,
)
print(flag, "->", cells.where("counts IS NULL").count(), "NULL cells")
Example output
None -> 2 NULL cells
False -> 0 NULL cells
Where tables differ from xarray

GeoBrix follows the Zarr store's own semantics, which differ from xarray's in three places.

  • Masked integers stay integers. xarray decodes a masked integer array to floating point so that the missing cells can be NaN. A table keeps the integer type and writes NULL.
  • The mask flag. xarray.open_zarr ignores use_zarr_fill_value_as_mask for the arrays it opens; gbx_zarr_read honours it, as ZarrStore does.
  • Booleans with a stored fill. For an integer array read as booleans that has an explicit fill, a cell equal to the fill is NULL in a table. Blocks follow xarray, which casts the masked cell's NaN to True. Neither marks the array's default fill_value of 0 as missing.

Output: GeoBrix tiles​

output="tiles" returns the (source, tile) DataFrame the other raster readers return, so every rst_* function applies to Zarr data without converting the cube to GeoTIFF. Each selected 2-D plane of each variable (the plane of its latitude and longitude, or y and x) is cut into windows, one inner chunk each. Every selected position along the other dimensions makes its own tile. The tile's metadata says which: zarr_array, sliceDims (such as time=0) and sliceCoord_<dim> (the label), as the NetCDF reader writes them.

  • tile_size=(width, height), in cells, overrides the window. A window never crosses the longitude seam. The largest tile a task may hold is 64 MiB decoded on Serverless and 256 MiB on classic clusters.
  • band_dim="time" stacks one dimension as the bands of one tile instead of making a tile for each position.
  • Tiles are north-up: when latitude ascends, the rows are flipped. The CRS is the array's grid_mapping, or EPSG:4326 for a plain latitude and longitude grid, and NoData is NaN for floating-point cells.
  • A 0 to 360 longitude axis gives EPSG:4326 tiles with x above 180. A cube whose longitude runs from 0 to 359 has tiles that reach x = 359.5. GeoBrix does not shift them into -180..180. Keep that in mind when you combine the tiles with rasters in the usual -180..180 convention.
  • The coordinates must form a regular grid over the selected cells (a constant step; what lies outside the selection does not matter). An irregular or curvilinear grid raises ZarrUnsupported: read it as blocks or a table. A variable with no such grid is left out with a warning, unless you named it in variables, which raises.
from pyspark.sql import functions as F

from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.zarr import gbx_zarr_read

tiles = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={"time": "2023-01-01T00:00"},
output="tiles",
)
tiles.select(
F.col("tile.metadata")["sliceDims"].alias("slice"),
"tile.window.col_off",
"tile.window.row_off",
"tile.window.width",
"tile.window.height",
rx.rst_crs("tile").alias("crs"), # answered from the header: no chunk is read
).orderBy("row_off", "col_off").show(truncate=False)
Example output
+------+-------+-------+-----+------+---------+
|slice |col_off|row_off|width|height|crs |
+------+-------+-------+-----+------+---------+
|time=0|0 |0 |180 |91 |EPSG:4326|
|time=0|180 |0 |180 |91 |EPSG:4326|
|time=0|0 |91 |180 |90 |EPSG:4326|
|time=0|180 |91 |180 |90 |EPSG:4326|
+------+-------+-------+-----+------+---------+

Virtual tiles and the rst_* functions​

With virtual_tiles=True (the default) a tile holds no pixels. It names its window and the storage objects behind it, and reads them when a function opens it. A table of tiles can therefore be saved and read back by tasks that have no session. This holds as long as the store stays on a Volume and, for a table, the pinned version stays readable. The Python rst_* functions open a tile on demand and read only the chunks its window touches. A header function such as rst_crs or rst_width reads no chunk at all.

from pyspark.sql import functions as F

from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.zarr import gbx_zarr_read

tiles = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={"time": "2023-01-01T00:00"},
output="tiles",
)
stats = tiles.select(
rx.rst_width("tile").alias("width"),
rx.rst_height("tile").alias("height"),
F.round(rx.rst_avg("tile").getItem(0), 1).alias("mean"),
)
stats.orderBy("height", "mean").show()
Example output
+-----+------+-----+
|width|height| mean|
+-----+------+-----+
| 180| 90|294.9|
| 180| 90|300.0|
| 180| 91|295.0|
| 180| 91|300.1|
+-----+------+-----+

band_dim stacks a dimension as bands, so six hours become the six bands of one tile:

from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.zarr import gbx_zarr_read

tiles = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T05:00")},
band_dim="time",
output="tiles",
)
tiles.select(rx.rst_numbands("tile").alias("bands")).distinct().show()
Example output
+-----+
|bands|
+-----+
| 6|
+-----+

Materialize for the JVM level​

The JVM level reads materialized tiles only. In SQL, GeoBrix sends a virtual tile to the Python level, so a gbx_rst_* call takes one on every compute (see Virtual Tiles). virtual_tiles=False reads each tile now and holds its pixels as GeoTIFF bytes, which both levels accept. It costs the read up front and the memory of the pixels, so keep the windows small.

from pyspark.sql import functions as F

from databricks.labs.gbx.zarr import gbx_zarr_read

tiles = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={"time": "2023-01-01T00:00"},
output="tiles",
virtual_tiles=False,
)
tiles.select((F.length("tile.raster") > 0).alias("holds_pixels")).distinct().show()
Example output
+------------+
|holds_pixels|
+------------+
| true|
+------------+

Writing tiles back​

A tile cut from Zarr still carries where it came from, so it can be written back to the array it was cut from. A pixel-producing rst_* function (a map algebra, a clip, a resample) returns a fresh tile without that provenance. Put the provenance back with withField("metadata", ...), or give gbx_zarr_write dim_cols and variables= to say where the tile goes. The Zarr Writer shows both.

How a read is split into tasks​

A blocks or table read packs whole chunks into tasks of about 16 MiB decoded. A read too small for that is spread over the cores of the session instead, down to 4 MiB per task. A 100 MB read therefore does not run as one task. A task never holds more than the compute's decoded-data limit (64 MiB on Serverless, 1536 MiB on a classic cluster), whatever the target. The cores are read from the session's configuration, so the plan is the same on Serverless, where there is no SparkContext. On a Databricks cluster that sets no parallelism or executor configuration, the cores are the cluster's worker count times the cores of a worker. The cores of a worker are taken from the driver when both use the same node type. A cluster of mixed node types is planned as one slot. A chunk counts only the cells inside the array: a 24-hour chunk of a 1-hour array counts one hour.

A plain store on a Volume is decoded by a thread pool larger than the core count. This is because each chunk read waits on the mount. Its size is eight threads per core, at most 64, lowered when the chunks the threads hold would not fit in the task's decoded-data limit. Set RAYON_NUM_THREADS before the first read of a process to choose it yourself. A store registered in a table reads its chunks through the manifest, which overlaps its reads already.

The zarr_gbx DataSource​

spark.read.format("zarr_gbx") reads a plain store with no table, no session call and no JAR. Register the *_gbx DataSources once per session. Options are strings, like the other *_gbx readers.

from databricks.labs import gbx

gbx.register(spark)
europe = (
spark.read.format("zarr_gbx")
.option("variables", "t2m")
.option("sel", '{"time": ["2023-01-01T00:00", "2023-01-01T01:00"]}')
.option("isel", '{"latitude": [30, 51], "longitude": [0, 31]}')
.option("output", "table")
.load(cube)
)
europe.printSchema()
print(europe.count(), "cells")
Example output
root
|-- time: timestamp (nullable = true)
|-- latitude: double (nullable = true)
|-- longitude: double (nullable = true)
|-- t2m: float (nullable = true)

1302 cells
OptionDefaultDescription
variablesevery gridded variableComma-separated arrays to read.
selwhole axesJSON, dimension to a label: {"latitude": [60, 50], "time": "2023-01-01T05"}. A list of two is an inclusive slice, a scalar is the nearest label, and a list of any other length is those labels.
iselwhole axesJSON, dimension to a position, with the same shapes. A list of two is a start:stop slice, the stop excluded.
outputblocksblocks, tiles or table, as in gbx_zarr_read.
virtualTilestrueoutput=tiles only. false materializes each tile.
dimIndex—dim=index,...: pin dimensions to one position, as netcdf_gbx does.
fanout—output=tiles only. The dimensions to expand into one tile for each position.
bandDim—output=tiles only. One dimension to stack as the bands of each tile.
tileSizeone inner chunkoutput=tiles only. w,h (or one n) in cells.
accessautoauto or fuse: the store is read through its path. direct raises ZarrAccessUnavailable, because direct access needs credentials that Unity Catalog issues to a session, and this reader has none.
engineautoauto (zarr-python), zarr or tensorstore: how a task opens the arrays of a blocks or table read. Not for tiles.

fanout, dimIndex and bandDim follow netcdf_gbx: give any of them and every other non-spatial dimension that is not pinned, selected or stacked is read at index 0. Give none and every selected position of every such dimension is expanded, which is what gbx_zarr_read does. An option that does not go with another (dimIndex and sel on one dimension, fanout on a non-tile read) raises ValueError with a Fix:. The reader defines no push-down filters, so prune with sel, isel and variables.

Sampling at points​

gbx_zarr_sample(spark, points_df, source, ...) reads a store at the points of a DataFrame, with the same geometry inputs as rst_sample: a POINT column of WKB, EWKB, WKT or EWKT. The result has every column of points_df. Next comes a column for each dimension that sel leaves in (time as TIMESTAMP). A column per variable follows, typed as in a table read. A point is repeated for each selected position of those dimensions, so one point's time series is its rows.

from databricks.labs.gbx.zarr import gbx_zarr_sample

points = spark.createDataFrame(
[("paris", "POINT(2.0 48.0)"), ("london", "POINT(0.0 51.0)")], ["name", "geom"]
)
sampled = gbx_zarr_sample(
spark,
points,
table,
variables=["t2m", "msl"],
sel={"time": "2023-01-01T06:00"},
)
sampled.orderBy("name").show(truncate=False)
Example output
+------+---------------+-------------------+-----+--------+
|name |geom |time |t2m |msl |
+------+---------------+-------------------+-----+--------+
|london|POINT(0.0 51.0)|2023-01-01 06:00:00|301.5|100380.0|
|paris |POINT(2.0 48.0)|2023-01-01 06:00:00|302.0|100530.0|
+------+---------------+-------------------+-----+--------+
  • Points are moved to the store's CRS and read at the nearest cell (method="nearest"; "bilinear" is planned and raises ZarrUnsupported). crs is the CRS of points that carry no SRID of their own. An EWKB or EWKT SRID wins, and a point with neither is taken to be in the store's CRS.
  • A point outside the grid (more than half a cell beyond its first or last cell) gives NULL. A missing or empty geometry and a cell equal to the fill value do the same. A geometry that is not a POINT raises ValueError.
  • variables defaults to every gridded variable, which must share a grid.
from databricks.labs.gbx.zarr import gbx_zarr_sample

sel = {"time": "2023-01-01T06:00"}
degrees = spark.createDataFrame([("POINT(2.0 48.0)",)], ["geom"])
metres = spark.createDataFrame([("POINT(222638.98 6106854.83)",)], ["geom"])
same = gbx_zarr_sample(spark, degrees, cube, variables=["t2m"], sel=sel)
moved = gbx_zarr_sample(
spark, metres, cube, variables=["t2m"], sel=sel, crs="EPSG:3857"
)
print(same.first()["t2m"], moved.first()["t2m"])
Example output
302.0 302.0

Up to 100,000 points are collected to the driver. Suppose the chunks they lie in hold no more than 16 MiB decoded and the answer is small. Then the driver reads them and no Spark job runs. Otherwise Spark reads each chunk once for all its points. A chunk with many time steps is split along time into tasks of whole time chunks. A store with few spatial chunks (a county in one chunk, say) therefore still uses every core. Within a chunk, scattered points are read cell by cell. A chunk's points may need a tenth or more of the cells of the box around them. If the box also fits in half of a task's decoded-data budget, the box is read once and every point takes its cells from it. The budget was 32 MiB on Serverless and 768 MiB on Databricks Runtime 19 in these measurements. A chunk whose box is larger is read cell by cell. Take the ERA5 store of the Zarr performance figures: one 90 x 90 chunk of 2-metre temperature, all 744 hours, on Serverless. Points on a quarter of the chunk's cells took 0.44 s this way against 1.7 s cell by cell. Points on every cell took 0.55 s against 6.7 s. At one point in 81 cells (100 points), reading cell by cell was the faster: 0.26 s against 0.40 s. Above 100,000 points points_df is evaluated more than once, up to about three times. The chunks to read are planned from a first pass over the geometry column, so the DataFrame must be deterministic. Write it to a table first, rather than sampling a random or time-dependent query.

Access​

access is how the tasks reach a store's files:

accessMeaning
"auto" (default)Direct where Unity Catalog can issue credentials (dedicated compute), through the Volume mount everywhere else, Serverless included. Never an error.
"fuse"Through the Volume mount. Sends no credentials.
"direct"Byte ranges straight from the Volume's cloud storage, with short-lived credentials the driver asks Unity Catalog for. Several times faster than the mount for many small ranges. Raises ZarrAccessUnavailable where the credentials cannot be had (Serverless, no permission, a path that is not on a Volume).

A plain store is read through its path, so "direct" raises for it: register the store to use direct access. Virtual tiles are opened later, over the mount, so "direct" needs virtual_tiles=False. Direct access sends credentials of at most one hour, scoped to the Volume, to the executors. Under memory pressure Spark may spill task data, those credentials included, to local disk. Turn on local-disk encryption on clusters that use it, or pass access="fuse".

Errors​

A mistake in the call is raised on the driver at once. Examples are a bad argument, an unknown variable, and a version that does not exist. It arrives as a Zarr* error or a ValueError, and its message ends with a Fix: that says how to resolve it:

from databricks.labs.gbx.zarr import gbx_zarr_read

try:
gbx_zarr_read(spark, cube, variables=["temperature"])
except ValueError as err:
print(type(err).__name__ + ":", err)
Example output
ValueError: unknown variable(s) ['temperature']; the store has ['latitude', 'longitude', 'msl', 't2m', 'time', 'u10', 'v10']. Fix: name arrays of the store.

The DataFrame is lazy, so an error raised while Spark runs the read arrives from the action that triggers it, such as collect or write. Such errors include a file that went missing, a chunk that cannot be decoded, and a registered store another tool changed. Each arrives as a pyspark.errors.PythonException. Its message keeps the GeoBrix error name and the Fix: line:

from pyspark.errors import PythonException

from databricks.labs.gbx.zarr import gbx_zarr_read

cell = {"time": "2023-01-01T00:00", "latitude": 0.0, "longitude": 0.0}
cells = gbx_zarr_read(spark, table, variables=["t2m"], sel=cell, output="table")
try:
cells.collect()
except PythonException as err:
# the message keeps the GeoBrix error name and its Fix: line
message = next(ln for ln in str(err).splitlines() if "ZarrStoreDrift: " in ln)
print(type(err).__name__)
print(message.split("Fix: ")[1])
Example output
PythonException
run gbx_zarr_sync(spark, <table>) on the store's manifest table to record the change, then read again

Next Steps​

  • Weather Features with Zarr — an end-to-end example notebook. It registers a weather cube in place and samples it into a features table. It then writes a derived variable back as a new version, maps it as raster tiles, and tags the version for training.
  • Zarr Writer — create, append, update and overwrite a store transactionally.
  • Zarr Function Reference — every function, parameter and error.
  • Zarr Stores — the transactional model and its topic pages: versions, tags, retention, file modes.
  • Virtual Tiles — how a tile that holds no pixels is opened.
  • Readers Overview — all readers, with their tiers and options.