Zarr Reader
Since: zarr_gbx v0.6.0
Read a Zarr store, or a cut of one, into Spark. Three entry points share one reader:
| Entry point | Reads | Use it for |
|---|---|---|
gbx_zarr_read | A plain store (a path), or the manifest table of a managed or registered store | Everything: versions, selections by label or position, and the three outputs (blocks, a table of cells, GeoBrix tiles) |
spark.read.format("zarr_gbx") | A plain store (a path) | A read that is a Spark source, with no call to a function. A DataSource has no session to read a manifest with, so it reads plain stores only |
gbx_zarr_sample | A plain store or a table, at the points of a DataFrame | The time series at a set of points |
All three run at the Python level, so there is no JAR to install. Install the
env6 wheel with its [zarr] extra, on Serverless environment 6 or Databricks Runtime 19.
Zarr Stores describes this. To write a store back see the
Zarr Writer; for every parameter, return value and error see the
Zarr Function Reference. The examples use two names that stand for
what is yours:
# A Zarr store is a path; a managed or registered store is the name of its manifest table.
cube = "/Volumes/main/geobrix_samples/geobrix-examples/global/era5-weather/era5_like_v3.zarr"
table = "main.weather.era5"
The sample cube is the one Zarr Stores generates. It is two days of
made-up hourly weather on a one-degree global grid, time (48) by latitude (181) by
longitude (360), with the variables t2m, u10, v10 and msl. Every code block on this
page is run by the documentation tests.
Choosing the source
gbx_zarr_read(spark, source, ...) takes the source as a string. source_type="auto" (the
default) treats a string that starts with / or has a URI scheme (dbfs:/, file:/, s3://) as
the path of a plain store. It treats anything else as the name of a manifest table. Pass
source_type="path" or "table" to say which.
- A plain store is read as it is on disk, with whatever bytes are there. It has no versions.
- A table is a managed store, or a store registered in place. A read pins one version of the
manifest when it is planned: the latest, or the one named by
version_as_of(a version number or"tag:<name>") ortimestamp_as_of. Pass one of the two, not both. Whatever commits meanwhile, the read sees one consistent snapshot.
from databricks.labs.gbx.zarr import gbx_zarr_read
def hours(**version):
return gbx_zarr_read(
spark,
table,
variables=["t2m"],
sel={"latitude": 48.0, "longitude": 2.0},
output="table",
**version,
).count()
print("latest:", hours(), "hours")
print("version 1:", hours(version_as_of=1), "hours")
print("tag first-day:", hours(version_as_of="tag:first-day"), "hours")
latest: 24 hours
version 1: 12 hours
tag first-day: 12 hours
A version older than the store's retain period raises ZarrVersionUnavailable; see
Lifecycle and retention. A store made
with file_mode="external" holds FILE values in its manifest. It must be read on compute that
supports them. Other compute refuses with a ZarrUnsupported whose Fix: names the options (see
File modes).
Choosing what to read
variables is a name or a list of names. The default is every gridded variable: coordinate
arrays, the arrays that CF coordinates and bounds attributes name, and scalars are left out. An array GeoBrix cannot
decode (a string coordinate, say) does not fail the read. It is left out and named only if you ask
for it, select by its labels, or need its labels for a table.
sel selects by label and isel by position. Each is a dict of dimension to a value:
| Value | sel (labels) | isel (positions) |
|---|---|---|
| A scalar | The exact label, else the nearest label within half the axis step. The dimension stays, with length one | An index. The dimension stays, with length one |
A slice(a, b) | Both ends included, in the order of the axis, so a descending latitude takes slice(60, 50). A time takes ISO strings | start:stop with the end excluded. A step other than 1 is refused |
| A list | Those labels; the output follows the array's order, not the list's | Those positions |
A longitude axis that covers the whole circle is periodic. A slice may cross its seam, so
slice(-10, 10) on a 0 to 360 axis is the labels from 350 round to 10. A label that is not on the
axis raises ValueError when it is a scalar, and selects nothing when it is a slice. Variables
that lack a selected dimension are read whole along the others.
from databricks.labs.gbx.zarr import gbx_zarr_read
by_label = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={
"time": slice("2023-01-01T00:00", "2023-01-01T02:00"),
"latitude": slice(60, 50), # latitude runs north to south
"longitude": slice(0, 10), # both ends are included
},
output="table",
)
by_position = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
isel={
"time": slice(0, 3), # positions: the end is not included
"latitude": slice(30, 41),
"longitude": slice(0, 11),
},
output="table",
)
print("by label:", by_label.count(), "cells")
print("by position:", by_position.count(), "cells")
by label: 363 cells
by position: 363 cells
Output: blocks
output="blocks" (the default) returns one row per variable and touched chunk, holding that
chunk's selected cells. It is the form to write back with gbx_zarr_write, and
the form that keeps every dtype, including complex ones.
| Column | Type | Holds |
|---|---|---|
variable | STRING | The array's name |
dims | ARRAY<STRING> | Its dimension names, in order |
start, stop | ARRAY<BIGINT> | Where the block sits in the array, per dimension: [start, stop) in cells |
coord_start, coord_stop | ARRAY<STRING> | The first and last label of each dimension |
dtype | STRING | The NumPy dtype of data, such as <f4 |
shape | ARRAY<BIGINT> | The block's shape |
data | BINARY | The cells, C-order bytes |
fill_value | STRING | What a never-written cell holds |
attrs | STRING | The array's attributes, as JSON |
block_to_numpy(row) turns a row into a NumPy array of the block's shape and dtype (a writable
copy), and block_to_xarray(row) into a named xarray.DataArray. A block carries the first and last
label of each dimension rather than every label, so the DataArray has no coordinates attached.
from databricks.labs.gbx.zarr import block_to_numpy, gbx_zarr_read
blocks = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T02:00")},
)
blocks.orderBy("start").select("variable", "dims", "start", "stop", "shape").show(
truncate=False
)
cells = block_to_numpy(blocks.orderBy("start").first()) # a numpy array
print(cells.shape, cells.dtype)
+--------+---------------------------+------------+-------------+------------+
|variable|dims |start |stop |shape |
+--------+---------------------------+------------+-------------+------------+
|t2m |[time, latitude, longitude]|[0, 0, 0] |[3, 91, 180] |[3, 91, 180]|
|t2m |[time, latitude, longitude]|[0, 0, 180] |[3, 91, 360] |[3, 91, 180]|
|t2m |[time, latitude, longitude]|[0, 91, 0] |[3, 181, 180]|[3, 90, 180]|
|t2m |[time, latitude, longitude]|[0, 91, 180]|[3, 181, 360]|[3, 90, 180]|
+--------+---------------------------+------------+-------------+------------+
(3, 91, 180) float32
decode=True (the default) applies the CF conventions the array declares: scale and offset,
missing values to NaN, and times to datetime64[ns]. The attributes it used are dropped from the
block's attrs. decode=False returns the stored values. A time axis that is not in the standard
calendars, or outside the years 1678 to 2262, raises ZarrUnsupported; read it with decode=False.
Output: a table of cells
output="table" returns one row per selected cell, with a column per dimension and a column per
variable. A dimension's column holds its labels, or the position as BIGINT when the dimension
has no coordinate array. The variables must share their dimensions and shape. Row order is not
defined.
from databricks.labs.gbx.zarr import gbx_zarr_read
europe = gbx_zarr_read(
spark,
cube,
variables=["t2m", "msl"],
sel={
"time": slice("2023-01-01T00:00", "2023-01-01T01:00"),
"latitude": slice(60, 40),
"longitude": slice(0, 10),
},
output="table",
)
europe.printSchema()
europe.orderBy("time", "latitude", "longitude").show(3)
root
|-- time: timestamp (nullable = true)
|-- latitude: double (nullable = true)
|-- longitude: double (nullable = true)
|-- t2m: float (nullable = true)
|-- msl: float (nullable = true)
+-------------------+--------+---------+-----+--------+
| time|latitude|longitude| t2m| msl|
+-------------------+--------+---------+-----+--------+
|2023-01-01 00:00:00| 40.0| 0.0|299.5|100580.0|
|2023-01-01 00:00:00| 40.0| 1.0|299.5|100570.0|
|2023-01-01 00:00:00| 40.0| 2.0|299.5|100580.0|
+-------------------+--------+---------+-----+--------+
only showing top 3 rows
Column types
A column keeps the type of the values it was made from, so a table never widens a count to a double:
| Stored | Column |
|---|---|
int8 | TINYINT |
int16, uint8 | SMALLINT |
int32, uint16 | INT |
int64, uint32 | BIGINT |
uint64 | DECIMAL(20,0) |
bool | BOOLEAN |
float16, float32 | FLOAT |
float64 | DOUBLE |
| CF times | TIMESTAMP, in UTC, truncated to the microsecond |
A packed array (scale_factor, add_offset) | FLOAT or DOUBLE, since it decodes to floating point |
A time column is a UTC instant. It does not move with the session time zone, and a time finer than
a microsecond is truncated. Complex arrays raise ZarrUnsupported as a table or as tiles: read them as blocks.
Missing data is NULL
A cell equal to the array's _FillValue or missing_value is NULL. The comparison is made on the
stored values, before any scaling or cast, so an integer array stays an integer array:
from databricks.labs.gbx.zarr import gbx_zarr_read
rain = gbx_zarr_read(spark, gauges, variables=["rain"], output="table")
rain.printSchema()
rain.orderBy("day", "station").show()
root
|-- day: timestamp (nullable = true)
|-- station: long (nullable = true)
|-- rain: short (nullable = true)
+-------------------+-------+----+
| day|station|rain|
+-------------------+-------+----+
|2023-06-01 00:00:00| 101| 0|
|2023-06-01 00:00:00| 102| 12|
|2023-06-01 00:00:00| 103|NULL|
|2023-06-02 00:00:00| 101| 3|
|2023-06-02 00:00:00| 102|NULL|
|2023-06-02 00:00:00| 103| 8|
|2023-06-03 00:00:00| 101| 1|
|2023-06-03 00:00:00| 102| 2|
|2023-06-03 00:00:00| 103| 0|
+-------------------+-------+----+
With decode=False the table holds the stored values and nothing is NULL. A few details
decide which cells count as missing:
- A NaN fill value gives
NULLcells. Where the fill is NaN, every NaN cell is missing. A NaN that is data beside a numeric fill stays NaN. - The array's
fill_valueis only a marker where xarray's rule says so. Zarr v2 has no_FillValueattribute, souse_zarr_fill_value_as_mask=None(the default, xarray's own) masks on the arrayfill_valuefor Zarr v2 and not for Zarr v3. An integer v2 array whosefill_valueis0therefore reads its zeros asNULLby default.use_zarr_fill_value_as_mask=Falsekeeps them, andTruemasks onfill_valuein either version. Thezarr_gbxDataSource has no option for this and uses the default. - An integer array read as booleans (a
dtypeattribute of"bool") is masked only on an explicit_FillValueormissing_value, never on the array'sfill_value. Zarr's default of0is a storedFalse, not a missing cell.
from databricks.labs.gbx.zarr import gbx_zarr_read
for flag in (None, False):
cells = gbx_zarr_read(
spark,
counts,
variables=["counts"],
output="table",
use_zarr_fill_value_as_mask=flag,
)
print(flag, "->", cells.where("counts IS NULL").count(), "NULL cells")
None -> 2 NULL cells
False -> 0 NULL cells
GeoBrix follows the Zarr store's own semantics, which differ from xarray's in three places.
- Masked integers stay integers. xarray decodes a masked integer array to floating point so
that the missing cells can be NaN. A table keeps the integer type and writes
NULL. - The mask flag.
xarray.open_zarrignoresuse_zarr_fill_value_as_maskfor the arrays it opens;gbx_zarr_readhonours it, asZarrStoredoes. - Booleans with a stored fill. For an integer array read as booleans that has an explicit fill, a
cell equal to the fill is
NULLin a table. Blocks follow xarray, which casts the masked cell's NaN toTrue. Neither marks the array's defaultfill_valueof0as missing.
Output: GeoBrix tiles
output="tiles" returns the (source, tile) DataFrame the other raster readers return, so every
rst_* function applies to Zarr data without converting the cube to GeoTIFF. Each selected 2-D
plane of each variable (the plane of its latitude and longitude, or y and x) is cut into
windows, one inner chunk each. Every selected position along the other dimensions makes its own
tile. The tile's metadata says which: zarr_array, sliceDims (such as time=0) and
sliceCoord_<dim> (the label), as the NetCDF reader writes them.
tile_size=(width, height), in cells, overrides the window. A window never crosses the longitude seam. The largest tile a task may hold is 64 MiB decoded on Serverless and 256 MiB on classic clusters.band_dim="time"stacks one dimension as the bands of one tile instead of making a tile for each position.- Tiles are north-up: when latitude ascends, the rows are flipped. The CRS is the array's
grid_mapping, orEPSG:4326for a plain latitude and longitude grid, and NoData is NaN for floating-point cells. - A 0 to 360 longitude axis gives
EPSG:4326tiles with x above 180. A cube whose longitude runs from 0 to 359 has tiles that reach x = 359.5. GeoBrix does not shift them into -180..180. Keep that in mind when you combine the tiles with rasters in the usual -180..180 convention. - The coordinates must form a regular grid over the selected cells (a constant step; what lies
outside the selection does not matter). An irregular or curvilinear grid raises
ZarrUnsupported: read it as blocks or a table. A variable with no such grid is left out with a warning, unless you named it invariables, which raises.
from pyspark.sql import functions as F
from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.zarr import gbx_zarr_read
tiles = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={"time": "2023-01-01T00:00"},
output="tiles",
)
tiles.select(
F.col("tile.metadata")["sliceDims"].alias("slice"),
"tile.window.col_off",
"tile.window.row_off",
"tile.window.width",
"tile.window.height",
rx.rst_crs("tile").alias("crs"), # answered from the header: no chunk is read
).orderBy("row_off", "col_off").show(truncate=False)
+------+-------+-------+-----+------+---------+
|slice |col_off|row_off|width|height|crs |
+------+-------+-------+-----+------+---------+
|time=0|0 |0 |180 |91 |EPSG:4326|
|time=0|180 |0 |180 |91 |EPSG:4326|
|time=0|0 |91 |180 |90 |EPSG:4326|
|time=0|180 |91 |180 |90 |EPSG:4326|
+------+-------+-------+-----+------+---------+
Virtual tiles and the rst_* functions
With virtual_tiles=True (the default) a tile holds no pixels. It names its window and the storage
objects behind it, and reads them when a function opens it. A table of tiles can therefore be
saved and read back by tasks that have no session. This holds as long as the store stays on a Volume and,
for a table, the pinned version stays readable. The Python rst_* functions open a tile on
demand and read only the chunks its window touches. A header function such as rst_crs or
rst_width reads no chunk at all.
from pyspark.sql import functions as F
from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.zarr import gbx_zarr_read
tiles = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={"time": "2023-01-01T00:00"},
output="tiles",
)
stats = tiles.select(
rx.rst_width("tile").alias("width"),
rx.rst_height("tile").alias("height"),
F.round(rx.rst_avg("tile").getItem(0), 1).alias("mean"),
)
stats.orderBy("height", "mean").show()
+-----+------+-----+
|width|height| mean|
+-----+------+-----+
| 180| 90|294.9|
| 180| 90|300.0|
| 180| 91|295.0|
| 180| 91|300.1|
+-----+------+-----+
band_dim stacks a dimension as bands, so six hours become the six bands of one tile:
from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.zarr import gbx_zarr_read
tiles = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={"time": slice("2023-01-01T00:00", "2023-01-01T05:00")},
band_dim="time",
output="tiles",
)
tiles.select(rx.rst_numbands("tile").alias("bands")).distinct().show()
+-----+
|bands|
+-----+
| 6|
+-----+
Materialize for the JVM level
The JVM level reads materialized tiles only. In SQL, GeoBrix sends a virtual tile to the Python
level, so a gbx_rst_* call takes one on every compute (see
Virtual Tiles). virtual_tiles=False reads
each tile now and holds its pixels as GeoTIFF bytes, which both levels accept. It costs the read up
front and the memory of the pixels, so keep the windows small.
from pyspark.sql import functions as F
from databricks.labs.gbx.zarr import gbx_zarr_read
tiles = gbx_zarr_read(
spark,
cube,
variables=["t2m"],
sel={"time": "2023-01-01T00:00"},
output="tiles",
virtual_tiles=False,
)
tiles.select((F.length("tile.raster") > 0).alias("holds_pixels")).distinct().show()
+------------+
|holds_pixels|
+------------+
| true|
+------------+
Writing tiles back
A tile cut from Zarr still carries where it came from, so it can be written back to the array it
was cut from. A pixel-producing rst_* function (a map algebra, a clip, a resample) returns a fresh
tile without that provenance. Put the provenance back with withField("metadata", ...), or give
gbx_zarr_write dim_cols and variables= to say where the tile goes.
The Zarr Writer shows both.
How a read is split into tasks
A blocks or table read packs whole chunks into tasks of about 16 MiB decoded. A read too small for
that is spread over the cores of the session instead, down to 4 MiB per task. A 100 MB read therefore does not
run as one task. A task never holds more than the compute's decoded-data limit (64 MiB on Serverless,
1536 MiB on a classic cluster), whatever the target. The cores are read from the session's configuration, so
the plan is the same on Serverless, where there is no SparkContext. On a Databricks cluster that sets no
parallelism or executor configuration, the cores are the cluster's worker count times the cores of a worker. The
cores of a worker are taken from the driver when both use the same node type. A cluster of mixed node types
is planned as one slot. A chunk counts only the cells inside the array: a 24-hour chunk of a 1-hour array counts
one hour.
A plain store on a Volume is decoded by a thread pool larger than the core count. This is because each chunk read
waits on the mount. Its size is eight threads per core, at most 64, lowered when the chunks the threads hold would
not fit in the task's decoded-data limit. Set RAYON_NUM_THREADS before the first read of a process to choose it
yourself. A store registered in a table reads its chunks through the manifest, which overlaps its reads already.
The zarr_gbx DataSource
spark.read.format("zarr_gbx") reads a plain store with no table, no session call and no JAR.
Register the *_gbx DataSources once per session. Options are strings, like the other
*_gbx readers.
from databricks.labs import gbx
gbx.register(spark)
europe = (
spark.read.format("zarr_gbx")
.option("variables", "t2m")
.option("sel", '{"time": ["2023-01-01T00:00", "2023-01-01T01:00"]}')
.option("isel", '{"latitude": [30, 51], "longitude": [0, 31]}')
.option("output", "table")
.load(cube)
)
europe.printSchema()
print(europe.count(), "cells")
root
|-- time: timestamp (nullable = true)
|-- latitude: double (nullable = true)
|-- longitude: double (nullable = true)
|-- t2m: float (nullable = true)
1302 cells
| Option | Default | Description |
|---|---|---|
variables | every gridded variable | Comma-separated arrays to read. |
sel | whole axes | JSON, dimension to a label: {"latitude": [60, 50], "time": "2023-01-01T05"}. A list of two is an inclusive slice, a scalar is the nearest label, and a list of any other length is those labels. |
isel | whole axes | JSON, dimension to a position, with the same shapes. A list of two is a start:stop slice, the stop excluded. |
output | blocks | blocks, tiles or table, as in gbx_zarr_read. |
virtualTiles | true | output=tiles only. false materializes each tile. |
dimIndex | — | dim=index,...: pin dimensions to one position, as netcdf_gbx does. |
fanout | — | output=tiles only. The dimensions to expand into one tile for each position. |
bandDim | — | output=tiles only. One dimension to stack as the bands of each tile. |
tileSize | one inner chunk | output=tiles only. w,h (or one n) in cells. |
access | auto | auto or fuse: the store is read through its path. direct raises ZarrAccessUnavailable, because direct access needs credentials that Unity Catalog issues to a session, and this reader has none. |
engine | auto | auto (zarr-python), zarr or tensorstore: how a task opens the arrays of a blocks or table read. Not for tiles. |
fanout, dimIndex and bandDim follow netcdf_gbx: give any of them and every other
non-spatial dimension that is not pinned, selected or stacked is read at index 0. Give none and
every selected position of every such dimension is expanded, which is what gbx_zarr_read does.
An option that does not go with another (dimIndex and sel on one dimension, fanout on a
non-tile read) raises ValueError with a Fix:. The reader defines no push-down filters, so
prune with sel, isel and variables.
Sampling at points
gbx_zarr_sample(spark, points_df, source, ...) reads a store at the points of a DataFrame, with
the same geometry inputs as rst_sample: a POINT column of WKB, EWKB, WKT or EWKT. The result has
every column of points_df. Next comes a column for each dimension that sel leaves in (time as
TIMESTAMP). A column per variable follows, typed as in a table read. A point is repeated for each
selected position of those dimensions, so one point's time series is its rows.
from databricks.labs.gbx.zarr import gbx_zarr_sample
points = spark.createDataFrame(
[("paris", "POINT(2.0 48.0)"), ("london", "POINT(0.0 51.0)")], ["name", "geom"]
)
sampled = gbx_zarr_sample(
spark,
points,
table,
variables=["t2m", "msl"],
sel={"time": "2023-01-01T06:00"},
)
sampled.orderBy("name").show(truncate=False)
+------+---------------+-------------------+-----+--------+
|name |geom |time |t2m |msl |
+------+---------------+-------------------+-----+--------+
|london|POINT(0.0 51.0)|2023-01-01 06:00:00|301.5|100380.0|
|paris |POINT(2.0 48.0)|2023-01-01 06:00:00|302.0|100530.0|
+------+---------------+-------------------+-----+--------+
- Points are moved to the store's CRS and read at the nearest cell (
method="nearest";"bilinear"is planned and raisesZarrUnsupported).crsis the CRS of points that carry no SRID of their own. An EWKB or EWKT SRID wins, and a point with neither is taken to be in the store's CRS. - A point outside the grid (more than half a cell beyond its first or last cell) gives
NULL. A missing or empty geometry and a cell equal to the fill value do the same. A geometry that is not a POINT raisesValueError. variablesdefaults to every gridded variable, which must share a grid.
from databricks.labs.gbx.zarr import gbx_zarr_sample
sel = {"time": "2023-01-01T06:00"}
degrees = spark.createDataFrame([("POINT(2.0 48.0)",)], ["geom"])
metres = spark.createDataFrame([("POINT(222638.98 6106854.83)",)], ["geom"])
same = gbx_zarr_sample(spark, degrees, cube, variables=["t2m"], sel=sel)
moved = gbx_zarr_sample(
spark, metres, cube, variables=["t2m"], sel=sel, crs="EPSG:3857"
)
print(same.first()["t2m"], moved.first()["t2m"])
302.0 302.0
Up to 100,000 points are collected to the driver. Suppose the chunks they lie in hold no more than 16 MiB
decoded and the answer is small. Then the driver reads them and no Spark job runs. Otherwise Spark reads each
chunk once for all its points. A chunk with many time steps is split along time into tasks of whole time chunks.
A store with few spatial chunks (a county in one chunk, say) therefore still uses every core. Within a chunk, scattered points are read cell by cell. A chunk's points may need a tenth or
more of the cells of the box around them. If the box also fits in half of a task's decoded-data budget,
the box is read once and every point takes its cells from it. The budget was 32 MiB on Serverless and
768 MiB on Databricks Runtime 19 in these measurements. A chunk whose box is larger is read cell by cell. Take the ERA5 store
of the Zarr performance figures: one 90 x 90 chunk of 2-metre
temperature, all 744 hours, on Serverless. Points on a quarter of the chunk's cells took 0.44 s this way
against 1.7 s cell by cell. Points on every cell took 0.55 s against 6.7 s. At one point in 81 cells
(100 points), reading cell by cell was the faster: 0.26 s against 0.40 s.
Above 100,000 points points_df is evaluated more than once, up to about three times.
The chunks to read are planned from a first pass over the geometry column, so the DataFrame must be
deterministic. Write it to a table first, rather than sampling a random or time-dependent query.
Access
access is how the tasks reach a store's files:
access | Meaning |
|---|---|
"auto" (default) | Direct where Unity Catalog can issue credentials (dedicated compute), through the Volume mount everywhere else, Serverless included. Never an error. |
"fuse" | Through the Volume mount. Sends no credentials. |
"direct" | Byte ranges straight from the Volume's cloud storage, with short-lived credentials the driver asks Unity Catalog for. Several times faster than the mount for many small ranges. Raises ZarrAccessUnavailable where the credentials cannot be had (Serverless, no permission, a path that is not on a Volume). |
A plain store is read through its path, so "direct" raises for it: register the store to use direct
access. Virtual tiles are opened later, over the mount, so "direct" needs virtual_tiles=False.
Direct access sends credentials of at most one hour, scoped to the Volume, to the executors. Under
memory pressure Spark may spill task data, those credentials included, to local disk. Turn on local-disk
encryption on clusters that use it, or pass access="fuse".
Errors
A mistake in the call is raised on the driver at once. Examples are a bad argument, an unknown variable, and a version that does not exist.
It arrives as a Zarr* error or a ValueError, and its message ends with a Fix: that
says how to resolve it:
from databricks.labs.gbx.zarr import gbx_zarr_read
try:
gbx_zarr_read(spark, cube, variables=["temperature"])
except ValueError as err:
print(type(err).__name__ + ":", err)
ValueError: unknown variable(s) ['temperature']; the store has ['latitude', 'longitude', 'msl', 't2m', 'time', 'u10', 'v10']. Fix: name arrays of the store.
The DataFrame is lazy, so an error raised while Spark runs the read arrives from the action that triggers
it, such as collect or write. Such errors include a file that went missing, a chunk
that cannot be decoded, and a registered store another tool changed. Each arrives as a pyspark.errors.PythonException. Its message keeps the
GeoBrix error name and the Fix: line:
from pyspark.errors import PythonException
from databricks.labs.gbx.zarr import gbx_zarr_read
cell = {"time": "2023-01-01T00:00", "latitude": 0.0, "longitude": 0.0}
cells = gbx_zarr_read(spark, table, variables=["t2m"], sel=cell, output="table")
try:
cells.collect()
except PythonException as err:
# the message keeps the GeoBrix error name and its Fix: line
message = next(ln for ln in str(err).splitlines() if "ZarrStoreDrift: " in ln)
print(type(err).__name__)
print(message.split("Fix: ")[1])
PythonException
run gbx_zarr_sync(spark, <table>) on the store's manifest table to record the change, then read again
Next Steps
- Weather Features with Zarr — an end-to-end example notebook. It registers a weather cube in place and samples it into a features table. It then writes a derived variable back as a new version, maps it as raster tiles, and tags the version for training.
- Zarr Writer — create, append, update and overwrite a store transactionally.
- Zarr Function Reference — every function, parameter and error.
- Zarr Stores — the transactional model and its topic pages: versions, tags, retention, file modes.
- Virtual Tiles — how a tile that holds no pixels is opened.
- Readers Overview — all readers, with their tiers and options.