Zarr: Performance and Limits
Since: v0.6.0 · (Python level; no JAR needed)
Use this page to size a Zarr workload and to check what is not supported. The performance section shows what reads and writes cost on Databricks and why. The limits section lists the runtimes, data types and layouts GeoBrix supports, with the fix for each refusal.
Performance
What a store costs to read and write depends on its layout and on the compute. The figures below were measured on Databricks to show the shape of the trade-offs. They are not a guarantee.
To time reads and writes on your own data and compute, run the Zarr at scale notebook. It runs the same steps on stores from 10 MB to 10 GB.
In short
- Reading a box over many hours is fast. A read fetches only the chunks the box touches, in a few large reads. A GeoTIFF needs one file per hour.
- Write the layout for how you read. A store that is read mostly as maps should be written with
chunks="maps". - Writing is slower than plain
xarray.to_zarr, because the commit adds versions, all-or-nothing writes and safe concurrent writers. - Write a store in a few large writes, not many small ones. Every write pays the commit.
Reads
| Workload | Read with | Runtime 19 | Serverless |
|---|---|---|---|
| One hour of the whole globe | gbx_zarr_read | 3.2 s | 3.0 s |
GeoTIFF: raster_gbx and rst_avg | 1.9 s | 3.3 s | |
| 1,000 points, every hour (744,000 values) | gbx_zarr_sample | 24 s | 16.5 s |
| 30 days over Europe (35-72 N, 10 W-40 E) | gbx_zarr_read | 4.8 s | 4.4 s |
GeoTIFF: 720 windows, raster_gbx and rst_avg | 232 s | 83 s |
- A single map is quick either way. This store keeps 24 hours in each inner chunk, so one hour reads
24 hours of chunks. A store that is read mostly as maps should be written with
chunks="maps". - A time series at points reads each chunk once for all the points in it, whatever the number of hours.
- A box over many hours reads only the chunks the box touches, in a few large reads. The GeoTIFFs need one file per hour.
Writes
| Write | With | Runtime 19, through the Volume | Runtime 19, direct to storage | Serverless |
|---|---|---|---|---|
| Create a 1 GB cube (240 x 721 x 1440, float32, from 10 tasks) | gbx_zarr_write | 19 s | 22 s | 20 s |
| of which writing the data files | 8.4 s (119 MB/s) | 12.4 s (80 MB/s) | 8.1 s (122 MB/s) | |
plain xarray.to_zarr, no transaction | 5.4 s (184 MB/s) | 6.1 s (163 MB/s) | ||
| Eight appends of one day each (100 MB) | gbx_zarr_write, all at once | 166 s | 132 s | |
gbx_zarr_write, one after another | 147 s | 150 s |
- Blocks that are whole shards are written where they are made. Each block of this cube is one shard (24 hours of the whole globe). The task that makes a block encodes it and writes its data file, and nothing is shuffled. Blocks that cut across chunks or shards are regrouped first, which costs more; see the Zarr Writer.
- A create has a fixed cost. It claims its root, creates the manifest table and commits. The checks before them take another 0.1-0.6 s. A write that is not a create pays the commit part of it. Write a store in a few large writes rather than many small ones.
- A plain
xarray.to_zarris faster and gives none of the guarantees. It has no versions, no all-or-nothing commit and no safe concurrent writers. - An input that is expensive to compute is evaluated twice by a create or an append (once for its positions). Write such an input to a table first; see the Zarr Writer.
- Concurrent appenders land one per round. Each rebases on the others after a jittered wait that doubles with every lost round (up to 16 s). The last of eight needed seven or eight attempts.
Where the time of a create goes
The table shows the median of each step over the warm runs. A create took 19.5 s on Serverless and 19.3 s on Runtime 19 through the Volume.
| Step | Serverless | Runtime 19, through the Volume |
|---|---|---|
| First pass over the block positions | 2.5 s | 1.7 s |
| Claiming the root in the ownership ledger | 2.1 s | 2.3 s |
| Creating the manifest table | 1.8 s | 1.8 s |
| Writing the data files | 8.1 s | 8.4 s |
| The commit (on Serverless, 3.0 s of it the MERGE into the manifest) | 4.6 s | 4.9 s |
Details behind the table:
- Fixed cost of a create. In the warm runs its median was 9.0 s through the Volume, 8.0 s direct to storage and 8.4 s on Serverless. The range was 7.4-10.5 s. The first create on new compute ran cold and took 20.8-38.3 s. The direct leg's first create (the cluster's second) took 21.0 s.
- Commit through the Volume. The commit's coordinate files cost 0.5-0.8 s through the Volume against 0.1-0.2 s direct.
- An expensive input. The cube is computed in the tasks. Its first pass took 1.5-8.1 s in the warm runs and 14-31 s in the first create on new compute.
- Direct to storage. Writing the data files varied most from run to run on one cluster, from 7.2 s to 25.5 s. Measured again on a second cluster, it took a median 6.7 s (149 MB/s), from 5.3 s to 29.7 s. That run sat beside plain object-store puts of the same size on the same nodes. The slow runs spent their time in the executors' JVM garbage collection. Spark handed each 100 MB block from the step that computed it to the step that writes it. The writes themselves took 0.6-2.6 s each. The plain puts, which never cross that step, took 0.5-4.9 s.
- Eight concurrent appenders. On Serverless, eight at once took 132 s against 150 s one after another. On Runtime 19 with two 4-core workers they took 166 s against 147 s. The eight shared eight cores as well as waiting between attempts.
How this was measured
- Data. The ERA5 reanalysis for January 2023: 2-metre temperature, 744 hourly steps on the 0.25-degree global grid (721 x 1440), float32. As Zarr, it is a v3 store. Its inner chunks are 24 x 90 x 90 cells, in shards of 168 x 360 x 360, compressed with zstd. For comparison, the same values as one tiled GeoTIFF per hour (512 x 512 tiles, ZSTD), read with the Raster Reader.
- Compute. The reads ran on Databricks Runtime 19 with one 4-core worker (
m5d.xlarge). The writes ran on Databricks Runtime 19 with two 4-core workers (m6gd.xlarge, on demand). Both used dedicated access. Both also ran on Serverless environment 6. - Method. Each figure is the median of three runs, the first of the three cold. The exception is the 1 GB create. Its figures are the median of five warm runs after a cold one. The two Runtime 19 legs (through the Volume, direct to storage) took turns.
Limits
Runtimes and data
- Runtimes. Databricks Runtime 19 and Serverless environment 6 (Python 3.12), at the Python level only. The JVM level reads Zarr data only as materialized GeoBrix tiles.
- Data types. Numeric types and CF times. Strings and objects are not supported. An array GeoBrix cannot decode is left out of reads and registered without being readable.
- Complex arrays. A complex array can only be read as blocks. Tables have no complex column, and tiles do not hold complex numbers yet.
- Tiles.
output="tiles"needs the coordinates to form a regular grid. A tile of a registered store does not check its files for changes. A file changed in place reads as its new bytes, or fails. Materialize the tiles you keep.
Layout and codecs
- Chunk grid. Only a regular chunk grid is supported. An array with a rectilinear grid is left out of reads and registered without being readable.
- Inner chunks per shard. A shard that
gbx_zarr_writecreates holds at most 4,096 inner chunks.- Every read of a shard fetches or builds its index, 16 bytes per inner chunk. A shard of many tiny chunks makes every read slow. On a Volume, reading one chunk of a shard of 93,312 chunks took a median of 0.87-1.41 s. At 4,096 chunks it took 0.10-0.15 s (ten runs on two runtimes).
- The computed layout stays within the limit. A
shards=that goes beyond it raisesValueErrorwith aFix:. - A registered store's shards are read as they are.
- Codecs. The codecs that zarr-python, numcodecs and zarrs can decode. Writes use
zstdandcrc32cunless you passcodecs.- A Zarr v3 array that has any codec beside
sharding_indexedin its own codec list is not supported. A transpose before it or a compressor after it are examples. - Registering such a store records the array as one GeoBrix cannot decode. A default read leaves it out
with a warning. Naming it in
variablesraisesZarrUnsupported. - To fix it, rewrite the array so that
sharding_indexedis its only outer codec, with compression and other codecs inside it. zarr-python does not write that layout.
- A Zarr v3 array that has any codec beside
- Reading with GDAL. A GDAL before 3.13 cannot read sharded Zarr v3 stores, or ones that use the
crc32ccodec, whichgbx_zarr_writewrites by default; see the Raster Reader. - File modes.
externalis a preview that needs a workspace with the File type preview and compute that supportsFILEvalues.managedis not available (see File modes).
Writers and storage access
- Outside writers. GeoBrix detects writes that other tools make to a registered store. It cannot make them transactional. On a Volume the identity of an object is its modification time and size, which is weaker than a content hash. A publish is not atomic for plain readers.
- Branches. Registrations of one store do not merge.
- Version history is bounded by
retain. Tag a version to keep it indefinitely. - Many simultaneous appenders. A rebase retries up to 20 times, and the last of N simultaneous
appenders needs about N attempts. A burst of more than about twenty writers appending to one array at
the same instant can exhaust that. The last of them fails with
ZarrWriteConflictand aFix:. Run it again. - Direct storage access.
access="auto"reads and writes cloud storage directly where Unity Catalog can issue temporary credentials (dedicated compute). Elsewhere, including Serverless, it goes through the Volume mount.- Direct access sends short-lived, Volume-scoped credentials (at most one hour) to the executors. Under memory pressure Spark may spill task data, those credentials included, to local disk.
- Turn on local-disk encryption on clusters that use direct access, or pass
access="fuse", which sends no credentials. - Direct access has been measured on AWS only. Azure and Google Cloud storage are not measured yet, and
access="fuse"reads and writes through the Volume mount there.
- Incomplete uploads. A pack upload that is refused or fails part way can leave an incomplete
multipart upload in the bucket. GeoBrix does not clean it up. Add a bucket lifecycle rule that aborts
incomplete multipart uploads (
AbortIncompleteMultipartUpload) to the bucket of the Volume.
Next Steps
- Lifecycle and retention:
gbx_zarr_optimizebrings read times back after many small writes. - Zarr Writer:
chunkspresets and shaping blocks to the store's grid. - Zarr at scale: a notebook that times five reads and a table to Zarr write from 10 MB to 10 GB, with reference timings.
- Limitations: the limits that apply across GeoBrix.