Skip to main content

Zarr: Performance and Limits

Since: v0.6.0 · (Python level; no JAR needed)

Use this page to size a Zarr workload and to check what is not supported. The performance section shows what reads and writes cost on Databricks and why. The limits section lists the runtimes, data types and layouts GeoBrix supports, with the fix for each refusal.

Performance​

What a store costs to read and write depends on its layout and on the compute. The figures below were measured on Databricks to show the shape of the trade-offs. They are not a guarantee.

To time reads and writes on your own data and compute, run the Zarr at scale notebook. It runs the same steps on stores from 10 MB to 10 GB.

In short

  • Reading a box over many hours is fast. A read fetches only the chunks the box touches, in a few large reads. A GeoTIFF needs one file per hour.
  • Write the layout for how you read. A store that is read mostly as maps should be written with chunks="maps".
  • Writing is slower than plain xarray.to_zarr, because the commit adds versions, all-or-nothing writes and safe concurrent writers.
  • Write a store in a few large writes, not many small ones. Every write pays the commit.

Reads​

WorkloadRead withRuntime 19Serverless
One hour of the whole globegbx_zarr_read3.2 s3.0 s
GeoTIFF: raster_gbx and rst_avg1.9 s3.3 s
1,000 points, every hour (744,000 values)gbx_zarr_sample24 s16.5 s
30 days over Europe (35-72 N, 10 W-40 E)gbx_zarr_read4.8 s4.4 s
GeoTIFF: 720 windows, raster_gbx and rst_avg232 s83 s
  • A single map is quick either way. This store keeps 24 hours in each inner chunk, so one hour reads 24 hours of chunks. A store that is read mostly as maps should be written with chunks="maps".
  • A time series at points reads each chunk once for all the points in it, whatever the number of hours.
  • A box over many hours reads only the chunks the box touches, in a few large reads. The GeoTIFFs need one file per hour.

Writes​

WriteWithRuntime 19, through the VolumeRuntime 19, direct to storageServerless
Create a 1 GB cube (240 x 721 x 1440, float32, from 10 tasks)gbx_zarr_write19 s22 s20 s
of which writing the data files8.4 s (119 MB/s)12.4 s (80 MB/s)8.1 s (122 MB/s)
plain xarray.to_zarr, no transaction5.4 s (184 MB/s)6.1 s (163 MB/s)
Eight appends of one day each (100 MB)gbx_zarr_write, all at once166 s132 s
gbx_zarr_write, one after another147 s150 s
  • Blocks that are whole shards are written where they are made. Each block of this cube is one shard (24 hours of the whole globe). The task that makes a block encodes it and writes its data file, and nothing is shuffled. Blocks that cut across chunks or shards are regrouped first, which costs more; see the Zarr Writer.
  • A create has a fixed cost. It claims its root, creates the manifest table and commits. The checks before them take another 0.1-0.6 s. A write that is not a create pays the commit part of it. Write a store in a few large writes rather than many small ones.
  • A plain xarray.to_zarr is faster and gives none of the guarantees. It has no versions, no all-or-nothing commit and no safe concurrent writers.
  • An input that is expensive to compute is evaluated twice by a create or an append (once for its positions). Write such an input to a table first; see the Zarr Writer.
  • Concurrent appenders land one per round. Each rebases on the others after a jittered wait that doubles with every lost round (up to 16 s). The last of eight needed seven or eight attempts.

Where the time of a create goes​

The table shows the median of each step over the warm runs. A create took 19.5 s on Serverless and 19.3 s on Runtime 19 through the Volume.

StepServerlessRuntime 19, through the Volume
First pass over the block positions2.5 s1.7 s
Claiming the root in the ownership ledger2.1 s2.3 s
Creating the manifest table1.8 s1.8 s
Writing the data files8.1 s8.4 s
The commit (on Serverless, 3.0 s of it the MERGE into the manifest)4.6 s4.9 s

Details behind the table:

  • Fixed cost of a create. In the warm runs its median was 9.0 s through the Volume, 8.0 s direct to storage and 8.4 s on Serverless. The range was 7.4-10.5 s. The first create on new compute ran cold and took 20.8-38.3 s. The direct leg's first create (the cluster's second) took 21.0 s.
  • Commit through the Volume. The commit's coordinate files cost 0.5-0.8 s through the Volume against 0.1-0.2 s direct.
  • An expensive input. The cube is computed in the tasks. Its first pass took 1.5-8.1 s in the warm runs and 14-31 s in the first create on new compute.
  • Direct to storage. Writing the data files varied most from run to run on one cluster, from 7.2 s to 25.5 s. Measured again on a second cluster, it took a median 6.7 s (149 MB/s), from 5.3 s to 29.7 s. That run sat beside plain object-store puts of the same size on the same nodes. The slow runs spent their time in the executors' JVM garbage collection. Spark handed each 100 MB block from the step that computed it to the step that writes it. The writes themselves took 0.6-2.6 s each. The plain puts, which never cross that step, took 0.5-4.9 s.
  • Eight concurrent appenders. On Serverless, eight at once took 132 s against 150 s one after another. On Runtime 19 with two 4-core workers they took 166 s against 147 s. The eight shared eight cores as well as waiting between attempts.

How this was measured​

  • Data. The ERA5 reanalysis for January 2023: 2-metre temperature, 744 hourly steps on the 0.25-degree global grid (721 x 1440), float32. As Zarr, it is a v3 store. Its inner chunks are 24 x 90 x 90 cells, in shards of 168 x 360 x 360, compressed with zstd. For comparison, the same values as one tiled GeoTIFF per hour (512 x 512 tiles, ZSTD), read with the Raster Reader.
  • Compute. The reads ran on Databricks Runtime 19 with one 4-core worker (m5d.xlarge). The writes ran on Databricks Runtime 19 with two 4-core workers (m6gd.xlarge, on demand). Both used dedicated access. Both also ran on Serverless environment 6.
  • Method. Each figure is the median of three runs, the first of the three cold. The exception is the 1 GB create. Its figures are the median of five warm runs after a cold one. The two Runtime 19 legs (through the Volume, direct to storage) took turns.

Limits​

Runtimes and data​

  • Runtimes. Databricks Runtime 19 and Serverless environment 6 (Python 3.12), at the Python level only. The JVM level reads Zarr data only as materialized GeoBrix tiles.
  • Data types. Numeric types and CF times. Strings and objects are not supported. An array GeoBrix cannot decode is left out of reads and registered without being readable.
  • Complex arrays. A complex array can only be read as blocks. Tables have no complex column, and tiles do not hold complex numbers yet.
  • Tiles. output="tiles" needs the coordinates to form a regular grid. A tile of a registered store does not check its files for changes. A file changed in place reads as its new bytes, or fails. Materialize the tiles you keep.

Layout and codecs​

  • Chunk grid. Only a regular chunk grid is supported. An array with a rectilinear grid is left out of reads and registered without being readable.
  • Inner chunks per shard. A shard that gbx_zarr_write creates holds at most 4,096 inner chunks.
    • Every read of a shard fetches or builds its index, 16 bytes per inner chunk. A shard of many tiny chunks makes every read slow. On a Volume, reading one chunk of a shard of 93,312 chunks took a median of 0.87-1.41 s. At 4,096 chunks it took 0.10-0.15 s (ten runs on two runtimes).
    • The computed layout stays within the limit. A shards= that goes beyond it raises ValueError with a Fix:.
    • A registered store's shards are read as they are.
  • Codecs. The codecs that zarr-python, numcodecs and zarrs can decode. Writes use zstd and crc32c unless you pass codecs.
    • A Zarr v3 array that has any codec beside sharding_indexed in its own codec list is not supported. A transpose before it or a compressor after it are examples.
    • Registering such a store records the array as one GeoBrix cannot decode. A default read leaves it out with a warning. Naming it in variables raises ZarrUnsupported.
    • To fix it, rewrite the array so that sharding_indexed is its only outer codec, with compression and other codecs inside it. zarr-python does not write that layout.
  • Reading with GDAL. A GDAL before 3.13 cannot read sharded Zarr v3 stores, or ones that use the crc32c codec, which gbx_zarr_write writes by default; see the Raster Reader.
  • File modes. external is a preview that needs a workspace with the File type preview and compute that supports FILE values. managed is not available (see File modes).

Writers and storage access​

  • Outside writers. GeoBrix detects writes that other tools make to a registered store. It cannot make them transactional. On a Volume the identity of an object is its modification time and size, which is weaker than a content hash. A publish is not atomic for plain readers.
  • Branches. Registrations of one store do not merge.
  • Version history is bounded by retain. Tag a version to keep it indefinitely.
  • Many simultaneous appenders. A rebase retries up to 20 times, and the last of N simultaneous appenders needs about N attempts. A burst of more than about twenty writers appending to one array at the same instant can exhaust that. The last of them fails with ZarrWriteConflict and a Fix:. Run it again.
  • Direct storage access. access="auto" reads and writes cloud storage directly where Unity Catalog can issue temporary credentials (dedicated compute). Elsewhere, including Serverless, it goes through the Volume mount.
    • Direct access sends short-lived, Volume-scoped credentials (at most one hour) to the executors. Under memory pressure Spark may spill task data, those credentials included, to local disk.
    • Turn on local-disk encryption on clusters that use direct access, or pass access="fuse", which sends no credentials.
    • Direct access has been measured on AWS only. Azure and Google Cloud storage are not measured yet, and access="fuse" reads and writes through the Volume mount there.
  • Incomplete uploads. A pack upload that is refused or fails part way can leave an incomplete multipart upload in the bucket. GeoBrix does not clean it up. Add a bucket lifecycle rule that aborts incomplete multipart uploads (AbortIncompleteMultipartUpload) to the bucket of the Volume.

Next Steps​

  • Lifecycle and retention: gbx_zarr_optimize brings read times back after many small writes.
  • Zarr Writer: chunks presets and shaping blocks to the store's grid.
  • Zarr at scale: a notebook that times five reads and a table to Zarr write from 10 MB to 10 GB, with reference timings.
  • Limitations: the limits that apply across GeoBrix.