Zarr: Transactions and Concurrent Writers
Since: v0.6.0 · (Python level; no JAR needed)
Use this page to learn what a write guarantees. Every gbx_zarr_write is all-or-nothing: readers see
the whole write or none of it. Several jobs can write to one store at the same time, and you can retry a
write safely. This page shows how a write works, what happens when two writers meet, and what to do when
one loses.
The examples use the names and the sample cube from Zarr Stores. For every
parameter see gbx_zarr_write, and for the how-to see the
Zarr Writer.
The four modes
| Mode | What it does |
|---|---|
create | Makes a new store. |
append | Extends one dimension, by default the first time dimension. |
update | Writes into the existing extent. |
overwrite | Replaces whole variables. Older versions stay readable until vacuumed. |
Writing a variable the store lacks creates it on the store's dimensions. Appends label the new steps. They must come after the store's last step. On a regular axis they land at their step, so appends may arrive out of order.
from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write
following = gbx_zarr_read(
spark,
cube,
variables=["t2m", "msl", "time", "latitude", "longitude"],
sel={"time": slice("2023-01-01T12:00", "2023-01-01T23:00")},
)
# The new `time` labels must come after the store's last one.
version = gbx_zarr_write(following, table, mode="append")
How a write commits
A write happens in two phases, and only the second one is visible.

- Files. The input is grouped by the shard it belongs to. Each task encodes the chunks it owns. It writes one pack file for each array it holds, each in a single sequential write. A chunk the input covers completely is encoded fresh. A chunk it only partly covers is read at the base version, patched and re-encoded. Each task hands back a small manifest row for every chunk it wrote.
- Commit. The driver collects the rows and runs one Delta MERGE. The MERGE writes:
- the chunk rows, with a compare-and-swap (CAS) on every key it replaces. The CAS compares the
write_idthe key had in the base version, or "still absent" for a new key; - the changed array and coordinate rows;
- a
commitrow keyed by the write'swrite_id.
- the chunk rows, with a compare-and-swap (CAS) on every key it replaces. The CAS compares the
Until the MERGE commits, nothing is visible. If anything fails before it, the version has not moved. The packs are unreferenced files that a later vacuum removes. After the MERGE commits, the whole write is visible at once.
Concurrent writers

The manifest decides, row by row.
-
Different data. Writers that touch different chunks touch different rows. Delta lands both without a retry.
-
Shared metadata. Two appenders both change the array's shape, and often a shared coordinate chunk. When a writer finds that another has committed those rows first, it rebases:
- It re-reads the latest version.
- It checks that none of the data keys it replaces has changed since its base (the CAS).
- It recomputes the shape (the maximum extent) and redoes the coordinate chunk.
- It retries the MERGE.
Its data files are never rewritten. A rebase retries up to 20 times with a jittered wait. Every lost round means another writer landed, so the last of N simultaneous appenders needs about N attempts. In GeoBrix's checks, 8 appenders to one array at the same moment all landed.
-
The same chunk. Suppose another writer committed a change to a chunk this write also touches, after this write's base version. Then the write raises
ZarrWriteConflict. That includes two partial writes to different cells of one chunk, because each is a read-modify-write of the whole chunk.
Two jobs appending at the same moment, as they would from two clusters:
from concurrent.futures import ThreadPoolExecutor
from databricks.labs.gbx.zarr import gbx_zarr_read, gbx_zarr_write
def hours(start, stop):
return gbx_zarr_read(
spark,
cube,
variables=["t2m", "msl", "time", "latitude", "longitude"],
sel={"time": slice(start, stop)},
)
morning = hours("2023-01-02T00:00", "2023-01-02T11:00")
afternoon = hours("2023-01-02T12:00", "2023-01-02T23:00")
with ThreadPoolExecutor(max_workers=2) as pool:
versions = list(
pool.map(
lambda block: gbx_zarr_write(block, table, mode="append"),
[morning, afternoon],
)
)
Both appends land, in whichever order they reach the commit. The store ends with all 48 hours in the right place.
Update a region
An update that corrects a region reads the blocks, changes them and writes them back by position:
from databricks.labs.gbx.zarr import (
BLOCKS_SCHEMA,
block_to_numpy,
gbx_zarr_read,
gbx_zarr_write,
)
region = {"time": slice("2023-01-01T00:00", "2023-01-01T11:00")}
rows = gbx_zarr_read(spark, table, variables=["t2m"], sel=region).collect()
corrected = [
{
**row.asDict(),
"data": (block_to_numpy(row) - 1.0).astype("float32").tobytes(),
}
for row in rows
]
patch = spark.createDataFrame(corrected, BLOCKS_SCHEMA)
version = gbx_zarr_write(patch, table, mode="update")
Handle a conflict
A conflict names the keys and the write that got there first. Nothing of the losing write is visible. The safe response is to read the region again and write again:
from databricks.labs.gbx.zarr import ZarrWriteConflict, gbx_zarr_write
try:
gbx_zarr_write(patch, table, mode="update")
except ZarrWriteConflict as err:
# Another write changed these chunks after this one read them; none of this write
# is visible. Read the region again, rebuild the patch and write again.
print("lost to", err.winner, "on", err.keys)
Retry safely with a write ID
A write can carry its own write_id (32 lowercase hex characters). Writing again with an id that has
already committed returns that version and writes nothing. A job that is retried after a timeout
cannot apply the same write twice:
import uuid
from databricks.labs.gbx.zarr import gbx_zarr_write
write_id = uuid.uuid4().hex # 32 hex characters, chosen before the first attempt
version = gbx_zarr_write(patch, table, mode="update", write_id=write_id)
# If the first attempt committed but never reported back, run it again. A write_id
# that already committed returns its version and writes nothing.
again = gbx_zarr_write(patch, table, mode="update", write_id=write_id)
Errors raised while Spark runs a read or a write reach the driver as a PythonException. Its message
keeps the GeoBrix error name and the Fix: line. Every GeoBrix Zarr error ends with a Fix: that says
how to resolve it.
Next Steps
- Versions, time travel and tags: read the store as it was before a write.
- Lifecycle and retention: clean up the files that failed or superseded writes leave.
- Performance and limits: how concurrent appenders behave at scale.
- Zarr Writer: the how-to guide.