Materialized Compression

When a virtual tile becomes bytes on disk or in a Spark row — during a materialize operation, a write, or a tile format conversion — compression is applied. This page explains what GeoBrix compresses by default, how it chooses compression settings for different tile sizes, and when you might want to override the defaults.

What gets compressed

A materialized tile carries its raster payload as encoded bytes. Those bytes flow through Spark DataFrames (each row's tile.raster field holds up to a few hundred megabytes) and are written to files when you persist results. At each step, GDAL's GeoTIFF codec is applied, choosing a balance of compression ratio, memory use, and encoding time.

For the structure of a tile and the full virtual↔materialized lifecycle, see Tile Structure and Virtual Tiles.

GeoBrix's approach: ZSTD + dtype-matched predictor

The default compression in GeoBrix is ZSTD paired with a dtype-matched predictor that reorders bytes to improve compression ratio:

• float32 / float64: predictor 3 (horizontal differencing for floating-point data)
• int16 / uint16 / int32 / uint32: predictor 2 (delta encoding for integer data)
• uint8 / int8: predictor 1 (byte data; predictor adds negligible benefit)

The auto (default) compression mode selects a size-adaptive ZSTD level. A full level sweep over real and synthetic GeoBrix tiles established L6 as the size knee: levels above L6 (L9, L12) add encode CPU and peak RSS with negligible ratio benefit, and L16 can produce larger output than L6 on floating-point data. Larger tiles get progressively lower levels to protect worker memory and write throughput:

Decoded tile size	ZSTD level	Why
≤ 128 MiB	6	Ratio knee — higher levels add encode cost with no output-size benefit
128 MiB – 1 GiB	3	Protects worker memory; large tiles need headroom
> 1 GiB	1	OOM guard; encode throughput is the constraint at this size

Note: The ≤ 128 MiB rung (L6) is validated. The > 128 MiB rungs (L3, L1) are extrapolated from the measured 128 MiB trend and pending Serverless confirmation at larger payloads.

This ladder is grounded by a benchmark of real encoding across codec families, sizes, and data types (see Evidence below).

When auto is the right choice

auto is the default and works for nearly all cases:

• Tiles are compressed once at materialize or write time.
• Readers (both in-cluster and off-cluster) decompress just-in-time, and read cost tracks output file size, not the level used to write — decompression speed is level-independent.
• The size-adaptive level protects Serverless workers from memory spikes at large payloads.

Use auto unless you have a specific reason not to.

GBX_ZSTD_LEVEL opt-in

Set GBX_ZSTD_LEVEL in the executor environment to override the size-adaptive ladder on the compress='auto' path only (an explicit compress="zstd" + compressLevel call is unaffected):

Value	Behaviour
unset or default	Size-adaptive ladder above (the default)
fast	L1 everywhere — highest encode throughput, output ~0.3–2.5% larger than L6
max	L9 everywhere — heavy-tier-parity ceiling; no smaller output than L6 on GeoBrix tiles
1–22	Explicit fixed level applied to all tile sizes, size-independent

Because decompression speed is level-independent, GBX_ZSTD_LEVEL helps encode throughput only, not reads. fast (L1) is the right choice when encode CPU is the bottleneck and a small size premium is acceptable; max (L9) matches the old ladder's ceiling if hard parity with the heavyweight tier is required.

When to reach for other codecs

compress="deflate" — For maximum portability to tools that don't support ZSTD:

• Older GDAL versions (pre-3.1) lack ZSTD support off-cluster.
• If you hand off files to external tools, DEFLATE is more widely supported.
• Trade-off: DEFLATE is 2–3× slower to write than ZSTD at the same ratio, and doesn't adapt to tile size.

compress="zstd" with explicit compressLevel — For write-once, read-many catalogs:

• If you're building a reference dataset that will be read many times, a high fixed level (e.g. compressLevel=19 for sizes up to 128 MiB) pays off over the lifetime of many reads.
• Warning: levels ≥ 19 get exponentially slower to write (100–300ms per small tile) and use significant memory (100+ MiB per encode); use only on the driver or small batches.

compress="lzw" or compress="none" — Rarely needed:

• LZW expands float32 and continuous data; use only for categorical/integer classification data.
• No compression is occasionally useful for debugging, but costs both space and downstream reading time.

Control surface: compress / compressLevel / predictor

When writing tiles, you can override the codec and settings:

# Write with the default size-adaptive ZSTD (recommended)
df.write.format("gtiff_gbx").option("compress", "auto").save(...)

# Write with DEFLATE, level 9 (maximum portability)
df.write.format("gtiff_gbx") \
  .option("compress", "deflate") \
  .option("compressLevel", "9") \
  .save(...)

# Explicit ZSTD at a fixed high level (write-once catalog)
df.write.format("gtiff_gbx") \
  .option("compress", "zstd") \
  .option("compressLevel", "16") \
  .save(...)

# No compression (debugging only)
df.write.format("gtiff_gbx").option("compress", "none").save(...)

For prepare_cogs (driver-based COG preparation), pass the same options as kwargs:

from databricks.labs.gbx.pyrx.core.preparer import prepare_cogs

prepare_cogs(
    input_dir,
    output_dir,
    compress="deflate",
    compressLevel=9,
    verbose=True
)

The deprecated option name cogCompression is accepted as an alias for compress in the writer.

Portability: ZSTD off-cluster

Cloud-Optimized GeoTIFFs prepared with ZSTD can be read on Databricks (in-cluster GDAL has ZSTD support), but off-cluster tools may not:

• GDAL 3.1+ (and rasterio 1.2+) supports ZSTD natively.
• GDAL 3.0 and earlier will fail to open ZSTD-compressed GeoTIFFs.

If you expect downstream processing by tools without ZSTD support, use compress="deflate" instead. The compression ratio is similar (1–2% difference), and DEFLATE is universally available.

Evidence table

The following table shows popular codec combinations (none, LZW, DEFLATE, and ZSTD at levels 9–16+predictor) across realistic tile sizes and data types. The benchmark ran on macOS arm64 with GDAL 3.12 and rasterio 1.5. A subsequent full level sweep (L1–L22, 11 tile configurations) confirmed that L6 is the ratio knee: L9 and L12 add write time and RSS with negligible size benefit, and L16 can produce larger output than L6 on high-entropy float32 tiles. Read decompression time was flat across all levels tested. Sizes 256 MiB–1 GiB are extrapolated and pending Serverless validation.

Size	Codec	Dtype	Ratio	Write (ms)	Read (ms)	Peak RSS (MiB)
1 MiB	none	float32	0.99×	8	0.5	14
1 MiB	DEFLATE z6 + p3	float32	1.17×	25	5.0	14
1 MiB	ZSTD L9 + p3	float32	1.17×	18	1.8	24
1 MiB	ZSTD L16 + p3	float32	1.18×	44	1.9	46
8 MiB	none	float32	1.00×	12	2.0	45
8 MiB	DEFLATE z6 + p3	float32	1.18×	168	39	35
8 MiB	ZSTD L9 + p3	float32	1.17×	76	11	45
8 MiB	ZSTD L12 + p3	float32	1.17×	110	11	75
32 MiB	none	float32	1.00×	23	8	143
32 MiB	DEFLATE z6 + p3	float32	1.19×	593	143	99
32 MiB	ZSTD L9 + p3	float32	1.19×	234	40	109
32 MiB	ZSTD L12 + p3	float32	1.19×	347	41	112
128 MiB	none	float32	1.00×	63	36	532
128 MiB	DEFLATE z6 + p3	float32	1.20×	2827	535	355
128 MiB	ZSTD L9 + p3	float32	1.20×	887	167	375
128 MiB	ZSTD L12 + p3	float32	1.20×	1291	166	395

Key observations:

• ZSTD L6 + predictor beats or matches DEFLATE on ratio, while writing 2–3× faster across all sizes.
• L6 is the ratio knee for GeoBrix tiles: L9 and L12 add encode time and peak RSS with negligible output-size benefit; L16 can produce larger output than L6 on high-entropy float32 data (level-sweep finding, not shown above).
• Read decompression speed is level-independent — output file size, not the write level, determines read cost. Choosing a higher write level does not improve read throughput.
• LZW expands float32 data (not shown); use only for categorical integer rasters.

---

Next steps

• Large Rasters — preparing COGs and understanding memory constraints
• Virtual Tiles — the virtual↔materialized lifecycle and when tiles materialize
• Tile Structure — the internal schema of a raster tile