Skip to main content

Raster Reader

Read rasters into the shared (source, tile) schema. GeoBrix offers two interchangeable tiers: a lightweight pure-Python/PySpark reader (raster_gbx, rasterio-backed, JAR-free, Serverless-safe) and a heavyweight GDAL-backed reader (gdal). They emit the same schema, so swapping is a one-line format(...) change — see Choosing an Execution Tier.

The heavyweight gdal reader supports the full set of GDAL drivers (NetCDF, HDF5, COG, …); the lightweight reader covers the common raster path. The pairing is a corresponding general raster reader per tier, not a feature-identical one.

Benchmark & tradeoff

The lightweight (*_gbx) and heavyweight readers emit the same schema, but your compute usually decides the tier: the lightweight tier needs no JAR or init script and is the only option on Serverless, standard (shared), and ARM clusters. The heavyweight tier requires a classic x86 cluster (JAR + GDAL init script); where it is available it uses native GDAL on the JVM and tends to pull ahead on large workloads. See the Benchmarking page for light-vs-heavy timings and methodology.

Options​

Lightweight (raster_gbx)​

The lightweight reader defaults to no splitting (splitStrategy=none): one whole-image tile per file. Splitting is opt-in; COG creation is a separate concern handled by the cog_gbx writer. See COG Reader and COG Writer for the prepare-then-read pipeline.

OptionDefaultDescription
virtualTiles"true"Default. Emit bytes-free virtual tiles — each row carries the source path + pixel window instead of raster bytes; pixels are read lazily when an operation needs them (the ingest-OOM-dissolving default for the light tier). Set "false" to materialize raster bytes into each row. See Virtual Tiles.
splitStrategy"none"When to split large rasters: none (default — one whole-image tile per file), auto (resolve to serverless or classic by environment), serverless (opt-in, 64 MiB decoded budget per tile), classic (opt-in, 1 536 MiB decoded budget per tile).
sizeInMB"-1"Power-user override: set a positive value to pin the per-tile decoded-memory budget in MiB. A positive value implies opt-in split regardless of splitStrategy. -1 = defer to splitStrategy.
filterRegex".*"When loading a directory, keep files whose full path matches this regex.
clipPolygonsnoneArea(s) of interest to clip to. One WKT/EWKT string for a single polygon, or a JSON-array string for a list, e.g. '["POLYGON((...))","POLYGON((...))"]' (raw WKB/EWKB bytes are accepted only from programmatic callers). One tile is emitted per polygon whose envelope intersects the raster; a polygon that misses the raster emits no tile. Materialized tiles are pre-clipped (pixels outside the polygon are NoData); virtual tiles carry the clip as a deferred instruction. Mutually exclusive with windows and tileSize. Pairs naturally with the COG Reader.
windowsnonePixel window(s) to read. A JSON 4-int array "[col,row,w,h]" for one, or a JSON array of them "[[..],[..]]" for a list. One tile per window; partial windows clip to the raster extent, fully-outside windows are skipped. Mutually exclusive with clipPolygons and tileSize.
clipCrsnoneCRS for clipPolygons that lack an embedded SRID (e.g. "EPSG:27700"). Precedence: an EWKB/EWKT polygon's embedded SRID → clipCrs → the raster's CRS.
tileSizenoneCut the whole raster into a regular fixed-size pixel grid: "w,h" (e.g. "512,512") or a single "512" for square tiles. One tile per grid cell; edge cells are clamped to the extent. Mutually exclusive with clipPolygons and windows. Materialized tiles are guarded to the ~2 GB Spark-cell limit (a too-large tileSize raises a clear error); virtual tiles (virtualTiles=true) carry no bytes and are unguarded.
overlapPercent0Overlap between adjacent tileSize cells, as a percentage of tile size (overlap_px = ceil(tile_dim · pct/100), step = tile_dim − overlap_px). Applies to tileSize only (an error otherwise). Default 0 = non-overlapping grid.
tilesTablenoneDelta table to use as a tile index (see pre-computed tile inputs below).
skipOrdering"false"Pass "true" to suppress the auto-sort-by-source-path that tilesTable reads apply by default. See tilesTable auto-order below.
Why overlap?

A hard tiling grid can slice straight through a spatial feature (a field, a building, an airport) that straddles a seam, so no single tile sees it whole. Set overlapPercent so adjacent tileSize cells overlap — a feature near a boundary then appears complete in at least one tile, which per-tile analysis needs. Overlap only makes sense for a reader-chosen grid (tileSize); windows and clipPolygons are extents you specify explicitly, so they take no overlap.

Virtual tiles are the default

Light raster readers now emit virtual tiles by default (virtualTiles=true) — bytes-free (path, window) references that read pixels lazily. Previously the reader materialized raster bytes into every row. To restore materialized reads, pass .option("virtualTiles", "false"). A virtual tile passed to a heavyweight function must be materialized first — see Virtual Tiles.

Default changed from auto to none

Prior to this release the reader defaulted to splitStrategy=auto, which auto-split large rasters on a decoded-memory budget. The default is now none — one tile per file. To opt back into splitting, pass .option("splitStrategy", "serverless") or .option("splitStrategy", "classic").

When splitting is enabled, split tiles are emitted as plain GeoTIFF. COG output is a cog_gbx writer concern, not a reader concern: the tileFormat, cogBlockSize, and cogOverviewResampling reader options are removed. For COG-encoded output, use the COG Writer to prepare master COGs, then read them with the COG Reader.

Striped GeoTIFFs (no internal tiling) split into full-width row-bands. Internally-tiled sources split on a block-snapped grid.

Directory reads​

Reading a directory plans every file (fast, stat-free listing — ~1.5 s at 10k files); for pre-computed windows or very large tile counts see Advanced: pre-computed tile inputs below.

Reading GeoBrix output with a stock Spark reader

Stock Spark file readers (spark.read.format("binaryFile"), text, parquet, …) silently skip any file whose name starts with _ or . — Spark's hidden-file filter, applied during directory listing. Some GeoBrix raster tiles are written with a leading _ in their (hashed) name, so a stock reader can miss a large fraction of a directory. Note that .option("pathGlobFilter", "*.tif") does not override this filter — the filter runs before the glob, so _-prefixed files stay excluded (an explicit _* glob, explicit _-file paths, and recursiveFileLookup do not help either).

To read every tile, enumerate the paths on the driver (which does not apply the filter) and build the DataFrame explicitly — ideal when you only need the file list, e.g. to feed a VRT builder:

from pathlib import Path
paths = [str(p) for p in Path(output_dir).rglob("*.tif")]
files_df = spark.createDataFrame([(p,) for p in paths], ["path"])

GeoBrix's own raster readers (raster_gbx / gtiff_gbx / cog_gbx) list files directly and are not affected — this applies only to stock Spark file readers.

Opt-in split (serverless budget):

# Two-axis control: split strategy and output format.
df = (
    spark.read.format("raster_gbx")
         .option("splitStrategy", "serverless")   # or: classic | none | auto
         .option("tileFormat", "cog")             # or: gtiff | auto
         .option("cogBlockSize", "512")           # tile size for COG internal grid (px)
         .option("cogOverviewResampling", "AVERAGE")  # overview resampling algorithm
         .load("/Volumes/main/geobrix_samples/geobrix-examples/nyc/sentinel2")
)
Classic READ_WITH_OPTIONS example

The sizeInMB option remains available as a power-user override:

# Options: sizeInMB (tile split threshold) + filterRegex (directory listing)
df = (spark.read.format("raster_gbx")
      .option("sizeInMB", "16")
      .option("filterRegex", r".*\.tif$")
      .load("{SAMPLE_RASTER_PATH}"))

gtiff_gbx is raster_gbx with the GeoTIFF driver preset.

COG lane for large rasters​

For large rasters, the recommended path is to prepare master COGs with the cog_gbx writer and then clip windows with the cog_gbx reader. This keeps the reader simple (no split, no re-encode) and lets GDAL's range-read mechanism fetch only the bytes that intersect the AOI.

Who benefits from pre-built COG overviews:

  • rst_tilexyz and rst_xyzpyramid — the XYZ tile-serving pipeline uses rio-tiler, which automatically selects the appropriate COG overview level for the requested zoom.
  • rst_resample* — the resample family triggers overview-level selection automatically when a COG source has pre-built overviews.

COG preparation + windowed read:

See COG Writer for the preparation step and COG Reader for the windowed read.

Force-writing COG output from a tile DataFrame: the gtiff_gbx writer still accepts a cog=true option to re-encode any (source, tile) DataFrame as COG when you already have tiles in memory:

# Force-convert to COG on write (any DataFrame with a tile column):
import tempfile
with tempfile.TemporaryDirectory() as out:
    df.write.format("gtiff_gbx") \
        .mode("overwrite") \
        .option("cog", "true") \
        .option("cogBlockSize", "512") \
        .option("cogOverviewResampling", "AVERAGE") \
        .option("cogCompression", "DEFLATE") \
        .save(out)
    cog_df = spark.read.format("raster_gbx").load(out)
    print(cog_df.count(), "COG tiles written and read back")

Heavyweight (gdal)​

OptionDefaultDescription
driverauto-detected from extensionExplicitly specify the GDAL driver to use (regardless of extension).
sizeInMB"-1"Default (<= 0) = no split: one whole-image tile per file. Set a positive MB value to split large files into multiple tiles for parallel processing.
filterRegex".*"Filter files by regex when reading from a directory.
readSubdatasets"false"Read subdatasets if present (e.g. HDF, NetCDF).
rasterAsGrid"false"Read as grid instead of tiles.
retile"false"Retile rasters for optimal processing.
tileSize"256"Tile size in pixels (if retiling enabled).

Example — forcing the GDAL driver explicitly:

# Read with explicit driver (sample-data Volumes path)
df = spark.read.format("gdal") \
    .option("driver", "GTiff") \
    .load("{SAMPLE_RASTER_PATH}")
df.show()
Example output
+--------------------------------------------------+-----+
|path                                              |tile |
+--------------------------------------------------+-----+
|/Volumes/.../nyc_sentinel2_red.tif                |{...}|
+--------------------------------------------------+-----+

Register​

# Register the lightweight raster DataSources (once per session)
from databricks.labs.gbx.ds.register import register
register(spark)
Lightweight readers and writers are not auto-registered

This is a key difference from the heavyweight tier. The heavyweight readers/writers (gdal, gtiff_gdal, …) are auto-discovered from the JAR on the classpath via Spark's JVM DataSourceRegister service loader, so spark.read.format("gdal") works with no setup call. The lightweight readers/writers are Python Data Source V2 sources, and Python has no classpath auto-discovery equivalent — so you must register them explicitly with register(spark) (above) before using format("raster_gbx") / format("gtiff_gbx") for reads or writes.

Importing databricks.labs.gbx.ds will opportunistically register them if a Spark session is already active, but the explicit register(spark) call is the reliable path, in case your session is created after imports. (This mirrors the heavyweight gbx_rst_* SQL functions, which also require an explicit register(spark).)

Read (catch-all)​

# Catch-all lightweight reader (any rasterio-readable raster)
df = spark.read.format("raster_gbx").load("{SAMPLE_RASTER_PATH}")
df.show()
Example output
+--------------------------------------------------+-----+
|source                                            |tile |
+--------------------------------------------------+-----+
|/Volumes/.../nyc_sentinel2_red.tif                |{...}|
+--------------------------------------------------+-----+

gtiff_gbx is raster_gbx with the GeoTIFF driver preset. See the Lightweight GeoTIFF Reader for the named-reader page. See the page-level Options section below for the available reader options.

See Choosing an Execution Tier for the full tradeoff and the Benchmarking page for light-vs-heavy timings.

It is the lightweight counterpart of the heavyweight gdal reader, supporting Python and SQL bindings (not Scala).

Common Raster Formats Explained​

GeoTIFF - The Universal Choice​

Best for: General-purpose geospatial rasters, aerial imagery, DEMs

GeoTIFF combines TIFF image format with embedded geospatial metadata (coordinate system, geotransform). It's the de facto standard because it's simple, widely supported, and works everywhere. Cloud-Optimized GeoTIFF (COG) adds internal tiling and overviews for efficient cloud storage access.

# Standard GeoTIFF
df = spark.read.format("gtiff_gdal").load("/path/to/elevation.tif")

# Works with COG too
df = spark.read.format("gtiff_gdal").load("s3://bucket/cog-file.tif")

NetCDF - Multi-Dimensional Science Data​

Best for: Climate models, oceanographic data, time-series rasters

NetCDF excels at storing multi-dimensional arrays with labeled dimensions (time, latitude, longitude, elevation). Common in scientific computing for weather forecasts, climate projections, and oceanographic measurements.

# NetCDF with multiple variables/subdatasets
df = spark.read.format("gdal") \
    .option("driver", "NetCDF") \
    .option("readSubdatasets", "true") \
    .load("/path/to/climate_model.nc")

HDF5 - Massive Hierarchical Data​

Best for: Large scientific datasets, satellite products (MODIS, Sentinel)

HDF5 (Hierarchical Data Format) handles extremely large datasets with complex internal structures. NASA and ESA use it for satellite products. Like NetCDF, it often contains multiple subdatasets.

# HDF5 from satellite products
df = spark.read.format("gdal") \
    .option("driver", "HDF5") \
    .option("readSubdatasets", "true") \
    .load("/path/to/MOD13Q1.hdf")

GRIB/GRIB2 - Weather Models​

Best for: Numerical weather prediction, meteorological data

GRIB (GRIdded Binary) is the standard format for weather model outputs from agencies like NOAA, ECMWF. Highly compressed and optimized for meteorological variables. Example uses NOAA HRRR weather data from sample-data (nyc/hrrr-weather).

Complete bundle only

The hrrr-weather dataset is included in the complete sample-data bundle, not the essential bundle. See Sample data for download options.

# GRIB2 weather data (sample-data HRRR)
df = spark.read.format("gdal") \
    .option("driver", "GRIB") \
    .load("{SAMPLE_HRRR_PATH}")
Example output
+--------------------------------------------------+-----+
|path                                              |tile |
+--------------------------------------------------+-----+
|.../nyc/hrrr-weather/hrrr_nyc_....grib2           |{...}|
+--------------------------------------------------+-----+

Format Selection Guide​

FormatSizeCompressionMulti-BandTime-SeriesCloud-Friendly
GeoTIFFGoodGoodYesNoYes (COG)
NetCDFExcellentGoodYesYesModerate
HDF5ExcellentGoodYesYesPoor
GRIB2ExcellentExcellentYesYesPoor
JPEG2000ExcellentExcellentYesNoModerate
ZarrExcellentGoodYesYesExcellent

General Rules:

  • Start with GeoTIFF (use COG for cloud)
  • Use NetCDF for multi-dimensional scientific data
  • Use HDF5 when required by data provider (e.g., MODIS)
  • Use GRIB2 for weather models
  • Use Zarr for cloud-native analysis at scale

Common functions: used vs excluded​

See GBX Common Functions for the full catalog of shared file-access primitives. The table below shows which are active in this reader and which are not, and why.

Common capabilityUsed here?How / why
list_local_files (session-free enumeration)UsedAll directory reads — recursive, include_hidden, extensions, path_glob_filter options are routed through this shared predicate.
enumerate_files (FILE-tier enumeration)Not in the DataSourceThe DataSource is session-less on Connect and does not call enumerate_files; FILE-tier enumeration is only available through gbx_file_read at the function layer.
gbx_file_read / gbx_file_write (FILE tier)Not in the DataSourceThe DataSource is FUSE-only (session-less on Connect); FILE reads go through gbx_file_read → rst_fromfile at the function layer.

Advanced: pre-computed tile inputs (manifest / tile index)​

The default directory read is fast — a stat-free listing plans ~10k files in ~1.5 s — so most reads need nothing special. The manifest and tilesTable options are an optional optimization for two specific cases:

  • Pre-computed windows you already maintain — a manifest or tile index lets the reader use your saved file listing directly, skipping the directory walk entirely.
  • Very large tile counts where even a fast listing accumulates across many repeated job runs.

Both options reduce plan time from the directory walk (~1.5 s at 10k files) to a single file or table read. See Benchmarking → Reader plan-time listing for the measured numbers.

  • manifest option — supply a JSON or Parquet file listing path + window for each tile; the reader skips the directory walk and header opens entirely. 
    spark.read.format("raster_gbx") \
        .option("manifest", "/Volumes/catalog/schema/vol/tiles.json") \
        .option("virtualTiles", "true") \
        .load("/Volumes/catalog/schema/vol/rasters")
  • tilesTable option — point to a Delta table that serves as a tile index: one row per tile with a path column (+ optional window / dimension columns). It is a catalog of file references, not the tile pixels — the reader still opens each referenced file and extracts its window; it only skips discovering the files by walking the directory. 
    spark.read.format("raster_gbx") \
        .option("tilesTable", "geospatial.myschema.tile_index") \
        .option("virtualTiles", "true") \
        .load("/")
  • Fewer, larger COGs — consolidate small files into Cloud-Optimized GeoTIFFs using the COG Writer; the reader then opens a small number of large files whose headers are comparatively cheap.

A tile index is not a materialized-tile table. manifest / tilesTable catalog where the rasters are (paths, plus optional windows) — essentially a saved file listing. The raster bytes stay in files and are read lazily (especially with virtualTiles=true); the index only lets the reader skip re-walking the directory. That is different from an ingest that has already decoded rasters into a Delta table with a tile-struct/bytes column — if you have that, read it directly with spark.table(...); you don't need a reader at all.

tilesTable auto-order and skipOrdering​

When tilesTable is set, the reader auto-orders tiles by source path before planning partitions. This groups all tiles from the same source GeoTIFF into adjacent partitions, so a worker that opens a large raster once can decode all its windows without reopening it. For a table with many tiles per source file (a common windowed-GeoTIFF ingest), this amortization meaningfully reduces open overhead — each file is opened once per partition instead of once per tile.

Pass skipOrdering="true" when the table is already physically ordered (e.g. written with layout="order" or layout="cluster") or when you apply ordering downstream:

spark.read.format("raster_gbx") \
    .option("tilesTable", "geospatial.myschema.tile_index") \
    .option("skipOrdering", "true") \
    .option("virtualTiles", "true") \
    .load("/")

Durable co-location: CLUSTER BY + OPTIMIZE​

layout="order" (the default write layout) sorts rows by path at insert time, which groups tiles per source for the lifetime of that write. New inserts and compaction can scatter them. For a stable layout that survives compaction, use layout="cluster" at write time and run OPTIMIZE afterward:

from databricks.labs.gbx.ds.file_gbx import gbx_file_write

gbx_file_write(
    tiles_df,
    target="geospatial.myschema.tile_index",
    layout="cluster",      # CLUSTER BY path in the DDL
    file_mode="auto",
)
-- After initial write (or after bulk inserts), materialize clustering:
OPTIMIZE geospatial.myschema.tile_index;

With layout="cluster", OPTIMIZE physically co-locates tiles from the same source into the same data files. Subsequent reads with tilesTable have source-adjacent rows from the start — skipOrdering="true" is safe once the table has been optimized, and it avoids re-sorting an already-ordered scan.

layout="order" (insert-time ORDER BY path) is the simpler default: no DDL change, no OPTIMIZE required. It works well for tables that are built once. Use layout="cluster" when the table is updated incrementally and you run periodic OPTIMIZE as part of maintenance.

Next Steps​

Shared file-access layer

Lightweight readers use the shared file_gbx file-access base for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there.