NetCDF Reader

Since: lightweight netcdf_gbx v0.4.1 · heavyweight netcdf_gdal / netcdf_ogr v0.4.3

Read CF-convention NetCDF into GeoBrix's existing schemas — no new downstream contracts. GeoBrix ships three NetCDF readers:

• netcdf_gbx — the lightweight (rasterio/netcdf4-backed, JAR-free) reader, with a raster and a vector mode. Runs on Serverless, standard (shared), and ARM clusters.
• netcdf_gdal — the heavyweight raster reader (RasterX): CF grid variables → the shared (source, tile) schema, native GDAL on the JVM.
• netcdf_ogr — the heavyweight vector reader (VectorX): native CF Discrete Sampling Geometry (DSG) features → the shared vector schema.

A NetCDF file is a bag of labelled N-D arrays, and only some layouts are maps, so the readers split along that grain: gridded variables go through the raster path (netcdf_gbx raster mode / netcdf_gdal), and feature/point data goes through the vector path (netcdf_gbx vector mode / netcdf_ogr).

Selecting variables — the unified contract

All three readers share one rule: variable / variables is an optional filter, not a required selector.

• A bare load (no variable / variables) processes all readable variables — in raster mode, one (source, tile) row per grid variable; in vector mode, all DSG / curvilinear features.
• Passing variable (one) or variables (a comma-separated list) filters to exactly those variables.

The source column on the raster path is the GDAL subdataset selector, NETCDF:"{path}":{var}, which identifies the originating variable per row.

Behavior change

Earlier netcdf_gbx required a variable / variables option — a bare load raised. It is now an optional filter, so a bare load returns every readable variable. Explicit-variable calls are unchanged.

netcdf_gbx — lightweight, two modes

The lightweight reader has two modes, chosen by the mode option:

• raster (default) — a regular lat/lon or projected grid is transcoded to the shared (source, tile) GeoTIFF schema, exactly like the other raster readers. Everything downstream (rst_h3_tessellate, band math, tiling) works unchanged. ERA5 reanalysis grids read this way.
• vector — point (discrete-sampling) data, or any 2-D field including a curvilinear/swath grid, is emitted as one point per cell (cell-centre lon/lat + values). This reads swath products losslessly, with no regridding or interpolation choices baked in. Sentinel-5P TROPOMI CH4 (a netCDF-4 swath) reads this way. Flattening a swath to per-cell points is a netcdf_gbx vector-mode capability; the heavyweight netcdf_ogr reads native DSG features only.

Honest by construction

The reader never resamples and never applies quality thresholds. In vector mode every variable — including a quality flag such as Sentinel-5P qa_value — travels through as its own column, so you decide how to filter. Raw sensor-geometry products (no per-pixel lon/lat) are rejected with an actionable error rather than silently guessed at.

NetCDF readers always materialize tiles

NetCDF raster reads are multidimensional per-variable reads that materialize tile bytes directly into each row. Virtual-tile support (virtualTiles) for the NetCDF readers is not part of this release.

Options

Option	Default	Description
mode	"raster"	raster (CF grid → tile) or vector (points / per-cell points).
variable / variables	— (all)	Optional filter. Absent → all readable variables. variable takes one; variables takes a comma-separated list (vector-mode attribute columns).
group	—	HDF5 group path for grouped NetCDF-4 files (e.g. Sentinel-5P "/PRODUCT").
dimIndex	—	Raster mode: pin one or more leading (non-spatial) dimensions to a specific integer index. Format: "dim=index" or "dim1=i1,dim2=i2". Example: "time=2,level=1". Absent dims fall back to index 0 (with a warning when the dim has more than one slice). An out-of-range index raises ValueError.
fanout	—	Raster mode: expand one or more leading dimensions across all their indices. Format: "dim" or "dim1,dim2". Each index combination becomes a separate (source, tile) row. A variable that lacks any of the named dims emits a single row (no expansion). Dims not listed are pinned by dimIndex or default to index 0. A dim that appears in both dimIndex and fanout raises ValueError.
bandDim	—	Raster mode: stack one leading dimension's slices as bands in a single multi-band tile. Format: "dim" (one dim only). bandDim="time" on a variable with (time=3, lat, lon) produces one 3-band tile. Composes with dimIndex (pins other dims) and fanout (one multi-band tile per fanout combination). A dim named in bandDim cannot also appear in dimIndex or fanout — this raises ValueError. Estimated decoded bytes > ~256 MiB emits a UserWarning and suggests fanout instead.
bbox / bboxCrs	—	Area-of-interest filter ("minx,miny,maxx,maxy").
filterRegex	".*"	When loading a directory, keep files whose full path matches.
sizeInMB	"-1"	Raster mode: <= 0 = one whole tile per variable.

Register the lightweight DataSources once per session:

# Register the lightweight DataSources (once per session)
from databricks.labs.gbx.ds.register import register
register(spark)

Raster mode — CF grids → tiles

# Raster mode (default): a CF regular lat/lon grid -> (source, tile).
# e.g. ERA5 2m-temperature on a regular grid.
df = (spark.read.format("netcdf_gbx")
      .option("variable", "t2m")
      .load("/Volumes/main/geobrix_samples/netcdf/era5_sample.nc"))
df.show()

The output is the standard (source, tile) schema — see Tile Structure. Only regular lat/lon or projected (CF grid_mapping) grids are accepted in raster mode; a curvilinear/swath variable raises an error steering you to mode=vector.

Multi-dimensional variables (time / level)

Climate reanalysis, forecast, and model-output files commonly have a (time, level, lat, lon) shape. By default the reader emits one row per variable and takes the first (index 0) slice of every non-spatial leading dimension, logging a warning for any dimension with more than one slice.

dimIndex selects a specific slice without expanding the result set:

# Read the slice at time index 2 and level index 1 (one row).
df = (
    spark.read.format("netcdf_gbx")
    .option("variable", "temp")
    .option("dimIndex", "time=2,level=1")
    .load("/Volumes/main/geobrix_samples/netcdf/era5_sample.nc")
)
df.show()

fanout expands one or more dimensions across all their indices, emitting one (source, tile) row per combination:

# Emit one row per (time, level) combination — 3 times × 4 levels = 12 rows.
df = (
    spark.read.format("netcdf_gbx")
    .option("variable", "temp")
    .option("fanout", "time,level")
    .load("/Volumes/main/geobrix_samples/netcdf/era5_sample.nc")
)
df.show()

Both options can be combined: fanout expands the listed dims while dimIndex pins any remaining leading dims. A dim in both options raises a ValueError.

The source column for a specific slice includes a bracketed suffix, NETCDF:"path":var[level=1,time=2], so each row is uniquely identifiable. The tile metadata map carries a sliceDims key (e.g. "level=1,time=2") and, for dims that have coordinate values, a sliceCoord_<dim> key with the coordinate value as a string.

Multi-band stacking (bandDim)

bandDim stacks a leading dimension's slices as bands in one tile, for per-pixel cross-slice math (e.g. anomaly detection, temporal differencing) that needs all time steps in a single raster.

# Stack all 3 time slices as bands in one tile.
df = (
    spark.read.format("netcdf_gbx")
    .option("variable", "temp")
    .option("bandDim", "time")
    .load("/Volumes/main/geobrix_samples/netcdf/era5_sample.nc")
)
df.show()

bandDim composes with dimIndex and fanout on distinct dimensions:

# One 3-band tile per level (time stacked as bands, level fanned out).
df = (
    spark.read.format("netcdf_gbx")
    .option("variable", "temp")
    .option("bandDim", "time")
    .option("fanout", "level")
    .load("/Volumes/main/geobrix_samples/netcdf/era5_sample.nc")
)

Band provenance is recorded in the tile metadata: bandDim (the stacked dimension name), bandCoords (comma-separated coordinate values for each band), and bandDescriptions (GDAL-style descriptions such as "time=0.0,time=1.0,time=2.0"). The tile's source column includes a bandDim=<dim> marker to identify the row.

bandDim vs fanout: use the right tool for the scale

bandDim is for a modest number of slices you intend to work across together in one tile — per-pixel aggregations, temporal differencing, or deriving an index from multiple time steps. Peak decoded RAM ≈ 1 slice, but the full N-band GTiff is held in the executor while downstream processes it.

fanout is memory-safe at scale: each row is one bounded 2-D slice, so any number of time steps can be processed in parallel without accumulating a large multi-band buffer per task.

If bandDim would produce a tile larger than ~256 MiB of decoded data, the reader emits a UserWarning and suggests fanout instead.

Vector mode — points & swath → per-cell points

# Vector mode: swath / point NetCDF -> one point per cell (lossless,
# no regridding). e.g. Sentinel-5P TROPOMI CH4 (netCDF-4 swath); the quality
# flag rides along as its own column so you filter downstream.
df = (spark.read.format("netcdf_gbx")
      .option("mode", "vector")
      .option("group", "/PRODUCT")
      .option("variables", "methane_mixing_ratio_bias_corrected,qa_value")
      .load("/Volumes/main/geobrix_samples/netcdf/s5p_ch4_sample.nc"))
# columns: <vars...>, geom_0 (WKB), geom_0_srid, geom_0_srid_proj

Vector mode emits attribute columns (one per variable) followed by a geom_0 WKB point column and geom_0_srid / geom_0_srid_proj string columns — the same shape as the other lightweight vector readers, so the GeoBrix built-in gbx_st_* functions, native Databricks ST, and H3 binning compose directly. A curvilinear/swath cell carries its own lon/lat, so each becomes an exact point; no target grid or resampling method is chosen for you.

netcdf_gdal — heavyweight raster (RasterX)

netcdf_gdal is the heavyweight (Scala/JVM, native GDAL) raster reader. It reads the CF grid variables of a .nc file into the shared (source, tile) schema — one row per grid variable — so tessellation, band math, and tiling work unchanged. It is read-only. Requires a classic x86 cluster (JAR + GDAL init script); on Serverless / standard / ARM compute use lightweight netcdf_gbx raster mode instead — the two emit the same schema.

# Bare load: one (source, tile) row per readable grid variable.
df = spark.read.format("netcdf_gdal").load("/Volumes/main/geobrix_samples/netcdf/era5_sample.nc")
df.show()

# Optional filter: restrict to specific variables.
df = (spark.read.format("netcdf_gdal")
      .option("variables", "t2m,tp")
      .load("/Volumes/main/geobrix_samples/netcdf/era5_sample.nc"))

Each row's source is the GDAL subdataset selector NETCDF:"{path}":{var}, identifying the originating variable; tile is the standard raster tile structure — see Tile Structure.

CF scale/offset decoding

netcdf_gdal applies CF scale_factor / add_offset automatically, decoding stored integer values to physical floating-point measurements. This matches netcdf_gbx raster mode (which also decodes), so both tiers produce equivalent physical values on a scaled variable — the two are expected to agree within the numeric tolerance of the decode (verified within 1e-4 on regular grids). The decoded output type on the heavyweight path is Float64. On an unscaled variable (no scale_factor / add_offset), both tiers return the native stored type unchanged.

Regular/projected grids only

netcdf_gdal enumerates only subdatasets that are true georeferenced regular or projected grids — specifically, subdatasets with a real CRS or a non-identity geotransform. A swath product such as Sentinel-5P L2 (where geolocation lives in a separate lat/lon array, not in the geotransform) has neither, so it produces zero rows from netcdf_gdal in raster mode. This is the correct behavior: the reader filters out data it cannot meaningfully georeference as a raster. For swath/curvilinear products use lightweight netcdf_gbx vector mode (one point per cell, using the per-pixel lat/lon directly), or netcdf_ogr for native CF-DSG features.

netcdf_ogr — heavyweight vector (VectorX)

netcdf_ogr is the heavyweight (Scala/JVM, native OGR) vector reader. It surfaces a NetCDF file's native CF Discrete Sampling Geometry (DSG) features into the shared vector schema: attribute columns plus a geom_0 WKB column and geom_0_srid / geom_0_srid_proj string columns, the same shape as the other vector readers. It is read-only.

# Bare load: all DSG features. Optional variables= filters attribute columns.
df = spark.read.format("netcdf_ogr").load("/Volumes/main/geobrix_samples/netcdf/dsg_stations.nc")
df.show()

DSG feature files only

netcdf_ogr reads CF-DSG feature layers. A grid-only .nc (no DSG features) has no vector layers, so netcdf_ogr raises at schema inference rather than returning an empty DataFrame — read gridded rasters with netcdf_gdal (or netcdf_gbx raster mode). Flattening a swath to one point per cell is a netcdf_gbx vector-mode capability only; netcdf_ogr does not do that transcoding.

netcdf_gbx writer

netcdf_gbx is also a lightweight writer — the inverse of the reader. df.write.format("netcdf_gbx") writes CF-compliant .nc files from raster tile DataFrames (the default raster mode) or from point DataFrames (vector mode), with singleFile / merge consolidation. See the NetCDF Writer page for the full write options, single-file/merge semantics, and round-trip guarantees.

Next Steps

• GeoTIFF Reader — for COG/GeoTIFF rasters (e.g. Sentinel-2, EMIT CH4).
• Raster Functions — tessellation, band math, tiling.
• Execution Tiers — lightweight vs. heavyweight.

Common functions: used vs excluded

See GBX Common Functions for the full catalog of shared file-access primitives. The table below shows which are active in this reader and which are not, and why.

Common capability	Used here?	How / why
list_local_files (session-free enumeration)	Used	All directory reads — recursive, include_hidden, extensions, path_glob_filter options are routed through this shared predicate.
enumerate_files (FILE-tier enumeration)	Not in the DataSource	The DataSource is session-less on Connect and does not call enumerate_files; FILE-tier enumeration is only available through gbx_file_read at the function layer.
gbx_file_read / gbx_file_write (FILE tier)	Not in the DataSource	The DataSource is FUSE-only (session-less on Connect); FILE reads go through gbx_file_read → rst_fromfile at the function layer.

Shared file-access layer

Lightweight readers use the shared file_gbx file-access base for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there.