Function Reference
RasterX functions run in two execution tiers — lightweight (pure-Python pyrx) and heavyweight (rasterx). As of v0.4.0, every RasterX function is available in both tiers (badged <Tier both/> per function below); per-function notes call out where the lightweight implementation differs. See Choosing an Execution Tier for the comparison and the lightweight install.
Complete reference for all RasterX functions with detailed descriptions, parameters, return values, and examples.
RasterX's lightweight tier is, in large part, distributed rasterio + a best-of-breed raster stack. See Rasterio Distributed for the coverage matrix and honest gaps.
Overview
RasterX is GeoBrix's raster data processing package, providing comprehensive tools for working with raster datasets such as satellite imagery, elevation models, and other gridded spatial data. It is a refactor and improvement of Mosaic raster functions, extended through v0.5.0 with terrain analysis, spectral indices, vector-raster bridging, web-mercator tile output, H3/quadbin/BNG grid aggregations, and — in v0.5.0 — virtual tiles, a COG-preparation lane, and VRT mosaics for memory-safe processing of multi-gigabyte rasters. Since the Databricks product does not (yet) support anything built-in specifically for raster processing, RasterX provides a gap-filling capability for raster operations on the Databricks platform.
Key Features
- GDAL-Powered: Leverages GDAL for robust raster format support
- Distributed Processing: Built on Spark for scalable raster operations
- Multiple Format Support: GeoTIFF, COG, NetCDF, and other GDAL-supported formats
- Metadata Extraction: Comprehensive raster metadata access
- Raster Operations: Clipping, resampling, transformations, map algebra
- Band Operations: Multi-band raster support, single-band extraction
- Terrain Analysis: Slope, aspect, hillshade, TRI, TPI, roughness, color-relief
- Spectral Indices: EVI, SAVI, NDWI, NBR, NDVI, plus a generic dispatcher
- Vector-Raster Bridge: Rasterize geometries, polygonize value regions
- Tile Publishing: Web-mercator XYZ tile generation (PNG / JPEG / WebP)
- Grid Aggregations: H3, CARTO quadbin v0, and BNG cell aggregations
- Virtual Tiles: Bytes-free windowed reads — process multi-gigabyte rasters without materializing them into executor memory
- COG Preparation & VRT Mosaics:
file_gbx+cog_gbxprepare spec-valid Cloud-Optimized GeoTIFFs and portable VRT mosaics for cheap windowed reads
Many RasterX functions accept CRS parameters. Use srid / out_srid (integer EPSG or ESRI code) or crs / out_crs (CRS string — EPSG, ESRI, OGC, WKT, PROJ4) to declare or project CRS. Functions that consume a clip polygon or sample geometry accept clip_crs / crs as a source-CRS fallback. rst_srid returns the stored SRID integer; rst_crs returns the canonical CRS string and always returns a value. See Coordinate reference systems for the full model, the EPSG→ESRI resolution order, and the ESRI/OGC authority-preservation behavior.
Function Categories
RasterX exposes 129 SQL functions (registered as gbx_rst_*; available in Python and Scala as rst_*), organized into the following categories (see rasterx/functions.scala):

- Accessor Functions: Read raster properties and metadata (bounds, dimensions, CRS, bands, pixel size, georeference, format, type, NoData, subdatasets, summary, etc.)
- Aggregator Functions: Combine or merge rasters in group-by (combineavg_agg, derivedband_agg, merge_agg, h3_rasterize_agg)
- Constructor Functions: Create or load rasters from paths, binary content, or bands
- Generator Functions: Produce multiple tiles or bands (h3_tessellate, maketiles, retile, separatebands, tooverlappingtiles)
- Grid Functions (H3): Aggregate raster values to H3 cells (rastertogrid avg/count/max/min/median/sum/variance/stddev)
- Grid Functions (quadbin): Aggregate raster values to CARTO quadbin v0 cells (rastertogrid avg/count/max/min/median/sum/variance/stddev; tessellate; rasterize_agg)
- Grid Functions (BNG): Aggregate raster values to British National Grid cells (rastertogrid avg/count/max/min/median/sum/variance/stddev; tessellate; rasterize_agg)
- Operations: Transform and analyze rasters (clip, transform, merge, asformat, ndvi, filter, convolve, map algebra, coordinate conversion, isEmpty, tryOpen, initNoData, updateType, combineavg, derivedband)
- Web-Mercator Tile Output: Reproject to EPSG:3857 and emit slippy-map XYZ tiles (to_webmercator, tilexyz, xyzpyramid)
- Vector-raster bridge: Burn polygons into rasters and trace contiguous regions back to polygons (rasterize, polygonize)
- Terrain Analysis: DEM-derived surfaces from
gdal.DEMProcessing(slope, aspect, hillshade, TRI, TPI, roughness, color relief) - Spectral Indices: Multi-band satellite math (EVI, SAVI, NDWI, NBR, plus the generic
rst_indexdispatcher)
Tile payload
Every RasterX (heavyweight) function returns a materialized tile whose raster field is a self-contained, in-memory raster (GTiff by default) — safe to serialize between Spark stages and executors, persist to Delta, hand off to rasterio / gdal, or write back out via the gdal writer. The bytes are never an XML reference to a per-executor /vsimem/ tempfile or to a path that only exists on the producing node.
See Tile structure for the full tile-struct schema (the shared cellid / raster / path / window / … struct), and Virtual Tiles for the bytes-free lightweight-tier variant and how tiles move between materialized and virtual.
Functions that internally build via an intermediate VRT — gbx_rst_merge, gbx_rst_merge_agg, gbx_rst_frombands, gbx_rst_combineavg, gbx_rst_combineavg_agg, gbx_rst_derivedband, gbx_rst_derivedband_agg — materialize the result to GTiff before returning, so downstream stages on different executors see real raster bytes. Inspect a tile's payload format from tile.metadata.driver; for any of the functions above, it will read GTiff (not VRT). See Release Notes for the v0.3.0 correctness fix that introduced this invariant. See Tile structure for the full tile-struct schema.
Virtual-tile force-output params
Tile-returning rst_* functions in the lightweight tier accept three optional keyword arguments
that control whether the produced tile carries raster bytes or a bytes-free virtual reference
(a path + window instead of in-memory pixels):
| Argument | Type | Meaning |
|---|---|---|
virtualize_dir | str (durable path) | Write the produced tile to <dir>/[<prefix>_]<name>.tif and return a virtual tile (path + window, no bytes). The directory must be a durable, executor-readable location such as a Unity Catalog Volume path. Cannot be set together with materialize=True. |
virtualize_prefix | str | Optional filename prefix added before the provenance-based filename — use this to deconflict when two different function outputs share the same virtualize_dir. |
materialize | bool | True — ensure the produced tile carries bytes. False — no-op. Default (unset) — auto: reference/passthrough ops (header reads, rst_clip, rst_setsrid, identity rst_transform) belong to the no-new-pixels class and incur no pixel computation; pixel-producing ops (slope, focal, mapalgebra, reproject, merge, combineavg, frombands, …) materialize. |
Default (auto) behavior: reference/passthrough operations (rst_clip, rst_setsrid, identity
rst_transform) produce no new pixels — they describe a region of or annotation on existing backing
data rather than computing fresh values, so they are the low-cost class. Pixel-producing operations
(slope, focal, mapalgebra, reproject, merge, combineavg, frombands, …) materialize and return bytes;
for these, virtualize_dir is the only way to get a virtual tile back — it writes the computed
result to a durable path and returns a light virtual row.
The lightweight tier is for light (virtual) raster tiles; the heavyweight tier is for heavy
(binary) raster tiles. The heavyweight (rasterx) tier does not accept virtualize_dir,
virtualize_prefix, or materialize, and accepts only materialized tiles (raster bytes
present) as input — both v1 and v2 materialized tiles are accepted; the result is always the v2
tile struct. Passing a virtual tile to a heavyweight function raises a clear error directing
you to materialize it first. For most raster work the lightweight tier covers the full function
set and is the recommended default. If you specifically need a heavyweight function, materialize
the tile first (materialize=True, or write via a writer, which always materializes) before
passing it. See Virtual tiles and the light→heavy bridge
for details.
Setup
With the geobrix library already installed (Installation), pick your execution tier and run this once. Both tiers alias the module as rx, so every example below is identical regardless of tier — only this setup differs. (See Choosing an Execution Tier for the comparison.)
The examples on this page read from four temp views, each a raster loaded as a tile column. Point the four path placeholders at your own rasters:
| Placeholder | Sample file | Reader (light / heavy) — Temp view | Backs |
|---|---|---|---|
GTIFF_SAMPLE_DIR | single-band GeoTIFF | gtiff_gbx / gtiff_gdal — rasters | Default — most accessor, tile-ops, transform, and generator examples |
GTIFF_MULTI_DIR | multi-band GeoTIFF (red/NIR/green) | gtiff_gbx / gtiff_gdal — multiband_rasters | band-math and spectral-index examples, rst_numbands, rst_bandmetadata |
DTM_DIR | digital elevation model | gtiff_gbx / gtiff_gdal — dem_rasters | terrain examples (rst_slope, rst_aspect, …) |
NETCDF_DIR | NetCDF with subdatasets | netcdf_gbx / netcdf_gdal — netcdf_rasters | rst_subdatasets, rst_getsubdataset |
Each format has a named reader in both tiers (light *_gbx / heavy *_gdal); the setup for each tier below uses its own. The generic readers (raster_gbx light, gdal heavy) also read any raster their tier's engine supports.
The four *_DIR placeholders in the code below are wired to small sample rasters committed in the GeoBrix repo under src/test/resources/binary/ (resolved by the _SAMPLE_PATHS[...] / *_path() helpers you'll see in the snippet). To run the examples yourself, stage those sample files — or your own rasters — in a Unity Catalog Volume directory and set each placeholder to its Volume path.
The setup below loads GTIFF_SAMPLE_DIR into rasters and then loads the other three the same way — swap the path and view name (and, for NetCDF, the reader). Every example on the page assumes these four views exist.
- Lightweight (pyrx)
- Heavyweight (rasterx)
from databricks.labs.gbx.pyrx import functions as rx
from databricks.labs.gbx.ds.register import register as register_readers
rx.register(spark)
# Register the lightweight Python DataSource readers (raster_gbx / gtiff_gbx /
# netcdf_gbx / cog_gbx). Pure-Python, no JAR — the light-tier counterpart of the
# heavy gdal / gtiff_gdal / netcdf_gdal readers.
register_readers(spark)
GTIFF_SAMPLE_DIR = _SAMPLE_PATHS["gtiff"]
GTIFF_MULTI_DIR = _SAMPLE_PATHS["gtiff_multi"]
DTM_DIR = _SAMPLE_PATHS["dtm"]
NETCDF_DIR = _SAMPLE_PATHS["netcdf"]
def load_tiles(path, reader, view):
df = spark.read.format(reader).load(path)
df.createOrReplaceTempView(view)
return df
# Load the default single-band raster into the `rasters` view:
rasters = load_tiles(GTIFF_SAMPLE_DIR, "gtiff_gbx", "rasters")
# The other three load the same way — GeoTIFFs use "gtiff_gbx", NetCDF uses "netcdf_gbx":
load_tiles(GTIFF_MULTI_DIR, "gtiff_gbx", "multiband_rasters")
load_tiles(DTM_DIR, "gtiff_gbx", "dem_rasters")
load_tiles(NETCDF_DIR, "netcdf_gbx", "netcdf_rasters")
Four temp views created — `rasters` (single-band), `multiband_rasters`,
`dem_rasters`, and `netcdf_rasters` — each a DataFrame with a `tile` column.
Every example on this page reads from one of these views.
from databricks.labs.gbx.rasterx import functions as rx
from ._fixtures import single_band_path, multiband_path, dem_path, netcdf_path
rx.register(spark)
GTIFF_SAMPLE_DIR = str(single_band_path())
GTIFF_MULTI_DIR = str(multiband_path())
DTM_DIR = str(dem_path())
NETCDF_DIR = str(netcdf_path())
def load_tiles(path, reader, view):
df = spark.read.format(reader).load(path)
df.createOrReplaceTempView(view)
return df
# Load the default single-band raster into the `rasters` view:
rasters = load_tiles(GTIFF_SAMPLE_DIR, "gtiff_gdal", "rasters")
# The other three load the same way — GeoTIFFs use "gtiff_gdal", NetCDF uses "netcdf_gdal":
load_tiles(GTIFF_MULTI_DIR, "gtiff_gdal", "multiband_rasters")
load_tiles(DTM_DIR, "gtiff_gdal", "dem_rasters")
load_tiles(NETCDF_DIR, "netcdf_gdal", "netcdf_rasters")
RasterX registered. Four temp views created — `rasters` (single-band),
`multiband_rasters`, `dem_rasters`, and `netcdf_rasters` — each with a `tile`
column. Every example on this page reads from one of these views.
Scala uses this same heavyweight (rasterx) setup: create the four views with spark.read.format("gtiff_gdal").load(...).createOrReplaceTempView("rasters") (and netcdf_gdal for the NetCDF view), then each Scala example reads val df = spark.table("multiband_rasters"). Registration is only needed for the SQL functions — the Scala rx.* Column API works without it.
Prefer pure SQL? Register RasterX (Python: rx.register(spark)), then create the same four views directly so the SQL examples can use FROM rasters, FROM multiband_rasters, and so on:
-- After registering RasterX (Python: rx.register(spark)), create the views:
CREATE OR REPLACE TEMP VIEW rasters AS
SELECT * FROM gdal.`{SAMPLE_RASTER_PATH}`; -- GTIFF_SAMPLE_DIR (single-band)
CREATE OR REPLACE TEMP VIEW multiband_rasters AS
SELECT * FROM gdal.`{MULTIBAND_RASTER_PATH}`; -- GTIFF_MULTI_DIR
CREATE OR REPLACE TEMP VIEW dem_rasters AS
SELECT * FROM gdal.`{DEM_RASTER_PATH}`; -- DTM_DIR
CREATE OR REPLACE TEMP VIEW netcdf_rasters AS
SELECT * FROM netcdf_gdal.`{NETCDF_RASTER_PATH}`; -- NETCDF_DIR
Views `rasters`, `multiband_rasters`, `dem_rasters`, and `netcdf_rasters` created.
Every example on this page reads from one of these views.
Tier availability
As of v0.4.0, all RasterX functions run in both execution tiers — the lightweight pyrx (pure-Python) and heavyweight rasterx tiers share the same rst_* / gbx_rst_* names, and each function below carries a :::note Lightweight tier (pyrx) admonition with its backing library and any behavioral differences. For the heavyweight VRT Python pixel-function configuration (used by gbx_rst_combineavg / gbx_rst_derivedband), see VRT Python pixel functions at the end of this page.
Examples — Conventions
Every function on this page shows one example, expressed identically across four tabs. This section defines the shared setup so each function tab contains only its invocation — nothing more.
Canonical sample files
Each function uses one of four canonical files, chosen by what the function demonstrates:
| File | What it demonstrates | Used by |
|---|---|---|
nyc_sentinel2_red.tif | Single-band GeoTIFF (Sentinel-2 red band, NYC area) | Default — most accessor, tile-ops, transform, and generator functions |
rgb_nir_small.tif | 3-band GeoTIFF with per-band metadata (red, NIR, green — 8×8 px) | Band-math and spectral-index functions (rst_ndvi, rst_evi, …), rst_numbands, rst_bandmetadata |
srtm_n40w073.tif | Digital Elevation Model (SRTM, NYC area) | Terrain functions (rst_slope, rst_aspect, rst_hillshade, …) |
CMIP5 NetCDF (prAdjust_day_…nc) | Multi-variable NetCDF with two subdatasets | rst_subdatasets, rst_getsubdataset only |
The tile-column convention
In every example on this page:
- SQL:
rastersis a temporary view whosetilecolumn holds the canonical sample loaded via the reader.FROM rastersmeans "the sample as tiles." - Python (light and heavy):
dfis a DataFrame with atilecolumn loaded from the canonical file.df.select(...)means "apply this function to the sample tiles." - Scala:
rastersis a DataFrame with the sametilecolumn.rasters.select(...)is identical in intent to the Python form.
Each function's example therefore shows only the invocation — the load is the convention, not the code.
How to read the four tabs
| Tab | Tier | Color |
|---|---|---|
| SQL | Both (default) | — |
| Python (light) | pyrx lightweight tier | — |
| Python (heavy) | rasterx heavyweight tier | Blue badge |
| Scala | rasterx heavyweight tier | Blue badge |
All four tabs show the same operation on the same file. Where a tier's output genuinely differs in form, a short note explains why — for example:
- Geometry-returning functions (
rst_boundingbox,rst_georeference): SQL returns WKT; Python and Scala return a binary column. The note reads...(WKB binary). - Subdataset maps (
rst_subdatasets): SQL returnsmap<string,string>; rendered as{SUBDATASET_1_NAME -> ..., SUBDATASET_1_DESC -> ...}. A note clarifies the key pattern. - Band-metadata maps (
rst_bandmetadata): similarly noted where the rendering differs between tiers.
Scalar values are identical across all tabs — same fixture, same function, same result.
Tile-returning functions (rst_clip, rst_resample, rst_transform, the aggregators, the constructors, …) return a v2 Tile. Their output is shown as a representative struct — {0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}} — rather than a literal byte dump. Tiles loaded via rst_fromcontent / rst_fromfile (as in Setup) are materialized: the raster field holds the encoded bytes and path is null. Virtual tiles — a populated path with lazily-read bytes — arise from the force-output parameters, not from the default load.
SQL naming and arguments: in the SQL tab, RasterX functions carry the gbx_ prefix (e.g. gbx_rst_boundingbox, gbx_rst_width); Python and Scala use the bare rst_* name via rx. In the lightweight tier the registered gbx_rst_* SQL functions require every argument to be passed explicitly — optional defaults are honored only through the Python rx.* API. See Language Bindings for more.
Per-function notes
Functions that use a non-default fixture (multiband GeoTIFF, DEM, or NetCDF) carry a one-line note immediately before their code example flagging the file. Functions that produce a tile rather than consuming one (constructors: rst_fromfile, rst_fromcontent, rst_frombands, …) show a fuller load/build example with a note — the bare-invocation model does not apply when there is no input tile yet.
Accessor Functions
Functions to read raster properties and metadata (29 total).
rst_avg
LightweightHeavyweightPowered by rasterio + NumPy. Per-band mean over valid (non-NoData) pixels.
Signature: rst_avg(tile: Column): Column — Per-band average pixel values.
Returns NULL for a band with zero valid pixels (all NoData) on both tiers.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands: red, NIR, green); the canonical single-band sentinel2 tile is all-NoData for this function.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_avg(tile) AS band_averages FROM multiband_rasters;
+-----------------------------+
|band_averages |
+-----------------------------+
|[83.59375, 153.125, 114.3125]|
+-----------------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_avg("tile").alias("band_averages")).first()
+------------------------------------+
|band_averages |
+------------------------------------+
|[83.59375, 153.125, 114.3125] |
+------------------------------------+
from databricks.labs.gbx.rasterx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_avg("tile").alias("band_averages")).first()
+------------------------------------+
|band_averages |
+------------------------------------+
|[83.59375, 153.125, 114.3125] |
+------------------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Reads the multiband_rasters Setup view (rgb_nir_small.tif, 3 bands);
// the single-band sentinel2 tile is all-NoData for this function.
val df = spark.table("multiband_rasters")
val result = df.select(rx.rst_avg(col("tile")).alias("band_averages"))
result.show()
+------------------------------------+
|band_averages |
+------------------------------------+
|[83.59375, 153.125, 114.3125] |
+------------------------------------+
rst_bandmetadata
LightweightHeavyweightPowered by rasterio.
Signature: rst_bandmetadata(tile: Column, band: Column): Column — Band metadata map.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands with per-band GDAL metadata tags); plain single-band GeoTIFFs without tags return an empty map {}.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_bandmetadata(tile, 1) AS band_meta FROM multiband_rasters;
+----------------------------------------------+
|band_meta |
+----------------------------------------------+
|{name -> red, wavelength_nm -> 665, band_in...|
+----------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
result = df.select(rx.rst_bandmetadata("tile", f.lit(1)).alias("band_meta")).first()
+----------------------------------------------+
|band_meta |
+----------------------------------------------+
|{name -> red, wavelength_nm -> 665, band_in...|
+----------------------------------------------+
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("multiband_rasters")
result = tile_df.select(
rx.rst_bandmetadata("tile", f.lit(1)).alias("band_meta")
).first()
+----------------------------------------------+
|band_meta |
+----------------------------------------------+
|{name -> red, wavelength_nm -> 665, band_in...|
+----------------------------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif) which carries per-band GDAL metadata tags.
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_bandmetadata(col("tile"), lit(1)).alias("band_meta"))
result.show(truncate = false)
+----------------------------------------------+
|band_meta |
+----------------------------------------------+
|{name -> red, wavelength_nm -> 665, band_in...|
+----------------------------------------------+
rst_boundingbox
LightweightHeavyweightPowered by rasterio.
Signature: rst_boundingbox(tile: Column): Column — Bounding box geometry.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT path, gbx_rst_boundingbox(tile) as bbox FROM rasters;
+--------------------+-----------------+
|path |bbox |
+--------------------+-----------------+
|.../nyc_sentinel2...|POLYGON ((-74....|
+--------------------+-----------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_boundingbox("tile").alias("bbox")).first()
+----+
|bbox|
+----+
|[...|
+----+
(WKB binary — bounding POLYGON of the raster extent in EPSG:32618)
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_boundingbox("tile").alias("bbox")).first()
+----+
|bbox|
+----+
|[...|
+----+
(WKB binary — bounding POLYGON of the raster extent in EPSG:32618)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_boundingbox(col("tile")).alias("bbox"))
result.show(truncate = false)
+----+
|bbox|
+----+
|[...|
+----+
(WKB binary — bounding POLYGON of the raster extent in EPSG:32618)
rst_crs
LightweightHeavyweightPowered by rasterio + pyproj.
Signature: rst_crs(tile: Column): Column — the tile's CRS as a string (authority string like EPSG:4326 / ESRI:54008, else WKT); always returns a value, including for non-EPSG rasters where rst_srid is NULL. See Coordinate Reference Systems.
Returns the authority string (AUTHORITY:CODE, e.g. EPSG:4326 or ESRI:54008) when the CRS has one, otherwise its WKT. Unlike rst_srid (which returns the integer EPSG code, or NULL/0 for a CRS with no EPSG code), rst_crs always returns a value — including for non-EPSG rasters defined by an ESRI code, WKT, or PROJ4 string.
rst_sridreturns the integer EPSG code (e.g.4326);NULL(lightweight) or0(heavyweight) when the CRS has no EPSG code. Use it when you need the numeric SRID for the native ST bridge or an EPSG-only workflow.rst_crsreturns the CRS string and never loses a non-EPSG CRS.rst_setcrs/rst_transformcrstake a CRS string. An int-castable string is treated as an EPSG SRID ('4326'behaves like SRID4326); otherwise the string is parsed as an authority code (EPSG:/ESRI:), WKT, or PROJ4. This is how ESRI codes, WKT, and PROJ4 definitions survive a round trip.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_crs(tile) AS crs FROM rasters;
+----------+
|crs |
+----------+
|EPSG:32618|
+----------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_crs("tile").alias("crs")).first()
+----------+
|crs |
+----------+
|EPSG:32618|
+----------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_crs("tile").alias("crs")).first()
+----------+
|crs |
+----------+
|EPSG:32618|
+----------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_crs(col("tile")).alias("crs"))
result.show(truncate = false)
+----------+
|crs |
+----------+
|EPSG:32618|
+----------+
rst_format
LightweightHeavyweightPowered by rasterio.
Signature: rst_format(tile: Column): Column — GDAL format name.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_format(tile) AS format FROM rasters;
+------+
|format|
+------+
|GTiff |
+------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_format("tile").alias("format")).first()
+------+
|format|
+------+
|GTiff |
+------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_format("tile").alias("format")).first()
+------+
|format|
+------+
|GTiff |
+------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_format(col("tile")).alias("format"))
result.show()
+------+
|format|
+------+
|GTiff |
+------+
rst_georeference
LightweightHeavyweightPowered by rasterio.
Signature: rst_georeference(tile: Column): Column — Georeference parameters as a map.
The result is a MapType with the following keys, corresponding to GDAL's 6-element geotransform:
| Key | Geotransform index | Meaning |
|---|---|---|
upperLeftX | GT(0) | X of the upper-left corner of the upper-left pixel |
upperLeftY | GT(3) | Y of the upper-left corner of the upper-left pixel |
scaleX | GT(1) | Pixel width (west–east resolution) |
scaleY | GT(5) | Pixel height (north–south resolution; often negative for north-up) |
skewX | GT(2) | Row rotation (typically 0) |
skewY | GT(4) | Column rotation (typically 0) |
See the GDAL geotransform tutorial and raster data model for details.
Since v0.2.0- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_georeference(tile) AS georeference FROM rasters;
+-------------------------------------------------------------+
|georeference |
+-------------------------------------------------------------+
|{scaleX -> 10.0, scaleY -> -10.0, upperLeftX -> 2121950.0,...|
+-------------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_georeference("tile").alias("georeference")).first()
+--------------------------------------------------------------+
|georeference |
+--------------------------------------------------------------+
|{scaleX -> 10.0, scaleY -> -10.0, upperLeftX -> 2121950.0,... |
+--------------------------------------------------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_georeference("tile").alias("georeference")).first()
+--------------------------------------------------------------+
|georeference |
+--------------------------------------------------------------+
|{scaleX -> 10.0, scaleY -> -10.0, upperLeftX -> 2121950.0,... |
+--------------------------------------------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_georeference(col("tile")).alias("georeference"))
result.show(truncate = false)
+--------------------------------------------------------------+
|georeference |
+--------------------------------------------------------------+
|{scaleX -> 10.0, scaleY -> -10.0, upperLeftX -> 2121950.0,... |
+--------------------------------------------------------------+
rst_getnodata
LightweightHeavyweightPowered by rasterio. Returns the dataset NoData value repeated once per band.
Signature: rst_getnodata(tile: Column): Column — NoData values per band.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_getnodata(tile) AS nodata FROM rasters;
+------+
|nodata|
+------+
|[0.0] |
+------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_getnodata("tile").alias("nodata")).first()
+--------+
|nodata |
+--------+
|[0.0] |
+--------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_getnodata("tile").alias("nodata")).first()
+--------+
|nodata |
+--------+
|[0.0] |
+--------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_getnodata(col("tile")).alias("nodata"))
result.show()
+--------+
|nodata |
+--------+
|[0.0] |
+--------+
rst_getsubdataset
LightweightHeavyweightPowered by rasterio. Subdataset availability depends on rasterio's bundled GDAL driver set.
Signature: rst_getsubdataset(tile: Column, subsetName: Column): Column — Extract subdataset.
Examples use the CMIP5 NetCDF fixture (prAdjust_day_HadGEM2-CC_*.nc) which has two subdatasets: time_bnds and prAdjust. Subdatasets require a multi-layer format such as NetCDF; plain GeoTIFFs return no subdatasets.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- netcdf_rasters view is from the CMIP5 NetCDF fixture (has time_bnds and prAdjust)
SELECT gbx_rst_getsubdataset(tile, 'prAdjust') AS subdataset FROM netcdf_rasters;
+-----------------------------------------------------------+
|subdataset |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(the extracted prAdjust subdataset as a tile — 720x360, 31 bands)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("netcdf_rasters")
result = df.select(
rx.rst_getsubdataset("tile", f.lit("prAdjust")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(light tier returns a materialized v2 Tile; extracted prAdjust subdataset — 720 pixels wide, 31 bands, 360 rows)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
nc_df = spark.table("netcdf_rasters")
result = nc_df.select(
rx.rst_getsubdataset("tile", f.lit("prAdjust")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(extracted prAdjust subdataset — 720 pixels wide, 31 bands, 360 rows)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses committed CMIP5 NetCDF fixture (has time_bnds and prAdjust subdatasets).
// Subdatasets require a multi-layer format such as NetCDF.
// rst_width wraps the result to return a real scalar proving extraction.
val rasters = spark.table("netcdf_rasters")
val result = rasters.select(
rx.rst_width(rx.rst_getsubdataset(col("tile"), lit("prAdjust"))).alias("width")
)
result.show()
+-----+
|width|
+-----+
| 720|
+-----+
(width of the extracted prAdjust subdataset — 720 pixels, 31 bands, 360 rows)
rst_height
LightweightHeavyweightPowered by rasterio.
Signature: rst_height(tile: Column): Column — Height in pixels.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_height(tile) AS height FROM rasters;
+------+
|height|
+------+
|161 |
+------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_height("tile").alias("height")).first()
+------+
|height|
+------+
|161 |
+------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_height("tile").alias("height")).first()
+------+
|height|
+------+
|161 |
+------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_height(col("tile")).alias("height"))
result.show()
+------+
|height|
+------+
|161 |
+------+
rst_max
LightweightHeavyweightPowered by rasterio + NumPy. Per-band maximum over valid (non-NoData) pixels.
Signature: rst_max(tile: Column): Column — Maximum pixel values per band.
Returns NULL for a band with zero valid pixels (all NoData) on both tiers.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands); the canonical single-band sentinel2 tile is all-NoData for this function.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_max(tile) AS band_max FROM multiband_rasters;
+---------------------+
|band_max |
+---------------------+
|[119.0, 197.0, 148.0]|
+---------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_max("tile").alias("band_max")).first()
+---------------------+
|band_max |
+---------------------+
|[119.0, 197.0, 148.0]|
+---------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_max("tile").alias("band_max")).first()
+---------------------+
|band_max |
+---------------------+
|[119.0, 197.0, 148.0]|
+---------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands); single-band sentinel2 is all-NoData.
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_max(col("tile")).alias("band_max"))
result.show()
+---------------------+
|band_max |
+---------------------+
|[119.0, 197.0, 148.0]|
+---------------------+
rst_median
LightweightHeavyweightPowered by rasterio + NumPy. Per-band median over valid (non-NoData) pixels.
Signature: rst_median(tile: Column): Column — Median pixel values per band.
Returns NULL for a band with zero valid pixels (all NoData) on both tiers.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands); the canonical single-band sentinel2 tile is all-NoData for this function.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_median(tile) AS band_median FROM multiband_rasters;
+--------------------+
|band_median |
+--------------------+
|[85.0, 157.5, 111.5]|
+--------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_median("tile").alias("band_median")).first()
+---------------------+
|band_median |
+---------------------+
|[85.0, 157.5, 111.5] |
+---------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_median("tile").alias("band_median")).first()
+---------------------+
|band_median |
+---------------------+
|[85.0, 157.5, 111.5] |
+---------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands); single-band sentinel2 is all-NoData.
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_median(col("tile")).alias("band_median"))
result.show()
+---------------------+
|band_median |
+---------------------+
|[85.0, 157.5, 111.5] |
+---------------------+
rst_memsize
LightweightHeavyweightPowered by rasterio. Returns the serialized raster size in bytes.
Signature: rst_memsize(tile: Column): Column — In-memory size in bytes.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_memsize(tile) AS memsize FROM rasters;
+-------+
|memsize|
+-------+
|71749 |
+-------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_memsize("tile").alias("memsize")).first()
+-------+
|memsize|
+-------+
|71749 |
+-------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_memsize("tile").alias("memsize")).first()
+-------+
|memsize|
+-------+
|71749 |
+-------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_memsize(col("tile")).alias("memsize"))
result.show()
+-------+
|memsize|
+-------+
|71749 |
+-------+
rst_metadata
LightweightHeavyweightPowered by rasterio.
Signature: rst_metadata(tile: Column): Column — Metadata map.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_metadata(tile) as metadata FROM rasters;
+--------------------------------------------------+
|metadata |
+--------------------------------------------------+
|{driver -> GTiff, crs -> EPSG:32618, count -> 1,..|
+--------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_metadata("tile").alias("metadata")).first()
+--------------------------------------------------+
|metadata |
+--------------------------------------------------+
|{driver -> GTiff, crs -> EPSG:32618, count -> 1,..|
+--------------------------------------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_metadata("tile").alias("metadata")).first()
+--------------------------------------------------+
|metadata |
+--------------------------------------------------+
|{driver -> GTiff, crs -> EPSG:32618, count -> 1,..|
+--------------------------------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_metadata(col("tile")).alias("metadata"))
result.show(truncate = false)
+--------------------------------------------------+
|metadata |
+--------------------------------------------------+
|{driver -> GTiff, crs -> EPSG:32618, count -> 1,..|
+--------------------------------------------------+
rst_min
LightweightHeavyweightPowered by rasterio + NumPy. Per-band minimum over valid (non-NoData) pixels.
Signature: rst_min(tile: Column): Column — Minimum pixel values per band.
Returns NULL for a band with zero valid pixels (all NoData) on both tiers.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands); the canonical single-band sentinel2 tile is all-NoData for this function.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_min(tile) AS band_min FROM multiband_rasters;
+-------------------+
|band_min |
+-------------------+
|[50.0, 102.0, 82.0]|
+-------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_min("tile").alias("band_min")).first()
+-------------------+
|band_min |
+-------------------+
|[50.0, 102.0, 82.0]|
+-------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_min("tile").alias("band_min")).first()
+-------------------+
|band_min |
+-------------------+
|[50.0, 102.0, 82.0]|
+-------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands); single-band sentinel2 is all-NoData.
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_min(col("tile")).alias("band_min"))
result.show()
+-------------------+
|band_min |
+-------------------+
|[50.0, 102.0, 82.0]|
+-------------------+
rst_numbands
LightweightHeavyweightPowered by rasterio.
Signature: rst_numbands(tile: Column): Column — Number of bands.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands: red, NIR, green).
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_numbands(tile) AS num_bands FROM multiband_rasters;
+---------+
|num_bands|
+---------+
|3 |
+---------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_numbands("tile").alias("num_bands")).first()
+---------+
|num_bands|
+---------+
|3 |
+---------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_numbands("tile").alias("num_bands")).first()
+---------+
|num_bands|
+---------+
|3 |
+---------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands: red, NIR, green).
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_numbands(col("tile")).alias("num_bands"))
result.show()
+---------+
|num_bands|
+---------+
|3 |
+---------+
rst_pixelcount
LightweightHeavyweightPowered by rasterio + NumPy. Per-band count of valid (non-NoData) pixels.
Signature: rst_pixelcount(tile: Column): Column — Total pixel count.
Examples use the multiband fixture (rgb_nir_small.tif, 8×8, 3 bands, no NoData set), yielding 64 valid pixels per band; the canonical single-band sentinel2 tile has NoData=0 with all pixels equal zero, returning [0].
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_pixelcount(tile) AS pixel_count FROM multiband_rasters;
+------------+
|pixel_count |
+------------+
|[64, 64, 64]|
+------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_pixelcount("tile").alias("pixel_count")).first()
+------------+
|pixel_count |
+------------+
|[64, 64, 64]|
+------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_pixelcount("tile").alias("pixel_count")).first()
+------------+
|pixel_count |
+------------+
|[64, 64, 64]|
+------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 8x8, no NoData); single-band sentinel2 returns [0].
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_pixelcount(col("tile")).alias("pixel_count"))
result.show()
+------------+
|pixel_count |
+------------+
|[64, 64, 64]|
+------------+
rst_pixelheight
LightweightHeavyweightPowered by rasterio.
Signature: rst_pixelheight(tile: Column): Column — Pixel height in ground units.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_pixelheight(tile) AS pixel_height FROM rasters;
+------------+
|pixel_height|
+------------+
|10.0 |
+------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_pixelheight("tile").alias("pixel_height")).first()
+------------+
|pixel_height|
+------------+
|10.0 |
+------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_pixelheight("tile").alias("pixel_height")).first()
+------------+
|pixel_height|
+------------+
|10.0 |
+------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_pixelheight(col("tile")).alias("pixel_height"))
result.show()
+------------+
|pixel_height|
+------------+
|10.0 |
+------------+
rst_pixelwidth
LightweightHeavyweightPowered by rasterio.
Signature: rst_pixelwidth(tile: Column): Column — Pixel width in ground units.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_pixelwidth(tile) AS pixel_width FROM rasters;
+-----------+
|pixel_width|
+-----------+
|10.0 |
+-----------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_pixelwidth("tile").alias("pixel_width")).first()
+-----------+
|pixel_width|
+-----------+
|10.0 |
+-----------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_pixelwidth("tile").alias("pixel_width")).first()
+-----------+
|pixel_width|
+-----------+
|10.0 |
+-----------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_pixelwidth(col("tile")).alias("pixel_width"))
result.show()
+-----------+
|pixel_width|
+-----------+
|10.0 |
+-----------+
rst_rotation
LightweightHeavyweightPowered by rasterio.
Signature: rst_rotation(tile: Column): Column — Rotation in radians.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_rotation(tile) AS rotation FROM rasters;
+--------+
|rotation|
+--------+
|0.0 |
+--------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_rotation("tile").alias("rotation")).first()
+--------+
|rotation|
+--------+
|0.0 |
+--------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_rotation("tile").alias("rotation")).first()
+--------+
|rotation|
+--------+
|0.0 |
+--------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_rotation(col("tile")).alias("rotation"))
result.show()
+--------+
|rotation|
+--------+
|0.0 |
+--------+
rst_scalex / rst_scaley
LightweightHeavyweightPowered by rasterio.
Signature: rst_scalex(tile: Column): Column, rst_scaley(tile: Column): Column — Scale (pixel size) in X/Y.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_scalex(tile) AS scale_x FROM rasters;
+-------+
|scale_x|
+-------+
|10.0 |
+-------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_scalex("tile").alias("scale_x")).first()
+-------+
|scale_x|
+-------+
|10.0 |
+-------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_scalex("tile").alias("scale_x")).first()
+-------+
|scale_x|
+-------+
|10.0 |
+-------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_scalex(col("tile")).alias("scale_x"))
result.show()
+-------+
|scale_x|
+-------+
|10.0 |
+-------+
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_scaley(tile) AS scale_y FROM rasters;
+-------+
|scale_y|
+-------+
|-10.0 |
+-------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_scaley("tile").alias("scale_y")).first()
+-------+
|scale_y|
+-------+
|-10.0 |
+-------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_scaley("tile").alias("scale_y")).first()
+-------+
|scale_y|
+-------+
|-10.0 |
+-------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_scaley(col("tile")).alias("scale_y"))
result.show()
+-------+
|scale_y|
+-------+
|-10.0 |
+-------+
rst_skewx / rst_skewy
LightweightHeavyweightPowered by rasterio.
Signature: rst_skewx(tile: Column): Column, rst_skewy(tile: Column): Column — Skew in X/Y.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_skewx(tile) AS skew_x FROM rasters;
+------+
|skew_x|
+------+
|0.0 |
+------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_skewx("tile").alias("skew_x")).first()
+------+
|skew_x|
+------+
|0.0 |
+------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_skewx("tile").alias("skew_x")).first()
+------+
|skew_x|
+------+
|0.0 |
+------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_skewx(col("tile")).alias("skew_x"))
result.show()
+------+
|skew_x|
+------+
|0.0 |
+------+
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_skewy(tile) AS skew_y FROM rasters;
+------+
|skew_y|
+------+
|0.0 |
+------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_skewy("tile").alias("skew_y")).first()
+------+
|skew_y|
+------+
|0.0 |
+------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_skewy("tile").alias("skew_y")).first()
+------+
|skew_y|
+------+
|0.0 |
+------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_skewy(col("tile")).alias("skew_y"))
result.show()
+------+
|skew_y|
+------+
|0.0 |
+------+
rst_srid
LightweightHeavyweightPowered by rasterio.
Signature: rst_srid(tile: Column): Column — the stored spatial reference ID integer (an EPSG or ESRI code), or NULL when the tile has none. Returns the code as stored; it is classified only when applied. See Coordinate Reference Systems for the SRID-vs-CRS-string model and the epsg→esri resolution rule.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_srid(tile) AS srid FROM rasters;
+-----+
|srid |
+-----+
|32618|
+-----+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_srid("tile").alias("srid")).first()
+-----+
|srid |
+-----+
|32618|
+-----+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_srid("tile").alias("srid")).first()
+-----+
|srid |
+-----+
|32618|
+-----+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_srid(col("tile")).alias("srid"))
result.show()
+-----+
|srid |
+-----+
|32618|
+-----+
rst_subdatasets
LightweightHeavyweightPowered by rasterio. Empty for single-dataset rasters (e.g. a plain GeoTIFF).
Signature: rst_subdatasets(tile: Column): Column — List of subdataset names.
Examples use the CMIP5 NetCDF fixture (prAdjust_day_HadGEM2-CC_*.nc) which has two subdatasets: time_bnds and prAdjust. Subdatasets require a multi-layer format; plain GeoTIFFs return an empty map.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- netcdf_rasters view is from the CMIP5 NetCDF fixture (has time_bnds and prAdjust)
SELECT gbx_rst_subdatasets(tile) AS subdatasets FROM netcdf_rasters;
+------------------------------------------------------+
|subdatasets |
+------------------------------------------------------+
|{SUBDATASET_1_NAME -> ..., SUBDATASET_1_DESC -> [31...|
+------------------------------------------------------+
(map with SUBDATASET_1_NAME/DESC for time_bnds and SUBDATASET_2_NAME/DESC for prAdjust)
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("netcdf_rasters")
result = df.select(rx.rst_subdatasets("tile").alias("subdatasets")).first()
+------------------------------------------------------+
|subdatasets |
+------------------------------------------------------+
|{SUBDATASET_1_NAME -> ..., SUBDATASET_1_DESC -> [31...|
+------------------------------------------------------+
(map with SUBDATASET_1_NAME/DESC for time_bnds and SUBDATASET_2_NAME/DESC for prAdjust)
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("netcdf_rasters")
result = tile_df.select(rx.rst_subdatasets("tile").alias("subdatasets")).first()
+------------------------------------------------------+
|subdatasets |
+------------------------------------------------------+
|{SUBDATASET_1_NAME -> ..., SUBDATASET_1_DESC -> [31...|
+------------------------------------------------------+
(map with SUBDATASET_1_NAME/DESC for time_bnds and SUBDATASET_2_NAME/DESC for prAdjust)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses committed CMIP5 NetCDF fixture (has time_bnds and prAdjust subdatasets).
val rasters = spark.table("netcdf_rasters")
val result = rasters.select(rx.rst_subdatasets(col("tile")).alias("subdatasets"))
result.show(truncate = false)
+------------------------------------------------------+
|subdatasets |
+------------------------------------------------------+
|{SUBDATASET_1_NAME -> ..., SUBDATASET_1_DESC -> [31...|
+------------------------------------------------------+
(map with SUBDATASET_1_NAME/DESC for time_bnds and SUBDATASET_2_NAME/DESC for prAdjust)
rst_summary
LightweightHeavyweightPowered by rasterio + NumPy. Returns a JSON summary (driver, size, CRS, geotransform, per-band statistics); the lightweight JSON shape differs from the heavyweight gdalinfo -json output.
Signature: rst_summary(tile: Column): Column — Statistical summary of values.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands) which has real pixel data.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_summary(tile) AS summary FROM multiband_rasters;
+------------------------------------------------------------+
|summary |
+------------------------------------------------------------+
|{"driverShortName": "GTiff", "size": [8, 8], "coordinateS...|
+------------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_summary("tile").alias("summary")).first()
+------------------------------------------------------------+
|summary |
+------------------------------------------------------------+
|{"driverShortName": "GTiff", "size": [8, 8], "coordinateS...|
+------------------------------------------------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_summary("tile").alias("summary")).first()
+------------------------------------------------------------+
|summary |
+------------------------------------------------------------+
|{"driverShortName": "GTiff", "size": [8, 8], "coordinateS...|
+------------------------------------------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands) which has real pixel data.
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_summary(col("tile")).alias("summary"))
result.show(truncate = false)
+------------------------------------------------------------+
|summary |
+------------------------------------------------------------+
|{"driverShortName": "GTiff", "size": [8, 8], "coordinateS...|
+------------------------------------------------------------+
rst_type
LightweightHeavyweightPowered by rasterio.
Signature: rst_type(tile: Column): Column — Data type per band.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands, UInt16).
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_type(tile) AS band_types FROM multiband_rasters;
+------------------------+
|band_types |
+------------------------+
|[UInt16, UInt16, UInt16]|
+------------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_type("tile").alias("band_types")).first()
+-----------------------------+
|band_types |
+-----------------------------+
|[UInt16, UInt16, UInt16] |
+-----------------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_type("tile").alias("band_types")).first()
+-----------------------------+
|band_types |
+-----------------------------+
|[UInt16, UInt16, UInt16] |
+-----------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands, UInt16).
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_type(col("tile")).alias("band_types"))
result.show()
+-----------------------------+
|band_types |
+-----------------------------+
|[UInt16, UInt16, UInt16] |
+-----------------------------+
rst_upperleftx / rst_upperlefty
LightweightHeavyweightPowered by rasterio.
Signature: rst_upperleftx(tile: Column): Column, rst_upperlefty(tile: Column): Column — Upper-left corner coordinates.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_upperleftx(tile) AS upper_left_x FROM rasters;
+------------+
|upper_left_x|
+------------+
|2121950.0 |
+------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_upperleftx("tile").alias("upper_left_x")).first()
+------------+
|upper_left_x|
+------------+
|2121950.0 |
+------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_upperleftx("tile").alias("upper_left_x")).first()
+------------+
|upper_left_x|
+------------+
|2121950.0 |
+------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_upperleftx(col("tile")).alias("upper_left_x"))
result.show()
+------------+
|upper_left_x|
+------------+
|2121950.0 |
+------------+
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_upperlefty(tile) AS upper_left_y FROM rasters;
+------------+
|upper_left_y|
+------------+
|-10790470.0 |
+------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_upperlefty("tile").alias("upper_left_y")).first()
+---------------+
|upper_left_y |
+---------------+
|-10790470.0 |
+---------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_upperlefty("tile").alias("upper_left_y")).first()
+---------------+
|upper_left_y |
+---------------+
|-10790470.0 |
+---------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_upperlefty(col("tile")).alias("upper_left_y"))
result.show()
+---------------+
|upper_left_y |
+---------------+
|-10790470.0 |
+---------------+
rst_width
LightweightHeavyweightPowered by rasterio.
Signature: rst_width(tile: Column): Column — Width in pixels.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_width(tile) AS width FROM rasters;
+-----+
|width|
+-----+
|236 |
+-----+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_width("tile").alias("width")).first()
+-----+
|width|
+-----+
|236 |
+-----+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_width("tile").alias("width")).first()
+-----+
|width|
+-----+
|236 |
+-----+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_width(col("tile")).alias("width"))
result.show()
+-----+
|width|
+-----+
|236 |
+-----+
Aggregator Functions
Combine or merge rasters in group-by (7 total).
rst_bng_rasterize_agg
LightweightHeavyweight Grouped-agg UDFThe lightweight implementation is backed by pygx._bng cell math and rasterio. BNG cell IDs (STRING) are parsed via pygx._bng to EPSG:27700-native cell geometries and burned into the output band — the output canvas is EPSG:27700 throughout with no warp step (this aggregator takes cellid+value rows, not a raster tile).
Pixels not covered by any geometry in the group are set to -9999.0 (band-registered NoData). Filter or mask downstream via gbx_rst_getnodata or IS NULL on the extracted band value.
Heavyweight gbx_rst_bng_rasterize_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_bng_rasterize_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
cell_id AS cellid,
gbx_rst_fromcontent(gbx_rst_bng_rasterize_agg(cellid, burn_value), 'GTiff') AS tile
FROM bng_cell_values
GROUP BY cell_id
Streaming aggregator that burns BNG cell geometry/value pairs (one row per cell) into a single rasterized tile per group. The input raster is automatically reprojected to EPSG:27700 (British National Grid) before rasterization. BNG cell IDs are STRING. The inverse of rst_bng_rastertogrid*: where those functions reduce raster pixels to per-cell statistics, this one synthesizes a raster from per-cell values.
out_crs parameter (no-op for BNG)The lightweight Python binding accepts an optional trailing out_crs (string CRS) argument that the heavyweight/Scala tier does not — but for BNG it is a no-op: the output is always EPSG:27700. Both out_srid and out_crs are ignored here (they exist only for signature parity with the H3/quadbin aggregators).
Signature: rst_bng_rasterize_agg(cellid: Column, value: Column, out_srid: Column, pixel_size: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, mode: Column, kring_pad: Column): Column
Parameters: cellid — BNG cell ID (STRING); value — numeric burn value; out_srid — EPSG code for the output CRS (typically 27700); pixel_size — output raster cell size in metres (usually 1.0 for 1km BNG cells); xmin/ymin/xmax/ymax — output extent in EPSG:27700 metres; width/height — output raster dimensions in pixels; mode — aggregation mode for overlapping values (typically "last"); kring_pad — cell neighbourhood expansion (typically 0)
Multi-row fixture: 3 BNG 1km STRING cell rows near central London (EPSG:27700) with burn values 1.0/2.0/3.0. All tabs use the same grouped-agg rasterize invocation.
Since v0.4.3- SQL
- Python (light)
- Python (heavy)
- Scala
-- Rasterize BNG cells into one raster tile per region. cellid is a STRING
-- (e.g. 'TQ38SW'); the srid argument is a no-op (BNG always forces EPSG:27700,
-- pass 27700 for clarity). The extent auto-derives from the cell set (null
-- canvas args).
SELECT region_id,
gbx_rst_bng_rasterize_agg(
cellid, burn_value,
27700, cast(null as double),
cast(null as double), cast(null as double),
cast(null as double), cast(null as double),
cast(null as int), cast(null as int),
'centroids', cast(0 as int)
) AS tile
FROM bng_cell_values
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|tile |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(tile, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|tile |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
from databricks.labs.gbx.gridx.bng import functions as bngx
bngx.register(spark)
spark.sql("""
CREATE OR REPLACE TEMP VIEW _bng_cells AS
SELECT region,
gbx_bng_eastnorthasbng(e, n, 3) AS cellid,
val AS value
FROM (VALUES
('R1', cast(530000.0 as double), cast(180000.0 as double), cast(1.0 as double)),
('R1', cast(531000.0 as double), cast(181000.0 as double), cast(2.0 as double)),
('R1', cast(529000.0 as double), cast(179000.0 as double), cast(3.0 as double))
) AS t(region, e, n, val)
""")
df = spark.table("_bng_cells")
result = (
df.groupBy("region")
.agg(rx.rst_bng_rasterize_agg("cellid", "value").alias("tile"))
.first()
)
spark.catalog.dropTempView("_bng_cells")
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.gridx.bng import functions as bngx
bngx.register(spark)
spark.sql("""
CREATE OR REPLACE TEMP VIEW _bng_cells_heavy AS
SELECT region,
gbx_bng_eastnorthasbng(e, n, 3) AS cellid,
val AS value
FROM (VALUES
('R1', cast(530000.0 as double), cast(180000.0 as double), cast(1.0 as double)),
('R1', cast(531000.0 as double), cast(181000.0 as double), cast(2.0 as double)),
('R1', cast(529000.0 as double), cast(179000.0 as double), cast(3.0 as double))
) AS t(region, e, n, val)
""")
df = spark.table("_bng_cells_heavy")
# cellid + value are the only required args; the extent/size/mode/kring
# options default to the same values the lightweight tier uses (BNG forces
# EPSG:27700, so out_srid is a no-op).
result = (
df.groupBy("region")
.agg(rx.rst_bng_rasterize_agg(df["cellid"], df["value"]).alias("tile"))
.first()
)
spark.catalog.dropTempView("_bng_cells_heavy")
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import com.databricks.labs.gbx.gridx.bng.{functions => bng}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
rx.register(spark)
bng.register(spark)
// Multi-row fixture: 3 BNG 1km cells near central London (EPSG:27700).
val schema = StructType(Seq(StructField("region", StringType()), StructField("e", DoubleType()), StructField("n", DoubleType()), StructField("val", DoubleType())))
val rows = Seq(("R1",530000.0,180000.0,1.0),("R1",531000.0,181000.0,2.0),("R1",529000.0,179000.0,3.0))
val raw = spark.createDataFrame(spark.sparkContext.parallelize(rows.map(r => org.apache.spark.sql.Row(r._1,r._2,r._3,r._4))), schema)
raw.createOrReplaceTempView("_bng_src")
val df = spark.sql("SELECT region, gbx_bng_eastnorthasbng(e, n, 3) AS cellid, val AS value FROM _bng_src")
val result = df.groupBy("region").agg(rx.rst_bng_rasterize_agg(col("cellid"), col("value")).alias("tile"))
result.show()
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_combineavg_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio + NumPy. Aggregate — groupBy(...).agg(rx.rst_combineavg_agg("tile")) returns the NoData-aware per-pixel mean as one tile per group; input tiles must share the same grid (shape/extent/CRS).
Heavyweight gbx_rst_combineavg_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_combineavg_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(gbx_rst_combineavg_agg(tile), 'GTiff') AS tile
FROM tiles
GROUP BY group_key
Signature: rst_combineavg_agg(tile: Column): Column — Average tiles per group.
Multi-tile fixture: 3 per-band rows from rgb_nir_small.tif (same grid). All tabs use the same grouped-agg invocation on multiple tile rows.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Group by region and average
SELECT
region,
gbx_rst_combineavg_agg(tile) as regional_average
FROM rasters
GROUP BY region;
# Heavyweight SQL — one tile struct per group:
+------+-----------------------------------------------------------+
|region|regional_average |
+------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(<agg>, 'GTiff') to rebuild a tile:
+------+----------------+
|region|regional_average|
+------+----------------+
|... |[B@... (BINARY) |
+------+----------------+
from databricks.labs.gbx.pyrx import functions as rx
df = _get_multi_band_tiles_df(spark)
result = (
df.groupBy("region")
.agg(rx.rst_combineavg_agg("tile").alias("avg_tile"))
.first()
)
+------+-----------------------------------------------------------+
|region|avg_tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
df = _get_multi_band_tiles_df_heavy(spark)
result = (
df.groupBy("region")
.agg(rx.rst_combineavg_agg("tile").alias("avg_tile"))
.first()
)
+------+-----------------------------------------------------------+
|region|avg_tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Multi-tile fixture: load multiband tif and split to 3 per-band rows (same grid).
val mb = spark.read.format("gdal").load("src/test/resources/binary/geotiff-small/rgb_nir_small.tif")
val b1 = mb.select(rx.rst_band(col("tile"), lit(1)).alias("tile")).withColumn("region", lit("R1"))
val b2 = mb.select(rx.rst_band(col("tile"), lit(2)).alias("tile")).withColumn("region", lit("R1"))
val b3 = mb.select(rx.rst_band(col("tile"), lit(3)).alias("tile")).withColumn("region", lit("R1"))
val df = b1.union(b2).union(b3)
val result = df.groupBy("region").agg(rx.rst_combineavg_agg(col("tile")).alias("avg"))
result.show()
+------+-----------------------------------------------------------+
|region|avg |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_derivedband_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio with GDAL VRT Python pixel functions. Aggregate — groupBy(...).agg(rx.rst_derivedband_agg("tile", pyfunc, funcName)) stacks the group's tiles as bands and applies your pixel function, returning one tile per group.
Heavyweight gbx_rst_derivedband_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_derivedband_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(gbx_rst_derivedband_agg(tile, 'def f(a,b): return a+b', 'f'), 'GTiff') AS tile
FROM tiles
GROUP BY group_key
Signature: rst_derivedband_agg(tile: Column, pyfunc: String, funcName: String): Column — Apply Python UDF to tiles per group.
Multi-tile fixture: 3 per-band rows from rgb_nir_small.tif. All tabs use the same grouped-agg invocation; pixel function selects band 0 (identity).
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT region, gbx_rst_derivedband_agg(tile, 'def f(a): return a', 'f') as result FROM rasters GROUP BY region;
# Heavyweight SQL — one v2 tile struct per group:
+------+-----------------------------------------------------------+
|region|result |
+------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(result, 'GTiff') to rebuild a tile struct:
+------+---------------+
|region|result |
+------+---------------+
|... |[B@... (BINARY)|
+------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
pyfunc = (
"def fn(in_ar, out_ar, xoff, yoff, xsize, ysize, "
"raster_xsize, raster_ysize, buf_radius, gt, **kwargs):\n"
" out_ar[:] = in_ar[0]\n"
)
df = _get_multi_band_tiles_df(spark)
result = (
df.groupBy("region")
.agg(rx.rst_derivedband_agg("tile", pyfunc, "fn").alias("derived"))
.first()
)
+------+-----------------------------------------------------------+
|region|derived |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
pyfunc = (
"def fn(in_ar, out_ar, xoff, yoff, xsize, ysize, "
"raster_xsize, raster_ysize, buf_radius, gt, **kwargs):\n"
" out_ar[:] = in_ar[0]\n"
)
df = _get_multi_band_tiles_df_heavy(spark)
result = (
df.groupBy("region")
.agg(
rx.rst_derivedband_agg("tile", f.lit(pyfunc), f.lit("fn")).alias("derived")
)
.first()
)
+------+-----------------------------------------------------------+
|region|derived |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Multi-tile fixture: load multiband tif and split to 3 per-band rows.
val mb = spark.read.format("gdal").load("src/test/resources/binary/geotiff-small/rgb_nir_small.tif")
val b1 = mb.select(rx.rst_band(col("tile"), lit(1)).alias("tile")).withColumn("region", lit("R1"))
val b2 = mb.select(rx.rst_band(col("tile"), lit(2)).alias("tile")).withColumn("region", lit("R1"))
val b3 = mb.select(rx.rst_band(col("tile"), lit(3)).alias("tile")).withColumn("region", lit("R1"))
val df = b1.union(b2).union(b3)
val fn = "def fn(in_ar, out_ar, xoff, yoff, xsize, ysize, raster_xsize, raster_ysize, buf_radius, gt, **kwargs):\n out_ar[:] = in_ar[0]\n"
val result = df.groupBy("region").agg(rx.rst_derivedband_agg(col("tile"), fn, "fn").alias("derived"))
result.show()
+------+-----------------------------------------------------------+
|region|derived |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_dtmfromgeoms_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio + SciPy (scipy.spatial.Delaunay). Aggregate — builds one TIN DTM tile per group from the group's Z-valued points via barycentric interpolation over an unconstrained Delaunay triangulation; breaklines, merge_tolerance, and snap_tolerance are accepted but not enforced (the heavyweight tier builds a constrained TIN). The lightweight Python binding also accepts an optional trailing out_crs (string CRS) argument that the SQL/heavyweight tiers do not — the heavyweight builder takes the int out_srid only. out_crs wins over out_srid when both are given. This is a lightweight superset, not a heavyweight regression.
Heavyweight gbx_rst_dtmfromgeoms_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_dtmfromgeoms_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(
gbx_rst_dtmfromgeoms_agg(point, null, 0.0, 0.0, 0,0,10,10, 8,8, 32633),
'GTiff'
) AS tile
FROM observations
GROUP BY group_key
Streaming aggregator that accepts one Z-valued point WKB per row and produces a TIN/Delaunay DTM raster tile per group; breaklines are supplied as a per-group constant array to enforce hard terrain edges.
Signature: rst_dtmfromgeoms_agg(point: Column, breaklines: Column, mergeTolerance: Column, snapTolerance: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, out_srid: Column): Column
Parameters: point — WKB point geometry with Z coordinate (one per row); breaklines — constant WKB array of breakline geometries per group (pass null or empty array if unused); remaining parameters match rst_dtmfromgeoms
Multi-row fixture: 4 Z-valued WKB POINT rows (elevation 100–250 m) over a [0,0,1,1] EPSG:4326 extent. All tabs use the same grouped-agg invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Stream survey points per region into one TIN DTM tile. Breaklines are a
-- per-group constant array; for 10 m cells over a 1000 m extent use 100 px.
SELECT region_id,
gbx_rst_dtmfromgeoms_agg(
point_wkb, breaklines_wkb_array,
0.0, 0.01,
bbox_xmin, bbox_ymin, bbox_xmax, bbox_ymax,
100, 100, 32633
) AS dtm
FROM survey_points
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|dtm |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(dtm, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|dtm |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
BinaryType,
StringType,
StructField,
StructType,
)
def _wkb_point_z(x, y, z):
"""Build a WKB POINT Z (ISO wkbType 1001) as bytes."""
import struct
return struct.pack("<bIddd", 1, 1001, x, y, z)
rows = [
(_wkb_point_z(0.1, 0.1, 100.0), "R1"),
(_wkb_point_z(0.9, 0.1, 200.0), "R1"),
(_wkb_point_z(0.1, 0.9, 150.0), "R1"),
(_wkb_point_z(0.9, 0.9, 250.0), "R1"),
]
schema = StructType(
[
StructField("pt", BinaryType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_dtmfromgeoms_agg(
"pt",
f.lit(None).cast("array<binary>"),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(1.0),
f.lit(1.0),
f.lit(8),
f.lit(8),
f.lit(4326),
).alias("dtm")
)
.first()
)
+------+-----------------------------------------------------------+
|region|dtm |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
BinaryType,
StringType,
StructField,
StructType,
)
def _wkb_point_z(x, y, z):
import struct
return struct.pack("<bIddd", 1, 1001, x, y, z)
rows = [
(_wkb_point_z(0.1, 0.1, 100.0), "R1"),
(_wkb_point_z(0.9, 0.1, 200.0), "R1"),
(_wkb_point_z(0.1, 0.9, 150.0), "R1"),
(_wkb_point_z(0.9, 0.9, 250.0), "R1"),
]
schema = StructType(
[
StructField("pt", BinaryType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_dtmfromgeoms_agg(
"pt",
f.lit(None).cast("array<binary>"),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(1.0),
f.lit(1.0),
f.lit(8),
f.lit(8),
f.lit(4326),
).alias("dtm")
)
.first()
)
+------+-----------------------------------------------------------+
|region|dtm |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import java.nio.{ByteBuffer, ByteOrder}
rx.register(spark)
// Multi-row fixture: 4 WKB POINT Z rows over [0,0,1,1] EPSG:4326.
def mkPtZ(x: Double, y: Double, z: Double): Array[Byte] = {
val buf = ByteBuffer.allocate(29).order(ByteOrder.LITTLE_ENDIAN)
buf.put(1.toByte); buf.putInt(1001); buf.putDouble(x); buf.putDouble(y); buf.putDouble(z); buf.array()
}
val schema = StructType(Seq(StructField("pt", BinaryType()), StructField("region", StringType())))
val rows = Seq((mkPtZ(0.1,0.1,100.0),"R1"),(mkPtZ(0.9,0.1,200.0),"R1"),(mkPtZ(0.1,0.9,150.0),"R1"),(mkPtZ(0.9,0.9,250.0),"R1"))
val df = spark.createDataFrame(spark.sparkContext.parallelize(rows.map { case (p,r) => org.apache.spark.sql.Row(p,r) }), schema)
val result = df.groupBy("region").agg(
rx.rst_dtmfromgeoms_agg(col("pt"), lit(null).cast("array<binary>"), lit(0.0), lit(0.0), lit(0.0), lit(0.0), lit(1.0), lit(1.0), lit(8), lit(8), lit(4326)).alias("dtm")
)
result.show()
+------+-----------------------------------------------------------+
|region|dtm |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
A Delaunay TIN must hold all of a group's points in memory to triangulate, so peak memory is
proportional to the points per group — and, unlike rst_binpoints_agg,
a triangulation is inherently global and cannot be made grid-bounded the way
bin_points_tiled bounds binning. At full LiDAR density a single tile's ground
returns (hundreds of thousands to millions of points) can exhaust worker RAM.
A bare-earth TIN is near-identical from a representative subset, so thin the input points before
the aggregate — df.sample(fraction=…) down to a bounded per-group count (e.g. ~150 k), or use
smaller tiles. This is standard LiDAR TIN practice and keeps memory bounded regardless of source
density. See the LiDAR reader and Performance.
rst_frombands_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio. Aggregate — groupBy(...).agg(rx.rst_frombands_agg("tile", "band_index")) stacks the group's tiles into one multi-band tile ordered by ascending band_index.
Heavyweight gbx_rst_frombands_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_frombands_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(gbx_rst_frombands_agg(tile, band_index), 'GTiff') AS tile
FROM bands
GROUP BY group_key
Streaming aggregator that collects ordered per-band tiles (one row per band) into a single multi-band raster tile per group; use when bands arrive as separate rows rather than a pre-built array.
Signature: rst_frombands_agg(tile: Column, bandIndex: Column): Column
Parameters: tile — Single-band raster tile; bandIndex — 1-based band position within the output raster
Multi-tile fixture: 3 per-band rows from rgb_nir_small.tif with band_index 1/2/3. All tabs stack via the same grouped-agg invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Collect per-band tiles in acquisition order into one multi-band raster per scene.
SELECT scene_id,
gbx_rst_frombands_agg(tile, band_index) AS multi_band
FROM band_tiles
GROUP BY scene_id;
# Heavyweight SQL — one v2 tile struct per group:
+--------+-----------------------------------------------------------+
|scene_id|multi_band |
+--------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+--------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(multi_band, 'GTiff') to rebuild a tile struct:
+--------+---------------+
|scene_id|multi_band |
+--------+---------------+
|... |[B@... (BINARY)|
+--------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
df = _get_multi_band_tiles_df(spark)
result = (
df.groupBy("region")
.agg(rx.rst_frombands_agg("tile", "band_index").alias("stacked"))
.first()
)
+------+-----------------------------------------------------------+
|region|stacked |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
df = _get_multi_band_tiles_df_heavy(spark)
result = (
df.groupBy("region")
.agg(rx.rst_frombands_agg("tile", "band_index").alias("stacked"))
.first()
)
+------+-----------------------------------------------------------+
|region|stacked |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Multi-tile fixture: load multiband tif and split to 3 per-band rows with band_index.
val mb = spark.read.format("gdal").load("src/test/resources/binary/geotiff-small/rgb_nir_small.tif")
val b1 = mb.select(rx.rst_band(col("tile"), lit(1)).alias("tile")).withColumn("band_index", lit(1)).withColumn("region", lit("R1"))
val b2 = mb.select(rx.rst_band(col("tile"), lit(2)).alias("tile")).withColumn("band_index", lit(2)).withColumn("region", lit("R1"))
val b3 = mb.select(rx.rst_band(col("tile"), lit(3)).alias("tile")).withColumn("band_index", lit(3)).withColumn("region", lit("R1"))
val df = b1.union(b2).union(b3)
val result = df.groupBy("region").agg(rx.rst_frombands_agg(col("tile"), col("band_index")).alias("stacked"))
result.show()
+------+-----------------------------------------------------------+
|region|stacked |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_gridfrompoints_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio + SciPy (cKDTree IDW). Aggregate — groupBy(...).agg(rx.rst_gridfrompoints_agg(...)) inverse-distance-interpolates the group's points into one Float64 grid tile (NoData −9999). The lightweight Python binding also accepts an optional trailing out_crs (string CRS) argument that the SQL/heavyweight tiers do not — the heavyweight builder takes the int out_srid only. out_crs wins over out_srid when both are given. This is a lightweight superset, not a heavyweight regression.
Heavyweight gbx_rst_gridfrompoints_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_gridfrompoints_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(
gbx_rst_gridfrompoints_agg(point, value, 0,0,10,10, 8,8, 32633, 2.0, 12),
'GTiff'
) AS tile
FROM observations
GROUP BY group_key
Streaming IDW-interpolation aggregator that accepts one point geometry and one scalar value per row and produces a Float64 GeoTIFF tile per group; use when observations arrive one per row rather than as pre-built arrays.
Signature: rst_gridfrompoints_agg(point: Column, value: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, widthPx: Column, heightPx: Column, out_srid: Column, power: Column, maxPts: Column): Column
Parameters: point — WKB point geometry (one per row); value — scalar observation for the point; xmin/ymin/xmax/ymax — output extent in CRS units (constant per group); widthPx/heightPx — output dimensions in pixels; srid — EPSG code; power — IDW distance-decay exponent (2.0 is standard); maxPts — maximum nearest neighbours considered per output pixel
Multi-row fixture: 4 WKB POINT rows with observations 10–40 over a [0,0,1,1] EPSG:4326 extent. All tabs use the same grouped-agg IDW invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Aggregate per-station observations into one IDW tile per region.
SELECT region_id,
gbx_rst_gridfrompoints_agg(
station_wkb, observation,
bbox_xmin, bbox_ymin, bbox_xmax, bbox_ymax,
256, 256, 32633
) AS idw
FROM observations
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|idw |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(idw, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|idw |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
BinaryType,
DoubleType,
StringType,
StructField,
StructType,
)
def _wkb_point(x, y):
"""Build a WKB POINT from (x, y) as bytes."""
import struct
return struct.pack("<bIdd", 1, 1, x, y)
rows = [
(_wkb_point(0.1, 0.1), 10.0, "R1"),
(_wkb_point(0.9, 0.1), 20.0, "R1"),
(_wkb_point(0.1, 0.9), 30.0, "R1"),
(_wkb_point(0.9, 0.9), 40.0, "R1"),
]
schema = StructType(
[
StructField("pt", BinaryType()),
StructField("val", DoubleType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_gridfrompoints_agg(
"pt",
"val",
f.lit(0.0),
f.lit(0.0),
f.lit(1.0),
f.lit(1.0),
f.lit(8),
f.lit(8),
f.lit(4326),
).alias("idw")
)
.first()
)
+------+-----------------------------------------------------------+
|region|idw |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
BinaryType,
DoubleType,
StringType,
StructField,
StructType,
)
def _wkb_point(x, y):
import struct
return struct.pack("<bIdd", 1, 1, x, y)
rows = [
(_wkb_point(0.1, 0.1), 10.0, "R1"),
(_wkb_point(0.9, 0.1), 20.0, "R1"),
(_wkb_point(0.1, 0.9), 30.0, "R1"),
(_wkb_point(0.9, 0.9), 40.0, "R1"),
]
schema = StructType(
[
StructField("pt", BinaryType()),
StructField("val", DoubleType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_gridfrompoints_agg(
"pt",
"val",
f.lit(0.0),
f.lit(0.0),
f.lit(1.0),
f.lit(1.0),
f.lit(8),
f.lit(8),
f.lit(4326),
).alias("idw")
)
.first()
)
+------+-----------------------------------------------------------+
|region|idw |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import java.nio.{ByteBuffer, ByteOrder}
rx.register(spark)
// Multi-row fixture: 4 WKB POINT rows with scalar observations, [0,0,1,1] EPSG:4326 extent.
def mkPt(x: Double, y: Double): Array[Byte] = {
val buf = ByteBuffer.allocate(21).order(ByteOrder.LITTLE_ENDIAN)
buf.put(1.toByte); buf.putInt(1); buf.putDouble(x); buf.putDouble(y); buf.array()
}
val schema = StructType(Seq(StructField("pt", BinaryType()), StructField("val", DoubleType()), StructField("region", StringType())))
val rows = Seq((mkPt(0.1,0.1),10.0,"R1"),(mkPt(0.9,0.1),20.0,"R1"),(mkPt(0.1,0.9),30.0,"R1"),(mkPt(0.9,0.9),40.0,"R1"))
val df = spark.createDataFrame(spark.sparkContext.parallelize(rows.map { case (p,v,r) => org.apache.spark.sql.Row(p,v,r) }), schema)
val result = df.groupBy("region").agg(
rx.rst_gridfrompoints_agg(col("pt"), col("val"), lit(0.0), lit(0.0), lit(1.0), lit(1.0), lit(8), lit(8), lit(4326)).alias("idw")
)
result.show()
+------+-----------------------------------------------------------+
|region|idw |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_h3_rasterize_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio + h3. Aggregate — groupBy(...).agg(rx.rst_h3_rasterize_agg("cellid", "value", ...)) burns each H3 cell's centroid pixel (or spatial-envelope pixels with mode='spatial_envelope') into one raster tile per group. When value is omitted or null, all burned pixels carry 1.0 (presence mask). The extent and pixel size are derived automatically from the H3 resolution unless explicit bounds are supplied; kring_pad (default 1) expands the canvas by that many rings to avoid clipping edge cells.
Heavyweight gbx_rst_h3_rasterize_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_h3_rasterize_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
region_id AS cellid,
gbx_rst_fromcontent(
gbx_rst_h3_rasterize_agg(cellid, burn_value, 4326, null, null,null,null,null, null,null, 'centroids', 1),
'GTiff'
) AS tile
FROM h3_cell_values
GROUP BY region_id
Streaming aggregator that burns H3 cell centroid pixels (or spatial-envelope pixels) into one raster tile per group. This is the inverse of rst_h3_rastertogrid*: where those functions reduce raster pixels to per-cell statistics, rst_h3_rasterize_agg reconstructs a raster from per-cell values. Use rst_frombands_agg to stack per-threshold rasters (each produced by one rst_h3_rasterize_agg call) into a single multi-band output.
out_crs parameterThe lightweight Python binding accepts an optional trailing out_crs (string CRS) argument that the heavyweight/Scala tier does not. When supplied it overrides the integer out_srid for the output CRS; the heavyweight tier accepts only the integer out_srid (its builder() is strictly 12-argument). This is a lightweight superset, not a heavyweight regression.
The H3 Rasterize notebook walks this through end to end on a San Francisco Bay Area DEM: elevation isobands → H3 polyfill → a shared canvas from rst_h3_gridspec → per-band rst_h3_rasterize_agg → multi-band stack via rst_frombands_agg, visualized with the gbx.vizx helpers. The same pattern maps directly to a telco multi-threshold signal-coverage stack.
Signature: rst_h3_rasterize_agg(cellid: Column, value: Column, out_srid: Column, pixel_size: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, mode: Column, kring_pad: Column): Column
Parameters:
cellid— H3 cell ID (BIGINT or STRING) to burn (one per row)value— numeric burn value; passnullfor a presence mask (all pixels →1.0)srid— EPSG code for the output CRS; defaults to4326(WGS 84)pixel_size— ground resolution in CRS units (derives from H3 resolution whennull)xmin/ymin/xmax/ymax— output canvas extent; auto-computed from cell bounds +kring_padwhennullwidth/height— output raster dimensions in pixels; auto-derived from extent + pixel_size whennullmode—'centroids'(default, burns the cell-centroid pixel only) or'spatial_envelope'(burns all pixels inside the hexagon envelope)kring_pad— ring count by which to expand the auto-computed canvas (default1)
Multi-row fixture: 3 H3 resolution-9 cell rows with burn values 1.0/2.0/3.0. All tabs use the same grouped-agg rasterize invocation.
Since v0.4.0- SQL
- Python (light)
- Python (heavy)
- Scala
-- Rasterize H3 cells into one raster tile per region. Each cell's value is
-- burned at the cell centroid pixel.
SELECT region_id,
gbx_rst_h3_rasterize_agg(
cellid, burn_value,
4326, cast(null as double),
cast(null as double), cast(null as double),
cast(null as double), cast(null as double),
cast(null as int), cast(null as int),
'centroids', cast(1 as int)
) AS tile
FROM h3_cell_values
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|tile |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(tile, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|tile |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
import h3
res = 9
cell_strs = [
h3.latlng_to_cell(0.01, 0.01, res),
h3.latlng_to_cell(0.02, 0.01, res),
h3.latlng_to_cell(0.01, 0.02, res),
]
cells = [h3.str_to_int(c) for c in cell_strs]
rows = [
(int(cells[0]), 1.0, "R1"),
(int(cells[1]), 2.0, "R1"),
(int(cells[2]), 3.0, "R1"),
]
from pyspark.sql.types import (
DoubleType,
LongType,
StringType,
StructField,
StructType,
)
schema = StructType(
[
StructField("cellid", LongType()),
StructField("value", DoubleType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(rx.rst_h3_rasterize_agg("cellid", "value").alias("tile"))
.first()
)
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
import h3
res = 9
cell_strs = [
h3.latlng_to_cell(0.01, 0.01, res),
h3.latlng_to_cell(0.02, 0.01, res),
h3.latlng_to_cell(0.01, 0.02, res),
]
cells = [h3.str_to_int(c) for c in cell_strs]
rows = [
(int(cells[0]), 1.0, "R1"),
(int(cells[1]), 2.0, "R1"),
(int(cells[2]), 3.0, "R1"),
]
from pyspark.sql.types import (
DoubleType,
LongType,
StringType,
StructField,
StructType,
)
schema = StructType(
[
StructField("cellid", LongType()),
StructField("value", DoubleType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
# cellid + value are the only required args; extent/size/mode/kring and the
# EPSG:4326 out_srid all take their defaults (same as the lightweight tier).
result = (
df.groupBy("region")
.agg(rx.rst_h3_rasterize_agg(df["cellid"], df["value"]).alias("tile"))
.first()
)
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
rx.register(spark)
// Multi-row fixture: 3 H3 resolution-9 cell ids (as BIGINT) with burn values.
val schema = StructType(Seq(
StructField("cellid", LongType()), StructField("value", DoubleType()), StructField("region", StringType())))
// H3 res-9 cells near origin
val rows = Seq((617733151020810239L, 1.0, "R1"), (617733151021334527L, 2.0, "R1"), (617733151085035519L, 3.0, "R1"))
val df = spark.createDataFrame(spark.sparkContext.parallelize(rows.map(r => org.apache.spark.sql.Row(r._1,r._2,r._3))), schema)
val result = df.groupBy("region").agg(rx.rst_h3_rasterize_agg(col("cellid"), col("value")).alias("tile"))
result.show()
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_merge_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio (rasterio.merge). Aggregate — groupBy(...).agg(rx.rst_merge_agg("tile")) merges the group's tiles into one mosaic tile (output spans the union extent).
Heavyweight gbx_rst_merge_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_merge_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(gbx_rst_merge_agg(tile), 'GTiff') AS tile
FROM tiles
GROUP BY group_key
Signature: rst_merge_agg(tile: Column): Column — Merge tiles per group.
Multi-tile fixture: 3 per-band rows from rgb_nir_small.tif. All tabs use the same grouped-agg mosaic invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT
scene_id,
gbx_rst_merge_agg(tile) as merged_scene
FROM satellite_tiles
GROUP BY scene_id;
# Heavyweight SQL — one v2 tile struct per group:
+--------+-----------------------------------------------------------+
|scene_id|merged_scene |
+--------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+--------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(merged_scene, 'GTiff') to rebuild a tile struct:
+--------+---------------+
|scene_id|merged_scene |
+--------+---------------+
|... |[B@... (BINARY)|
+--------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
df = _get_multi_band_tiles_df(spark)
result = df.groupBy("region").agg(rx.rst_merge_agg("tile").alias("mosaic")).first()
+------+-----------------------------------------------------------+
|region|mosaic |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
df = _get_multi_band_tiles_df_heavy(spark)
result = df.groupBy("region").agg(rx.rst_merge_agg("tile").alias("mosaic")).first()
+------+-----------------------------------------------------------+
|region|mosaic |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Multi-tile fixture: load multiband tif and split to 3 per-band rows.
val mb = spark.read.format("gdal").load("src/test/resources/binary/geotiff-small/rgb_nir_small.tif")
val b1 = mb.select(rx.rst_band(col("tile"), lit(1)).alias("tile")).withColumn("region", lit("R1"))
val b2 = mb.select(rx.rst_band(col("tile"), lit(2)).alias("tile")).withColumn("region", lit("R1"))
val b3 = mb.select(rx.rst_band(col("tile"), lit(3)).alias("tile")).withColumn("region", lit("R1"))
val df = b1.union(b2).union(b3)
val result = df.groupBy("region").agg(rx.rst_merge_agg(col("tile")).alias("mosaic"))
result.show()
+------+-----------------------------------------------------------+
|region|mosaic |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_quadbin_rasterize_agg
LightweightHeavyweight Grouped-agg UDFThe lightweight implementation is backed by pygx._quadbin cell math and rasterio — it burns each quadbin cell's geometry/value into the output band via rasterio.features.rasterize, matching the heavyweight cell set and burn values exactly.
Pixels not covered by any geometry in the group are set to -9999.0 (band-registered NoData). Filter or mask downstream via gbx_rst_getnodata or IS NULL on the extracted band value.
Heavyweight gbx_rst_quadbin_rasterize_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_quadbin_rasterize_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
cell_id AS cellid,
gbx_rst_fromcontent(gbx_rst_quadbin_rasterize_agg(cellid, burn_value), 'GTiff') AS tile
FROM quadbin_cell_values
GROUP BY cell_id
Streaming aggregator that burns quadbin cell geometry/value pairs (one row per cell) into a single rasterized tile per group. The input raster is interpreted as EPSG:4326 (lon/lat); resolution is the quadbin zoom level (0..26). The inverse of rst_quadbin_rastertogrid*: where those functions reduce raster pixels to per-cell statistics, this one synthesizes a raster from per-cell values.
out_crs parameterThe lightweight Python binding accepts an optional trailing out_crs (string CRS) argument that the heavyweight/Scala tier does not. When supplied it overrides the integer out_srid for the output CRS; the heavyweight tier accepts only the integer out_srid (its builder() is strictly 12-argument). This is a lightweight superset, not a heavyweight regression.
Signature: rst_quadbin_rasterize_agg(cellid: Column, value: Column, out_srid: Column, pixel_size: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, mode: Column, kring_pad: Column): Column
Parameters: cellid — quadbin cell ID (BIGINT); value — numeric burn value; out_srid — EPSG code for the output CRS; pixel_size — output raster cell size in the target CRS; xmin/ymin/xmax/ymax — output extent in the target CRS; width/height — output raster dimensions in pixels; mode — aggregation mode for overlapping values (typically "last"); kring_pad — cell neighbourhood expansion (typically 0)
Multi-row fixture: 3 quadbin zoom-12 cell rows near central London with burn values 1.0/2.0/3.0. All tabs use the same grouped-agg rasterize invocation.
Since v0.4.3- SQL
- Python (light)
- Python (heavy)
- Scala
-- Rasterize quadbin cells into one raster tile per region. Each cell's value
-- is burned at the cell centroid pixel; the extent auto-derives from the cell
-- set (null canvas args). cellid is BIGINT.
SELECT region_id,
gbx_rst_quadbin_rasterize_agg(
cellid, burn_value,
4326, cast(null as double),
cast(null as double), cast(null as double),
cast(null as double), cast(null as double),
cast(null as int), cast(null as int),
'centroids', cast(0 as int)
) AS tile
FROM quadbin_cell_values
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|tile |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(tile, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|tile |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
from databricks.labs.gbx.gridx.quadbin import functions as qbx
qbx.register(spark)
spark.sql("""
CREATE OR REPLACE TEMP VIEW _qb_cells AS
SELECT region,
gbx_quadbin_pointascell(cast(lon as double), cast(lat as double), 12) AS cellid,
cast(val as double) AS value
FROM (VALUES
('R1', -0.10, 51.50, 1.0),
('R1', -0.11, 51.51, 2.0),
('R1', -0.09, 51.49, 3.0)
) AS t(region, lon, lat, val)
""")
df = spark.table("_qb_cells")
result = (
df.groupBy("region")
.agg(rx.rst_quadbin_rasterize_agg("cellid", "value").alias("tile"))
.first()
)
spark.catalog.dropTempView("_qb_cells")
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.gridx.quadbin import functions as qbx
qbx.register(spark)
spark.sql("""
CREATE OR REPLACE TEMP VIEW _qb_cells_heavy AS
SELECT region,
gbx_quadbin_pointascell(cast(lon as double), cast(lat as double), 12) AS cellid,
cast(val as double) AS value
FROM (VALUES
('R1', -0.10, 51.50, 1.0),
('R1', -0.11, 51.51, 2.0),
('R1', -0.09, 51.49, 3.0)
) AS t(region, lon, lat, val)
""")
df = spark.table("_qb_cells_heavy")
# cellid + value are the only required args; extent/size/mode/kring and the
# EPSG:4326 out_srid all take their defaults (same as the lightweight tier).
result = (
df.groupBy("region")
.agg(rx.rst_quadbin_rasterize_agg(df["cellid"], df["value"]).alias("tile"))
.first()
)
spark.catalog.dropTempView("_qb_cells_heavy")
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import com.databricks.labs.gbx.gridx.{functions => gx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
rx.register(spark)
gx.register(spark)
// Multi-row fixture: 3 quadbin zoom-12 cells near central London.
val schema = StructType(Seq(StructField("region", StringType()), StructField("lon", DoubleType()), StructField("lat", DoubleType()), StructField("val", DoubleType())))
val rows = Seq(("R1",-0.10,51.50,1.0),("R1",-0.11,51.51,2.0),("R1",-0.09,51.49,3.0))
val raw = spark.createDataFrame(spark.sparkContext.parallelize(rows.map(r => org.apache.spark.sql.Row(r._1,r._2,r._3,r._4))), schema)
raw.createOrReplaceTempView("_qb_src")
val df = spark.sql("SELECT region, gbx_quadbin_pointascell(lon, lat, 12) AS cellid, val AS value FROM _qb_src")
val result = df.groupBy("region").agg(rx.rst_quadbin_rasterize_agg(col("cellid"), col("value")).alias("tile"))
result.show()
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_custom_rasterize_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio + pygx custom-grid cell math. Aggregate — groupBy(...).agg(rx.rst_custom_rasterize_agg("cellid", "value", ...)) burns each custom-grid cell's centroid pixel (or spatial-envelope pixels with mode='spatial_envelope') into one raster tile per group. Cell IDs are BIGINT encoded by the custom-grid layout. When value is omitted or null, all burned pixels carry 1.0 (presence mask). The extent auto-derives from the cell set when canvas parameters are null.
Heavyweight gbx_rst_custom_rasterize_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_custom_rasterize_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
region_id AS cellid,
gbx_rst_fromcontent(
gbx_rst_custom_rasterize_agg(
cellid, burn_value,
gbx_custom_grid(...),
27700, cast(null as double),
cast(null as double), cast(null as double), cast(null as double), cast(null as double),
cast(null as int), cast(null as int), 'centroids', cast(1 as int)
),
'GTiff'
) AS tile
FROM custom_cell_values
GROUP BY region_id
Streaming aggregator that burns custom-grid cell values into one raster tile per group. Cell IDs are BIGINT encoded by the gbx_custom_grid(...) layout. This is the inverse of rst_custom_rastertogrid*: where those functions reduce raster pixels to per-cell statistics, rst_custom_rasterize_agg reconstructs a raster from per-cell values. Pass the custom-grid struct (from gbx_custom_grid(...)) as the third argument; the canvas extent auto-derives from the cell set when all spatial parameters are null.
Signature: rst_custom_rasterize_agg(cellid: Column, value: Column, grid: Column, out_srid: Column, pixel_size: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, mode: Column, kring_pad: Column): Column
Parameters: cellid — custom-grid cell ID (BIGINT); value — numeric burn value (null → 1.0 presence mask); grid — custom-grid struct from gbx_custom_grid(...); out_srid — EPSG code for the output CRS; pixel_size — output raster cell size in target CRS units (null → auto-derived from cell dimensions); xmin/ymin/xmax/ymax — output extent (null → auto-derived from the cell set + kring_pad); width/height — output dimensions in pixels (null → auto-derived); mode — "centroids" (default) or "spatial_envelope"; kring_pad — neighbourhood expansion in rings for auto-extent (default 1)
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Rasterize custom-grid cells into one raster tile per region. cellid is BIGINT.
-- Pass the custom grid struct as the third argument; the extent auto-derives from
-- the cell set (null canvas args).
SELECT region_id,
gbx_rst_custom_rasterize_agg(
cellid, burn_value,
{_CUSTOM_GRID_LITERAL},
27700, cast(null as double),
cast(null as double), cast(null as double),
cast(null as double), cast(null as double),
cast(null as int), cast(null as int),
'centroids', cast(1 as int)
) AS tile
FROM custom_cell_values
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|tile |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY; wrap with
# gbx_rst_fromcontent(tile, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|tile |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
from databricks.labs.gbx.pygx import functions as gx
from pyspark.sql import functions as f
rx.register(spark)
gx.register(spark)
# Custom grid: 4km BNG London square, root cell 4000m, splits=2 -> res 1 = four 2km cells.
_GRID_SQL = (
"gbx_custom_grid(529000, 533000, 179000, 183000, 2, 4000, 4000, 27700)"
)
# Compute cell IDs by calling gbx_custom_pointascell in a SELECT over a VALUES table
# (UDFs cannot appear in VALUES clauses directly — use SELECT-from-VALUES instead).
spark.sql(f"""
CREATE OR REPLACE TEMP VIEW _custom_rasterize_cells AS
SELECT region,
gbx_custom_pointascell(wkt, {_GRID_SQL}, 1) AS cellid,
val AS value
FROM (VALUES
('R1', 'POINT(530000 180000)', cast(1.0 as double)),
('R1', 'POINT(532000 180000)', cast(2.0 as double)),
('R1', 'POINT(530000 182000)', cast(3.0 as double))
) AS t(region, wkt, val)
""")
df = spark.table("_custom_rasterize_cells")
grid_col = f.call_function(
"gbx_custom_grid",
f.lit(529000), f.lit(533000), f.lit(179000), f.lit(183000),
f.lit(2), f.lit(4000), f.lit(4000), f.lit(27700),
)
result = (
df.groupBy("region")
.agg(
rx.rst_custom_rasterize_agg("cellid", grid_col, f.col("value")).alias("tile")
)
.first()
)
spark.catalog.dropTempView("_custom_rasterize_cells")
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null; three 2km custom-grid cells burned)
if rx is None:
raise ImportError("rasterx not installed")
from databricks.labs.gbx.gridx.custom import functions as cx
from pyspark.sql import functions as f
rx.register(spark)
cx.register(spark) # registers gbx_custom_pointascell and gbx_custom_grid
_GRID_SQL = (
"gbx_custom_grid(529000, 533000, 179000, 183000, 2, 4000, 4000, 27700)"
)
# Compute three resolution-1 cell IDs via SQL (three of the four 2km cells).
# Use SELECT-from-VALUES: UDFs cannot appear directly in VALUES clauses.
spark.sql(f"""
CREATE OR REPLACE TEMP VIEW _custom_rasterize_cells_heavy AS
SELECT region,
gbx_custom_pointascell(wkt, {_GRID_SQL}, 1) AS cellid,
val AS value
FROM (VALUES
('R1', 'POINT(530000 180000)', cast(1.0 as double)),
('R1', 'POINT(532000 180000)', cast(2.0 as double)),
('R1', 'POINT(530000 182000)', cast(3.0 as double))
) AS t(region, wkt, val)
""")
df = spark.table("_custom_rasterize_cells_heavy")
result = (
df.groupBy("region")
.agg(
rx.rst_custom_rasterize_agg(
"cellid", _custom_grid_col(), f.col("value"),
out_srid=f.lit(27700), # heavy tier requires non-null out_srid
kring_pad=f.lit(0), # no padding: avoid expanding outside grid bounds
).alias("tile")
)
.first()
)
spark.catalog.dropTempView("_custom_rasterize_cells_heavy")
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null; three 2km custom-grid cells burned)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
rx.register(spark)
// Custom grid: 4km BNG London square, root cell 4000m, splits=2 -> res 1 = four 2km cells.
val grid = call_function("gbx_custom_grid",
lit(529000), lit(533000), lit(179000), lit(183000), lit(2), lit(4000), lit(4000), lit(27700))
// Compute three resolution-1 cell IDs (centroids of three 2km cells within the grid).
val schema = StructType(Seq(
StructField("region", StringType()),
StructField("cellid", LongType()),
StructField("value", DoubleType())))
val cells = spark.sql(
"SELECT 'R1' AS region, gbx_custom_pointascell('POINT(530000 180000)', " +
"gbx_custom_grid(529000,533000,179000,183000,2,4000,4000,27700), 1) AS cellid, 1.0 AS value " +
"UNION ALL SELECT 'R1', gbx_custom_pointascell('POINT(532000 180000)', " +
"gbx_custom_grid(529000,533000,179000,183000,2,4000,4000,27700), 1), 2.0 " +
"UNION ALL SELECT 'R1', gbx_custom_pointascell('POINT(530000 182000)', " +
"gbx_custom_grid(529000,533000,179000,183000,2,4000,4000,27700), 1), 3.0")
val result = cells.groupBy("region").agg(
rx.rst_custom_rasterize_agg(col("cellid"), col("value"), grid).alias("tile"))
result.show()
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null; three 2km custom-grid cells burned)
rst_rasterize_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio (rasterio.features). Aggregate — burns the group's (geom, value) rows into one tile over the given extent/size/SRID (last-wins on overlap).
Heavyweight gbx_rst_rasterize_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_rasterize_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(
gbx_rst_rasterize_agg(geom, value, 0,0,10,10, 8,8, 32633),
'GTiff'
) AS tile
FROM features
GROUP BY group_key
Streaming aggregator that burns geometry/value pairs (one row per feature) into a single rasterized tile per group; use when features arrive as individual rows rather than as a pre-built collection.
Signature: rst_rasterize_agg(geom: Column, value: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, out_srid: Column): Column
Parameters: geom — WKB geometry to burn; value — numeric burn value; xmin/ymin/xmax/ymax — output extent (in the target CRS); width/height — output raster dimensions in pixels; srid — EPSG code for the output CRS
Multi-row fixture: 3 polygon rows burned at values 1.0/2.0/3.0 over a [0,0,4,4] EPSG:4326 extent. All tabs use the same grouped-agg burn invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Aggregate per-feature burn values into one rasterized tile per region.
SELECT region_id,
gbx_rst_rasterize_agg(
geom_wkb, burn_value,
bbox_xmin, bbox_ymin, bbox_xmax, bbox_ymax,
256, 256, 4326
) AS tile
FROM features
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|tile |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(tile, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|tile |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
BinaryType,
DoubleType,
StringType,
StructField,
StructType,
)
# WKB POLYGON((0 0, 4 0, 4 4, 0 4, 0 0)) in EPSG:4326
poly = bytes.fromhex(
"0103000000010000000500000000000000000000000000000000000000"
"0000000000001040000000000000000000000000000010400000000000001040"
"000000000000000000000000000010400000000000000000"
"0000000000000000"
)
rows = [(poly, 1.0, "R1"), (poly, 2.0, "R1"), (poly, 3.0, "R1")]
schema = StructType(
[
StructField("geom", BinaryType()),
StructField("value", DoubleType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_rasterize_agg(
"geom",
"value",
f.lit(0.0),
f.lit(0.0),
f.lit(4.0),
f.lit(4.0),
f.lit(8),
f.lit(8),
f.lit(4326),
).alias("burned")
)
.first()
)
+------+-----------------------------------------------------------+
|region|burned |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
BinaryType,
DoubleType,
StringType,
StructField,
StructType,
)
# WKB POLYGON((0 0, 4 0, 4 4, 0 4, 0 0)) in EPSG:4326
poly = bytes.fromhex(
"0103000000010000000500000000000000000000000000000000000000"
"0000000000001040000000000000000000000000000010400000000000001040"
"000000000000000000000000000010400000000000000000"
"0000000000000000"
)
rows = [(poly, 1.0, "R1"), (poly, 2.0, "R1"), (poly, 3.0, "R1")]
schema = StructType(
[
StructField("geom", BinaryType()),
StructField("value", DoubleType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_rasterize_agg(
"geom",
"value",
f.lit(0.0),
f.lit(0.0),
f.lit(4.0),
f.lit(4.0),
f.lit(8),
f.lit(8),
f.lit(4326),
).alias("burned")
)
.first()
)
+------+-----------------------------------------------------------+
|region|burned |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Multi-row fixture: 3 rows with WKB polygons + burn values over a 4x4 EPSG:4326 canvas.
spark.sql("CREATE OR REPLACE TEMP VIEW _rst_agg_src AS SELECT 'R1' AS region, unhex('010300000001000000050000000000000000000000000000000000000000000000000000000000000000001040000000000000104000000000000000000000000000001040000000000000104000000000000000000000000000000000') AS geom, 1.0 AS value UNION ALL SELECT 'R1', unhex('010300000001000000050000000000000000000000000000000000000000000000000000000000000000001040000000000000104000000000000000000000000000001040000000000000104000000000000000000000000000000000'), 2.0 UNION ALL SELECT 'R1', unhex('010300000001000000050000000000000000000000000000000000000000000000000000000000000000001040000000000000104000000000000000000000000000001040000000000000104000000000000000000000000000000000'), 3.0")
val df = spark.table("_rst_agg_src")
val result = df.groupBy("region").agg(
rx.rst_rasterize_agg(col("geom"), col("value"), lit(0.0), lit(0.0), lit(4.0), lit(4.0), lit(8), lit(8), lit(4326)).alias("burned")
)
result.show()
+------+-----------------------------------------------------------+
|region|burned |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
Constructor Functions
Create or load rasters from path, binary content, or bands (4 total).
rst_dtmfromgeoms
LightweightHeavyweightPowered by rasterio + SciPy (scipy.spatial.Delaunay). Barycentric interpolation over an unconstrained Delaunay TIN; cells outside the convex hull are NoData. breaklines, merge_tolerance, and snap_tolerance are accepted for signature parity but not enforced — the heavyweight tier builds a constrained TIN that honors them.
Create a DTM raster tile via TIN/Delaunay interpolation from an array of Z-valued point WKB geometries, with an optional array of breakline WKB geometries to preserve sharp terrain transitions.
out_crs parameterThe lightweight Python binding accepts an optional trailing out_crs (string CRS) argument that the SQL and heavyweight/Scala tiers do not — the heavyweight RST_DTMFromGeoms builder is strictly ≤12-argument (…, out_srid, [no_data]) and rejects a 13th argument. In lightweight Python, out_crs (string) wins over the int out_srid. This is a lightweight superset, not a heavyweight regression.
Signature: rst_dtmfromgeoms(points_array: Column, breaklines_array: Column, merge_tolerance: Column, snap_tolerance: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width_px: Column, height_px: Column, out_srid: Column, [no_data: Column = null]): Column — output CRS via out_srid (int); input points assumed already in the output CRS. See Coordinate Reference Systems.
Parameters: points_array — Array of WKB point geometries with Z coordinates; breaklines_array — Array of WKB line/polygon geometries enforcing hard edges (pass null or empty array if unused); merge_tolerance/snap_tolerance — Delaunay triangulation tolerances (vertex-merge distance and snapping distance; small values such as 0.0 and 0.01 are typical); xmin/ymin/xmax/ymax — output extent in CRS units; width_px/height_px — output raster dimensions in pixels (for N-metre cells set width_px = round((xmax-xmin)/N)); out_srid — EPSG code for the output CRS. An optional trailing no_data argument overrides the default fill for cells outside the triangulated hull.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- TIN interpolation from arrays of Z-valued point WKB and breakline WKB.
-- Output is a 100 x 100 Float64 GTiff over the extent. For N-metre cells set
-- width_px = round((xmax-xmin)/N): here a 1000 m extent at 10 m cells -> 100 px.
SELECT gbx_rst_dtmfromgeoms(
points_wkb_array, breaklines_wkb_array,
0.0, 0.01,
0.0, 0.0, 1000.0, 1000.0,
100, 100, 32633
) AS dtm
FROM survey_points;
+-----------------------------------------------------------+
|dtm |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import StructType, StructField, ArrayType, BinaryType
rx.register(spark)
# Create a synthetic survey point DataFrame with 4 points inset from the grid
# edges (Delaunay needs the hull to cover interior cells).
# POINT Z geometries: (100,100,50), (900,100,80), (900,900,120), (100,900,60)
# Using explicit schema to avoid type inference errors
survey_data = [
(
[
bytes.fromhex(
"0101000080000000000000594000000000000059400000000000004940"
),
bytes.fromhex(
"01010000800000000000208c4000000000000059400000000000005440"
),
bytes.fromhex(
"01010000800000000000208c400000000000208c400000000000005e40"
),
bytes.fromhex(
"010100008000000000000059400000000000208c400000000000004e40"
),
],
[],
)
]
schema = StructType(
[
StructField("points_wkb_array", ArrayType(BinaryType()), True),
StructField("breaklines_wkb_array", ArrayType(BinaryType()), True),
]
)
df = spark.createDataFrame(survey_data, schema=schema)
result = df.select(
rx.rst_dtmfromgeoms(
f.col("points_wkb_array"),
f.col("breaklines_wkb_array"),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(1000.0),
f.lit(1000.0),
f.lit(100),
f.lit(100),
f.lit(32618),
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TIN-interpolated DTM over specified extent and pixel count)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
StructType,
StructField,
ArrayType,
BinaryType,
)
rx.register(spark)
# Create synthetic survey point data with 4 points inset from the grid edges
# POINT Z geometries: (100,100,50), (900,100,80), (900,900,120), (100,900,60)
# Using explicit schema to avoid type inference errors
survey_data = [
(
[
bytes.fromhex(
"0101000080000000000000594000000000000059400000000000004940"
),
bytes.fromhex(
"01010000800000000000208c4000000000000059400000000000005440"
),
bytes.fromhex(
"01010000800000000000208c400000000000208c400000000000005e40"
),
bytes.fromhex(
"010100008000000000000059400000000000208c400000000000004e40"
),
],
[],
)
]
schema = StructType(
[
StructField("points_wkb_array", ArrayType(BinaryType()), True),
StructField("breaklines_wkb_array", ArrayType(BinaryType()), True),
]
)
df = spark.createDataFrame(survey_data, schema=schema)
result = df.select(
rx.rst_dtmfromgeoms(
f.col("points_wkb_array"),
f.col("breaklines_wkb_array"),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(1000.0),
f.lit(1000.0),
f.lit(100),
f.lit(100),
f.lit(32618),
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TIN-interpolated DTM over specified extent and pixel count)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Create synthetic survey points with Z-valued WKB geometries
val survey = spark.createDataFrame(Seq(
(Array[Array[Byte]](/* WKB point 1 */, /* WKB point 2 */), Array[Array[Byte]]())
)).toDF("points_wkb", "breaklines_wkb")
val result = survey.select(
rx.rst_dtmfromgeoms(col("points_wkb"), col("breaklines_wkb"), lit(0.0), lit(0.01),
lit(0.0), lit(0.0), lit(1000.0), lit(1000.0), lit(100), lit(100), lit(32633)).alias("tin")
)
result.show(truncate = false)
+-----------------------------------------------------------+
|tin |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TIN-interpolated DTM over specified extent and pixel count)
rst_frombands
LightweightHeavyweightPowered by rasterio. Stacks an ARRAY of single-band tiles into one multi-band tile in array order (element 0 → band 1), preserving georeference/CRS/dtype/NoData from the first.
Create a raster from an array of band tiles.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_frombands(bands: Column): Column
Constructor — stacks per-band tiles into a multi-band tile. Example splits the multiband fixture into per-band tiles via rst_band, then re-stacks them into a 3-band tile.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT
gbx_rst_frombands(array(band1, band2, band3)) as multi_band
FROM separated_bands;
+-----------------------------------------------------------+
|multi_band |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
# Build an array of per-band tiles (band 1, band 2, band 3) then stack
with_bands = df.select(
f.array(
rx.rst_band("tile", f.lit(1)),
rx.rst_band("tile", f.lit(2)),
rx.rst_band("tile", f.lit(3)),
).alias("bands")
)
result = with_bands.select(rx.rst_frombands("bands").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(3 per-band tiles stacked back into a 3-band tile; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("multiband_rasters")
with_bands = tile_df.select(
f.array(
rx.rst_band("tile", f.lit(1)),
rx.rst_band("tile", f.lit(2)),
rx.rst_band("tile", f.lit(3)),
).alias("bands")
)
result = with_bands.select(rx.rst_frombands("bands").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(3 per-band tiles stacked back into a 3-band tile)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Constructor: extract per-band tiles from the multiband fixture, then stack them back.
val rasters = spark.table("multiband_rasters")
val withBands = rasters.select(
array(
rx.rst_band(col("tile"), lit(1)),
rx.rst_band(col("tile"), lit(2)),
rx.rst_band(col("tile"), lit(3))
).alias("bands")
)
val result = withBands.select(rx.rst_frombands(col("bands")).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(3 per-band tiles stacked back into a 3-band tile)
rst_fromcontent
LightweightHeavyweightPowered by rasterio. The driver defaults to GTiff when unspecified and outputs are re-encoded as GeoTIFF; readable input formats are limited to the GDAL build bundled with rasterio.
Create a raster from binary content.
Signature: rst_fromcontent(content: Column, driver: Column): Column
Parameters: content — Binary column; driver — GDAL driver name
Returns: Binary raster tile data
Constructor — builds a tile from binary content. Example uses Spark's binaryFile reader to load bytes, then rst_fromcontent to decode them — this is the canonical tier-agnostic pattern (works on any compute including Serverless).
rst_fromcontent materializes bytes that are already in a column. Those bytes are in executor memory before this call; GeoBrix does not guard their size. On Serverless / Spark Connect the connect-aware cap is 64 MiB per tile — keep individual content values at or under that limit. For larger sources, use the virtual-tile raster readers (the default) and let the runtime page bytes lazily. See Serverless & Memory.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Load from binary table
SELECT
path,
gbx_rst_fromcontent(content, 'GTiff') as tile
FROM binary_raster_table;
+----+-----------------------------------------------------------+
|path|tile |
+----+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+----+-----------------------------------------------------------+
from ._fixtures import single_band_path
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
path = str(single_band_path())
binary_df = spark.read.format("binaryFile").load(path)
tile_df = binary_df.select(
rx.rst_fromcontent(f.col("content"), f.lit("GTiff")).alias("tile")
)
result = tile_df.select(rx.rst_format("tile").alias("format")).first()
+------+
|format|
+------+
|GTiff |
+------+
(format of the tile loaded from binary content via binaryFile reader)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
from path_config import SAMPLE_DATA_BASE
path = f"{SAMPLE_DATA_BASE}/nyc/sentinel2/nyc_sentinel2_red.tif"
binary_df = spark.read.format("binaryFile").load(path)
tile_df = binary_df.select(
rx.rst_fromcontent(f.col("content"), f.lit("GTiff")).alias("tile")
)
result = tile_df.select(rx.rst_format("tile").alias("format")).first()
+------+
|format|
+------+
|GTiff |
+------+
(format of the tile loaded from binary content via binaryFile reader)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Constructor: binaryFile reader + rst_fromcontent is the canonical tier-agnostic pattern.
// binaryFile runs in Spark (holds the Volume credential) and works on any compute.
val binary = spark.read.format("binaryFile")
.load("/Volumes/main/default/test-data/geobrix-examples/nyc/sentinel2/nyc_sentinel2_red.tif")
val tiles = binary.select(rx.rst_fromcontent(col("content"), lit("GTiff")).alias("tile"))
val result = tiles.select(rx.rst_format(col("tile")).alias("format"))
result.show()
+------+
|format|
+------+
|GTiff |
+------+
(format of the tile loaded from binary content via binaryFile reader)
rst_fromfile
LightweightReference a raster on disk as a tile, by path.
The lightweight tier returns a virtual tile by default — bytes-free, pointing at path over its whole-file window, with width/height/CRS read from the header. No pixels are read until a downstream op needs them, so this is the lazy way to load rasters by path. There is no virtualize_dir/virtualize_prefix argument here: the source file already is the durable backing store, so nothing needs writing. A path that cannot be opened returns null.
Signature: rst_fromfile(path: Column, driver: Column = 'GTiff'[, materialize: bool = False]) — materialize is a lightweight-Python argument; from Python, materialize=True reads the pixels now and returns a materialized tile.
Parameters: path — raster file path; driver — GDAL driver-name hint carried into metadata (rasterio auto-detects the real format on open); materialize (light Python only) — True reads pixels now and returns a materialized tile, default False returns a virtual tile.
gbx_rst_fromfile is a Python UDF (requires a light-tier extra, e.g. geobrix[light_env6]; no Scala/JVM form — the executor JVM cannot read a UC Volume /Volumes/... FUSE path, whose credential is held only by Spark's managed Python worker). The SQL call is the same 2-argument form in both tiers, but the tier that registered decides the result: the lightweight registration (pyrx) returns a virtual tile, the heavyweight registration (rasterx) returns a materialized tile (JVM/heavy callers cannot use a virtual path-only tile). Whichever register() ran last wins. From Python, pass materialize=True to force bytes regardless of tier. Without a light-tier extra the function is not registered and the Python binding raises with guidance.
materialize=True errors over the Serverless capCalling rst_fromfile(..., materialize=True) on a file larger than the connect-aware stream cap (64 MiB on Serverless / Spark Connect, 256 MiB on classic) raises a ValueError. The error is intentional — silently materializing an oversized tile would OOM the executor. The virtual default (materialize=False, which is the lightweight default) is Serverless-safe: pixels are read lazily, size-gated, at each downstream operation. See Serverless & Memory.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- gbx_rst_fromfile is a Python UDF (no JVM form; requires a light-tier extra, e.g. geobrix[light_env6]). The
-- SQL call is the same 2-argument form in both tiers — the tier you register
-- decides the result (whichever register() ran last wins):
-- Lightweight (rx.register / pyrx): returns a VIRTUAL tile (bytes-free,
-- path + whole-file window; lazy).
-- Heavyweight (rasterx register): returns a MATERIALIZED tile (raster
-- bytes present) — JVM/heavy callers
-- cannot use a virtual path-only tile.
SELECT
gbx_rst_fromfile('/Volumes/main/geobrix_samples/nyc/sentinel2.tif', 'GTiff') AS tile;
-- Either way, accessors read what they need — width/height come from the header
-- (no pixel read even for the virtual tile):
SELECT
path,
gbx_rst_width(gbx_rst_fromfile(path, 'GTiff')) as width,
gbx_rst_height(gbx_rst_fromfile(path, 'GTiff')) as height
FROM raster_paths;
# Both tiers return the SAME v2 tile struct (cellid, raster, path, window, ...);
# only the field values differ — virtual carries the path, materialized the bytes.
# Lightweight registration — a VIRTUAL v2 tile (raster null; path + window set):
+---------------------------------------------------------------+
|tile |
+---------------------------------------------------------------+
|{0, null, /Volumes/..., {0, 0, 10980, 10980}, ..., null, {...}}|
+---------------------------------------------------------------+
# Heavyweight registration — a MATERIALIZED v2 tile (raster bytes; path null):
+------------------------------------------------------------+
|tile |
+------------------------------------------------------------+
|{0, <raster bytes>, null, null, ..., {driver -> GTiff, ...}}|
+------------------------------------------------------------+
# width/height (either tier) read from the header:
+----+-----+------+
|path|width|height|
+----+-----+------+
|... |10980|10980 |
+----+-----+------+
from ._fixtures import single_band_path
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
path = str(single_band_path())
path_df = spark.createDataFrame([(path,)], ["path"])
# Default — a VIRTUAL v2 tile (same struct as always, just bytes-free):
+------------------------------------------------------------+
|tile |
+------------------------------------------------------------+
|{0, null, /Volumes/..., {0, 0, 236, 161}, ..., null, {...}} |
+------------------------------------------------------------+
(raster is null → bytes-free; path + whole-file window carry the reference)
# rst_fromfile("path", materialize=True) instead returns a MATERIALIZED v2 tile:
# {0, <raster bytes>, null, null, ..., {driver -> GTiff, ...}} (raster set)
from ._fixtures import single_band_path
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
path = str(single_band_path())
path_df = spark.createDataFrame([(path,)], ["path"])
# Heavy rst_fromfile delegates to pyrx with materialize=True → bytes present.
return path_df.select(
rx.rst_fromfile("path", f.lit("GTiff")).alias("tile")
).first()["tile"]
+------------------------------------------------------------+
|tile |
+------------------------------------------------------------+
|{0, <raster bytes>, null, null, ..., {driver -> GTiff, ...}}|
+------------------------------------------------------------+
(a MATERIALIZED v2 tile: raster bytes present, path/window null)
Not available in this tier.
rst_fromfile pulls columnar raster paths into a tile column inline. To ingest rasters as a normal Spark job — partitioned parallel reads, optional tiling (sizeInMB), FUSE-safe Volume staging — use the Raster Reader (raster_gbx / gtiff_gbx, or heavyweight gdal / gtiff_gdal): spark.read.format("raster_gbx").load(path). It, too, emits virtual tiles by default.
rst_fromcontentIf you already hold raster bytes in a column (e.g. from Spark's built-in binaryFile reader), build the tile with gbx_rst_fromcontent(content, driver). Note this is not a lazy equivalent of rst_fromfile in the lightweight tier: rst_fromcontent takes bytes you have already read, so it always produces a materialized tile. (In the heavyweight tier all tiles are materialized, so there the two are interchangeable.) It also works without a light-tier extra and on any compute, since the binaryFile reader runs in Spark, which holds the Volume credential:
df = (
spark.read.format("binaryFile")
.load("/Volumes/main/geobrix_samples/geobrix-examples/nyc/*.tif")
.selectExpr("path", "gbx_rst_fromcontent(content, 'GTiff') AS tile")
)
rst_gridfrompoints
LightweightHeavyweightSee interpolation & fill to choose among sampling, resampling, and gap-filling.
Powered by rasterio + SciPy (cKDTree IDW). Inverse-distance interpolation (power, max_pts) to a single-band Float64 grid; NoData −9999. Matches the heavyweight invdist defaults (power=2.0, max_pts=12).
IDW-interpolate an array of Z-valued point geometries to a Float64 GeoTIFF tile covering an explicit bounding box and pixel grid. Supply the points and their scalar values as arrays in a single row; use rst_gridfrompoints_agg when points arrive one per row.
out_crs parameterThe lightweight Python binding accepts an optional trailing out_crs (string CRS) argument that the SQL and heavyweight/Scala tiers do not — the heavyweight RST_GridFromPoints builder is strictly ≤11-argument (…, out_srid, [power, [max_pts]]) and rejects a 12th argument. In lightweight Python, out_crs (string) wins over the int out_srid. This is a lightweight superset, not a heavyweight regression.
Signature: rst_gridfrompoints(points_array: Column, values_array: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width_px: Column, height_px: Column, out_srid: Column, [power: Column, [max_pts: Column]]): Column — output CRS via out_srid (int); input points are assumed already in the output CRS. See Coordinate Reference Systems.
Parameters: points_array — ARRAY<BINARY> of WKB point geometries; values_array — ARRAY<DOUBLE> of scalar observations, one per point; xmin/ymin/xmax/ymax — output extent in CRS units; width_px/height_px — output dimensions in pixels; out_srid — EPSG code; power — IDW distance-decay exponent (2.0 is the standard); max_pts — maximum nearest neighbours considered per output pixel
- SQL
- Python (light)
- Python (heavy)
- Scala
-- IDW (power=2, max_points=12) from arrays of point WKB and values.
-- Output is a 256 x 256 Float64 GTiff covering the requested extent.
SELECT gbx_rst_gridfrompoints(
points_wkb_array, values_array,
0.0, 0.0, 1000.0, 1000.0,
256, 256, 32633
) AS idw
FROM point_clouds;
+-----------------------------------------------------------+
|idw |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(IDW-interpolated tile over specified extent)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import ArrayType, BinaryType, DoubleType
rx.register(spark)
# Create a synthetic point cloud DataFrame with WKB points and values
# Using simple WKB POINT(2.0, 2.0) and POINT(4.0, 4.0)
point_data = [
(
[
bytes.fromhex("010100000000000000000000400000000000000040"),
bytes.fromhex("010100000000000000000008400000000000000840"),
],
[100.0, 110.0],
)
]
df = spark.createDataFrame(
point_data,
[
"points_wkb_array",
"values_array",
],
)
result = df.select(
rx.rst_gridfrompoints(
f.col("points_wkb_array"),
f.col("values_array"),
f.lit(0.0),
f.lit(0.0),
f.lit(1000.0),
f.lit(1000.0),
f.lit(256),
f.lit(256),
f.lit(32633),
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(IDW-interpolated tile over specified extent)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
# Create synthetic point cloud: 2 points with WKB and values
point_data = [
(
[
bytes.fromhex("010100000000000000000004400000000000000040"),
bytes.fromhex("010100000000000000000008400000000000000040"),
],
[100.0, 110.0],
)
]
df = spark.createDataFrame(point_data, ["points_wkb_array", "values_array"])
result = df.select(
rx.rst_gridfrompoints(
f.col("points_wkb_array"),
f.col("values_array"),
f.lit(0.0),
f.lit(0.0),
f.lit(1000.0),
f.lit(1000.0),
f.lit(256),
f.lit(256),
f.lit(32633),
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(IDW-interpolated tile over specified extent)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Create synthetic point cloud with WKB-encoded points and values
val points = spark.createDataFrame(Seq(
(Array[Array[Byte]](/* WKB point 1 */, /* WKB point 2 */), Array[Double](100.0, 110.0))
)).toDF("points_wkb", "values")
val result = points.select(
rx.rst_gridfrompoints(col("points_wkb"), col("values"), lit(0.0), lit(0.0),
lit(1000.0), lit(1000.0), lit(256), lit(256), lit(32633)).alias("idw")
)
result.show(truncate = false)
+-----------------------------------------------------------+
|idw |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(IDW-interpolated tile over specified extent)
LiDAR / Point-Cloud DSM
Four functions for building raster products from LiDAR point clouds. Available on both tiers (pyrx lightweight and rasterx heavyweight). Load point clouds with the lidar_gbx reader. For an end-to-end walkthrough — .laz → DSM/DTM binning → Canopy Height Model — see the Wireless Coverage example.
rst_binpoints
LightweightHeavyweightBin a per-row ARRAY<DOUBLE> of (x, y, z) coordinate triples into a single-band
Float32 raster tile. Each pixel carries the reduction of all points whose centroid falls
within the pixel cell. Empty cells carry NoData (−9999). The bounding box uses a
half-open interval on the upper x/y edges to prevent double-counting when DSM tiles
are assembled from adjacent strips.
What it computes. This is the point-cloud→raster (DSM-building) step: every input
point is dropped into the grid cell its (x, y) falls in, and the pixel value is the chosen
reduction of the z-values that landed there. With the default "max" that is the tallest
return per cell — a surface model (DSM), exactly the input rst_chm differences
against a DEM; "min"/"mean"/"median"/"count"/"percentile:<p>" reduce the same
points into other surfaces.
Signature: rst_binpoints(x_array, y_array, z_array, xmin, ymin, xmax, ymax, width_px, height_px, srid, [statistic])
Parameters: x_array, y_array, z_array — parallel ARRAY<DOUBLE> columns of point coordinates and z-values; xmin/ymin/xmax/ymax — extent in CRS units; width_px/height_px — output pixel dimensions; srid — EPSG code; statistic — one of "max" (default, highest return for DSMs), "min", "mean", "median", "count", or "percentile:<p>" (e.g. "percentile:90").
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_binpoints(
array(550100.0, 550200.0, 550300.0),
array(180100.0, 180200.0, 180500.0),
array(42.5, 45.1, 38.7),
550000.0, 180000.0, 551000.0, 181000.0,
10, 10, 27700, 'max'
) AS dsm;
+-----------------------------------------------------------+
|dsm |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band Float32 DSM tile: max-z per 100 m cell over the 1 km extent)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.range(1).select(
f.array(f.lit(550100.0), f.lit(550200.0), f.lit(550300.0)).alias("x"),
f.array(f.lit(180100.0), f.lit(180200.0), f.lit(180500.0)).alias("y"),
f.array(f.lit(42.5), f.lit(45.1), f.lit(38.7)).alias("z"),
)
result = df.select(
rx.rst_binpoints(
"x", "y", "z",
f.lit(550000.0), f.lit(180000.0),
f.lit(551000.0), f.lit(181000.0),
f.lit(10), f.lit(10),
f.lit(27700),
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(Float32 10×10 BNG tile; three points binned by max z-value, empty cells = NoData -9999.0; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.range(1).select(
f.array(f.lit(550100.0), f.lit(550200.0), f.lit(550300.0)).alias("x"),
f.array(f.lit(180100.0), f.lit(180200.0), f.lit(180500.0)).alias("y"),
f.array(f.lit(42.5), f.lit(45.1), f.lit(38.7)).alias("z"),
)
result = df.select(
rx.rst_binpoints(
"x", "y", "z",
f.lit(550000.0), f.lit(180000.0),
f.lit(551000.0), f.lit(181000.0),
f.lit(10), f.lit(10),
f.lit(27700),
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(Float32 10×10 BNG tile; three points binned by max z-value, empty cells = NoData -9999.0)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Inline 3-point BNG (EPSG:27700) set — no external raster file needed.
val points = spark.range(1).select(
array(lit(550100.0), lit(550200.0), lit(550300.0)).alias("x"),
array(lit(180100.0), lit(180200.0), lit(180500.0)).alias("y"),
array(lit(42.5), lit(45.1), lit(38.7)).alias("z")
)
val result = points.select(
rx.rst_binpoints(
col("x"), col("y"), col("z"),
lit(550000.0), lit(180000.0),
lit(551000.0), lit(181000.0),
lit(10), lit(10),
lit(27700)
).alias("tile")
)
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(Float32 10×10 BNG tile; three points binned by max z-value, empty cells = NoData -9999.0)
rst_binpoints_agg
LightweightHeavyweight Grouped-agg UDFAggregator variant of rst_binpoints: stream one scalar (x, y, z) point
per row into a DSM tile via groupBy(...).agg(rx.rst_binpoints_agg(...)). The
xmin/ymin/xmax/ymax/width_px/height_px/srid/statistic arguments are
per-group constants. The per-cell binning is identical to rst_binpoints —
see What it computes there for the reduction semantics and statistic modes — it just
streams one point per row instead of a per-row array.
The lightweight SQL aggregate returns BINARY (raster bytes). Wrap with
gbx_rst_fromcontent(<agg>, 'GTiff') to rebuild a tile struct. The lightweight Python
wrapper rx.rst_binpoints_agg(...) returns the full tile struct.
Signature: rst_binpoints_agg(x, y, z, xmin, ymin, xmax, ymax, width_px, height_px, srid, [statistic])
Parameters: x, y — scalar DOUBLE columns of point coordinates; z — scalar DOUBLE z-value; remaining parameters match rst_binpoints.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT tile_id,
gbx_rst_binpoints_agg(x, y, z, 550000.0, 180000.0, 551000.0, 181000.0, 10, 10, 27700, 'max') AS dsm
FROM (
VALUES (1, 550100.0, 180100.0, 42.5), (1, 550200.0, 180200.0, 45.1), (1, 550300.0, 180500.0, 38.7)
) AS t(tile_id, x, y, z)
GROUP BY tile_id;
+-------+--------------+
|tile_id|dsm |
+-------+--------------+
|1 |<raster bytes>|
+-------+--------------+
(BINARY raster bytes — wrap with gbx_rst_fromcontent(dsm, 'GTiff') to rebuild a tile struct)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
DoubleType,
StringType,
StructField,
StructType,
)
rows = [
("R1", 550100.0, 180100.0, 42.5),
("R1", 550200.0, 180200.0, 45.1),
("R1", 550300.0, 180500.0, 38.7),
]
schema = StructType(
[
StructField("region", StringType()),
StructField("x", DoubleType()),
StructField("y", DoubleType()),
StructField("z", DoubleType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_binpoints_agg(
"x",
"y",
"z",
f.lit(550000.0),
f.lit(180000.0),
f.lit(551000.0),
f.lit(181000.0),
f.lit(10),
f.lit(10),
f.lit(27700),
).alias("tile")
)
.first()
)
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(Float32 10x10 BNG tile; three scalar points binned by max z-value, empty cells = NoData -9999.0)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
DoubleType,
StringType,
StructField,
StructType,
)
rows = [
("R1", 550100.0, 180100.0, 42.5),
("R1", 550200.0, 180200.0, 45.1),
("R1", 550300.0, 180500.0, 38.7),
]
schema = StructType(
[
StructField("region", StringType()),
StructField("x", DoubleType()),
StructField("y", DoubleType()),
StructField("z", DoubleType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_binpoints_agg(
"x",
"y",
"z",
f.lit(550000.0),
f.lit(180000.0),
f.lit(551000.0),
f.lit(181000.0),
f.lit(10),
f.lit(10),
f.lit(27700),
).alias("tile")
)
.first()
)
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(Float32 10x10 BNG tile; three scalar points binned by max z-value, empty cells = NoData -9999.0)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
rx.register(spark)
// Multi-row fixture: 3 BNG (EPSG:27700) scalar x/y/z rows over a 1 km extent at 10×10 pixels.
val schema = StructType(Seq(
StructField("region", StringType()), StructField("x", DoubleType()),
StructField("y", DoubleType()), StructField("z", DoubleType())))
val rows = Seq(("R1",550100.0,180100.0,42.5),("R1",550200.0,180200.0,45.1),("R1",550300.0,180500.0,38.7))
val df = spark.createDataFrame(spark.sparkContext.parallelize(rows.map(r => org.apache.spark.sql.Row(r._1,r._2,r._3,r._4))), schema)
val result = df.groupBy("region").agg(
rx.rst_binpoints_agg(col("x"), col("y"), col("z"),
lit(550000.0), lit(180000.0), lit(551000.0), lit(181000.0),
lit(10), lit(10), lit(27700)).alias("tile")
)
result.show()
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(Float32 10x10 BNG tile; three scalar points binned by max z-value, empty cells = NoData -9999.0)
Both rst_binpoints_agg (grouped — realizes a whole group's points in the worker) and
rst_binpoints (scalar — realizes a row's point arrays) hold memory proportional to the
points per group. That is fine at moderate density, but a full-resolution tile (millions of
returns — e.g. a 1 km tile of 1 m LiDAR) can exhaust worker RAM.
For large or full-density point clouds, use the bounded light-tier helper
rx.bin_points_tiled(df, x, y, z, by=[…], xmin=…, ymin=…, xmax=…, ymax=…, width, height, srid, stat).
It runs a two-stage native aggregation — per-point pixel index → native groupBy(tile, cell)
reduction → placement — so peak memory is the raster grid (W×H), independent of point count.
It returns the same tile struct (drops straight into rst_chm), and its output is identical to
groupBy(by).agg(rst_binpoints_agg(…)) for max/min/mean/count. median/percentile (which
need every value in a cell) delegate to the buffered aggregator — decimate or use smaller tiles for
those. Prefer bin_points_tiled over a collect_list(all points) → rst_binpoints pattern, which
buffers the whole cloud. See Performance and the LiDAR reader.
rst_isoband
LightweightHeavyweightReclassify a raster band into value bins defined by a breaks array and return one
polygon per contiguous patch of pixels in the same bin. Useful for producing
height-band polygons from a CHM or DSM, or elevation isobands from a DEM.
Returns ARRAY<STRUCT<geom_wkb BINARY, band INT, lower DOUBLE, upper DOUBLE>> — one
entry per contiguous region. geom_wkb is a WKB Polygon in the raster's own CRS.
NoData pixels are excluded. Pixels below breaks[0] or >= breaks[-1] are dropped.
Each bin is half-open — [breaks[i], breaks[i+1]) — closed on the lower bound and
open on the upper, so a value equal to a break belongs to the band that break opens.
For example, with breaks = array(0, 50, 100, 150, 200), a value of 55.0 — and 50.0
itself — falls in the [50.0, 100.0) band (band index 1), while 100.0 moves up to
the [100.0, 150.0) band and 200.0 (equal to breaks[-1]) is dropped.
Implemented with numpy.digitize + rasterio.features.shapes — no GDAL required.
Signature: rst_isoband(tile, breaks)
Parameters: tile — tile struct column; breaks — ARRAY<DOUBLE> of N+1 strictly-ascending boundary values defining N bands.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_isoband(tile, array(0.0, 50.0, 100.0, 150.0, 200.0)) AS patches
FROM dem_rasters;
+-----------------------------------------------------------+
|patches |
+-----------------------------------------------------------+
|[{[BINARY], 0, 0.0, 50.0}, {[BINARY], 1, 50.0, 100.0}, ...]|
+-----------------------------------------------------------+
(array of per-patch structs: geom_wkb WKB polygon in raster CRS, band index, lower/upper break values)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from ._fixtures import dem_tile_df
df = dem_tile_df(spark)
result = df.select(
rx.rst_isoband(
"tile",
f.array(
f.lit(0.0), f.lit(50.0), f.lit(100.0),
f.lit(150.0), f.lit(200.0), f.lit(311.0),
),
).alias("patches")
).first()
+-----------------------------------------------------------+
|patches |
+-----------------------------------------------------------+
|[{[BINARY], 0, 0.0, 50.0}, {[BINARY], 1, 50.0, 100.0}, ...]|
+-----------------------------------------------------------+
(ARRAY of per-patch structs: geom_wkb WKB polygon in raster CRS, band index, lower/upper break values)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
from ._fixtures import dem_tile_df_heavy
rx.register(spark)
df = dem_tile_df_heavy(spark)
result = df.select(
rx.rst_isoband(
"tile",
f.array(
f.lit(0.0), f.lit(50.0), f.lit(100.0),
f.lit(150.0), f.lit(200.0), f.lit(311.0),
),
).alias("patches")
).first()
+-----------------------------------------------------------+
|patches |
+-----------------------------------------------------------+
|[{[BINARY], 0, 0.0, 50.0}, {[BINARY], 1, 50.0, 100.0}, ...]|
+-----------------------------------------------------------+
(ARRAY of per-patch structs: geom_wkb WKB polygon in raster CRS, band index, lower/upper break values)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Reclassify DEM elevation into five 50-metre bands [0,50),[50,100),...,[200,311).
// Returns ARRAY<struct(geom_wkb BINARY, band INT, lower DOUBLE, upper DOUBLE)>.
val dem = spark.table("dem_rasters")
val breaks = array(lit(0.0), lit(50.0), lit(100.0), lit(150.0), lit(200.0), lit(311.0))
val result = dem.select(rx.rst_isoband(col("tile"), breaks).alias("patches"))
result.show(truncate = false)
+-----------------------------------------------------------+
|patches |
+-----------------------------------------------------------+
|[{[BINARY], 0, 0.0, 50.0}, {[BINARY], 1, 50.0, 100.0}, ...]|
+-----------------------------------------------------------+
(ARRAY of per-patch structs: geom_wkb WKB polygon in raster CRS, band index, lower/upper break values)
rst_chm
LightweightHeavyweightCompute a Canopy Height Model (CHM) from a Digital Surface Model (DSM) tile and a
Digital Elevation Model (DEM) tile: CHM = clamp(align(DSM → DEM grid) − DEM, min=0).
What it computes. The DSM holds the height of the top surface in each cell (canopy,
rooftops) and the DEM holds bare-earth ground elevation, so subtracting one from the other
leaves the height of whatever stands above the ground — canopy or object height — in the
DEM's units. It is a per-cell subtraction, not an aggregation: taking the max z per cell
is how you build the DSM upstream (e.g. rst_binpoints over LiDAR returns), and rst_chm
then differences that surface against the DEM.
The DSM is warped onto the DEM's grid (CRS, extent, width, height) via nearest-neighbour resampling before subtraction. Negative differences (where the DSM dips below the DEM due to noise) are clamped to 0. NoData in either input propagates to the output. Output is a Float32 GeoTIFF tile with NoData −9999.
Signature: rst_chm(dsm_tile, dem_tile)
Parameters: dsm_tile — Digital Surface Model tile (defines the surface heights); dem_tile — Digital Elevation Model tile (defines the bare-earth reference; also defines the output grid).
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_chm(tile, tile) AS chm FROM dem_rasters;
+-----------------------------------------------------------+
|chm |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(Float32 CHM tile: clamp(DSM - DEM, min=0); NoData propagates from either input)
from databricks.labs.gbx.pyrx import functions as rx
from ._fixtures import dem_tile_df
df = dem_tile_df(spark)
result = df.select(rx.rst_chm("tile", "tile").alias("chm")).first()
+-----------------------------------------------------------+
|chm |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(Float32 CHM tile: clamp(DSM - DEM, min=0); NoData propagates from either input; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from ._fixtures import dem_tile_df_heavy
rx.register(spark)
df = dem_tile_df_heavy(spark)
result = df.select(rx.rst_chm("tile", "tile").alias("chm")).first()
+-----------------------------------------------------------+
|chm |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(Float32 CHM tile: clamp(DSM - DEM, min=0); NoData propagates from either input)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Compute CHM = clamp(align(DSM -> DEM) - DEM, min=0).
// Using the same tile for both is a degenerate (all-zero) but valid example.
// In production: pass a LiDAR-derived DSM (rst_binpoints over LiDAR returns)
// and a matching DEM tile to produce real canopy heights.
val dem = spark.table("dem_rasters")
val result = dem.select(rx.rst_chm(col("tile"), col("tile")).alias("chm"))
result.show(truncate = false)
+-----------------------------------------------------------+
|chm |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(Float32 CHM tile: clamp(DSM - DEM, min=0); NoData propagates from either input)
Generator Functions
Produce multiple tiles or bands (7 total).
rst_bng_tessellate
LightweightHeavyweight Streaming UDTFThe lightweight implementation is backed by pygx._bng cell math and rasterio. It auto-warps the input to EPSG:27700 via rasterio.warp, enumerates the overlapping BNG cells, and clips one tile per cell — rendering STRING BNG cell IDs at the row boundary and matching the heavyweight cell set and clip windows.
Tessellate a raster to British National Grid (BNG) cells — one row per overlapping cell, each clipped to that cell's extent. The raster is automatically reprojected to EPSG:27700 (British National Grid) before tessellation, so any projected or geographic input is handled transparently. Resolution accepts integer indices ±1..±6 (1 = 100 km, 2 = 10 km, 3 = 1 km, 4 = 100 m, 5 = 10 m, 6 = 1 m; negative indices address quadrant sub-cells) or string keys from BNG.resolutionMap (e.g. "1km", "100m"). Cell IDs are STRING (e.g. "TQ28", "SU3412").
Natural fit for CV image-tiling workflows that require Ordnance Survey–scale cells aligned to the national grid — for example tiling aerial or satellite imagery into 1 km BNG cells for object-detection model inference. See issue #49 for the background.
Signature: rst_bng_tessellate(tile: Column, resolution: Column, [assignment: Column], [coverage: Column]): Column — assignment is "centroid" (default) or "covering"; coverage is "complete" (default) or "sparse".
SQL:
Since v0.4.3- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: the generator in SELECT explodes to one row per overlapping BNG
-- cell. 3rd arg is assignment ('covering' shown; default is 'centroid').
-- '1km' == integer resolution 3; a raster in any CRS is warped to EPSG:27700 first.
SELECT gbx_rst_bng_tessellate(tile, '1km', 'covering') FROM rasters;
-- Lightweight (pyrx): registered as a streaming table function — call with LATERAL.
SELECT t.* FROM rasters, LATERAL gbx_rst_bng_tessellate(tile, '1km', 'covering') t;
-- Explicit: assignment='centroid' (default) + coverage='complete' (default).
SELECT t.* FROM rasters, LATERAL gbx_rst_bng_tessellate(tile, '1km', 'centroid', 'complete') t;
+------+------+--------------+
|source|cellid|raster |
+------+------+--------------+
|... |TQ2979|<raster bytes>|
+------+------+--------------+
(SELECT t.* expands the v2-tile struct; cellid is the BNG grid-square STRING)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
# A 2km x 2km square over central London, in EPSG:27700 metres.
london_wkt = (
"POLYGON((529000 179000, 531000 179000, "
"531000 181000, 529000 181000, 529000 179000))"
)
london = spark.range(1).select(
rx.rst_rasterize(
f.lit(london_wkt),
f.lit(1.0),
f.lit(529000.0),
f.lit(179000.0),
f.lit(531000.0),
f.lit(181000.0),
f.lit(200),
f.lit(200),
f.lit(27700),
).alias("tile")
)
london.createOrReplaceTempView("rasters")
return spark.sql(
"SELECT t.* FROM rasters, LATERAL gbx_rst_bng_tessellate(tile, 3) t"
).take(3)
+------+-----------------------------------------------------------+
|cellid|... |
+------+-----------------------------------------------------------+
|TQ2979|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
|TQ2980|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
|TQ3079|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2-Tile row per 1km BNG cell overlapping the London raster; cellid is the
BNG grid-square STRING)
from pyspark.sql import functions as f
if rx is None:
raise ImportError("rasterx not installed")
london_wkt = (
"POLYGON((529000 179000, 531000 179000, "
"531000 181000, 529000 181000, 529000 179000))"
)
df = spark.range(1).select(
rx.rst_rasterize(
f.lit(london_wkt),
f.lit(1.0),
f.lit(529000.0),
f.lit(179000.0),
f.lit(531000.0),
f.lit(181000.0),
f.lit(200),
f.lit(200),
f.lit(27700),
).alias("tile")
)
+-----------------------------------------------------------+
|bng_cell |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(one v2-tile-struct row per 1km BNG cell overlapping the London raster; the
generator explodes to rows in a plain select)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_bng_tessellate(col("tile"), lit(3)).alias("bng_cells"))
result.show(truncate = false)
+---+
|bng|
+---+
|[{ |
+---+
(array of tile structs per BNG cell; raster rewarped to EPSG:27700)
rst_h3_tessellate
LightweightHeavyweight Streaming UDTFPowered by rasterio + h3. Clips the raster to each overlapping H3 cell at resolution, returning one row per cell via streaming UDTF (matches the heavyweight generator behavior).
Signature: rst_h3_tessellate(tile: Column, resolution: Column, [assignment: Column], [coverage: Column]): Column — Tessellate raster to H3 cells. assignment is "centroid" (default — pixel-centroid single-assignment) or "covering" (every overlapping cell, clipped); coverage is "complete" (default) or "sparse". See Grid raster tessellation for the full guide.
Covering mode uses a positive-area overlap rule (shared by all three grids — rst_h3_tessellate, rst_quadbin_tessellate, rst_bng_tessellate): a cell is emitted iff its geometry has greater-than-zero area overlap with the raster. A cell that merely touches the raster along a boundary edge or corner (zero pixel overlap — common on grid-aligned tiles where the raster edges land on cell boundaries) is excluded. A within-extent cell whose pixels are all NoData is emitted — NoData renders in place, it does not punch a gap into the mosaic. This rule is identical on the lightweight and heavyweight tiers, so covering-mode cell sets match exactly across tiers.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: the generator in SELECT explodes to one row per overlapping H3
-- cell (default assignment='centroid', coverage='complete').
SELECT gbx_rst_h3_tessellate(tile, 7, 'covering') FROM rasters;
-- Lightweight (pyrx): registered as a streaming table function — call with LATERAL.
SELECT t.* FROM rasters, LATERAL gbx_rst_h3_tessellate(tile, 7, 'covering') t;
-- Explicit: assignment='centroid' (default) + coverage='complete' (default).
SELECT t.* FROM rasters, LATERAL gbx_rst_h3_tessellate(tile, 7, 'centroid', 'complete') t;
+------+------------------+--------------+
|source|cellid |raster |
+------+------------------+--------------+
|... |599686042433355775|<raster bytes>|
+------+------------------+--------------+
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
single_band_tile_df(spark).createOrReplaceTempView("rasters")
return spark.sql(
"SELECT t.* FROM rasters, LATERAL gbx_rst_h3_tessellate(tile, 3) t"
).take(3)
+-------------------+-----------------------------------------------------------+
|cellid |... |
+-------------------+-----------------------------------------------------------+
|577586652210266111 |{..., <raster bytes>, ..., {driver -> GTiff, ...}} |
+-------------------+-----------------------------------------------------------+
(one v2-Tile row per H3 cell, cellid = the H3 index)
from pyspark.sql import functions as f
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("rasters")
# Partition into H3 cells (covering mode, returns array of structs)
result = df.select(rx.rst_h3_tessellate("tile", f.lit(7)).alias("h3_cells")).first()
+---+
|h3_|
+---+
|[{ |
+---+
(array of structs: [{cellid: LONG, raster: BINARY}, ...])
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_h3_tessellate(col("tile"), lit(7)).alias("h3_cells"))
result.show(truncate = false)
+---+
|h3_|
+---+
|[{ |
+---+
(array of structs: [{cellid: LONG, raster: BINARY}, ...])
rst_maketiles
LightweightHeavyweight Streaming UDTFPowered by rasterio. Streams one tile row per subdivided region via streaming UDTF. It derives a square tile size from the MB budget and always partitions; it does not honor the heavyweight size_in_mb = -1 (single tile) or 0 (64 MB) sentinels or the power-of-four split, so tile counts and dimensions differ.
Signature: rst_maketiles(tile: Column, sizeInMB: Column): Column — Subdivide into smaller tiles by approximate size in MB.
SQL:
Since v0.2.0- SQL
- Python (light)
- Python (heavy)
- Scala
-- Subdivide into MB-sized tiles using LATERAL. The second argument is a target
-- size in MB, not pixel dimensions; the tile grid is derived from the MB budget.
SELECT t.*
FROM rasters,
LATERAL gbx_rst_maketiles(tile, 4) t;
+------+--------------+----+----------------------+
|cellid|raster |path|... |
+------+--------------+----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+----+----------------------+
(one row per sub-tile; t.* expands the v2-Tile struct fields)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
single_band_tile_df(spark).createOrReplaceTempView("rasters")
# Subdivide raster into approximately 1.0 MB tiles
return spark.sql(
"SELECT t.* FROM rasters, " "LATERAL gbx_rst_maketiles(tile, 1.0) t"
).collect()
+------+--------------+-----+----------------------+
|cellid|raster |path |... |
+------+--------------+-----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+-----+----------------------+
(one row per sub-tile; t.* expands the v2-Tile struct fields)
if rx is None:
raise ImportError("rasterx not installed")
# The `rasters` view is created in Setup; subdivide into ~1.0 MB tiles via SQL LATERAL.
return spark.sql(
"SELECT t.* FROM rasters, " "LATERAL gbx_rst_maketiles(tile, 1.0) t"
).collect()
+------+--------------+-----+----------------------+
|cellid|raster |path |... |
+------+--------------+-----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+-----+----------------------+
(one row per sub-tile; t.* expands the v2-Tile struct fields)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_maketiles(col("tile"), lit(1.0)).alias("tiles"))
result.show(truncate = false)
+-----+
|tiles|
+-----+
|[{0, |
+-----+
(array of tile structs per MB subdivision)
rst_quadbin_tessellate
LightweightHeavyweight Streaming UDTFThe lightweight implementation is backed by pygx._quadbin cell math and rasterio. It enumerates the overlapping quadbin cells for the raster bbox and clips one tile per cell, matching the heavyweight cell set and clip windows.
Tessellate a raster to CARTO quadbin v0 cells — one row per overlapping cell, each clipped to that cell's extent. The input raster must be in EPSG:4326 (lon/lat); reproject upstream with rst_transform if your source CRS differs. Resolution is the quadbin zoom level (0..26). Each output row carries the quadbin cell ID (BIGINT) and the clipped tile struct.
Signature: rst_quadbin_tessellate(tile: Column, resolution: Column, [assignment: Column], [coverage: Column]): Column — assignment is "centroid" (default) or "covering"; coverage is "complete" (default) or "sparse".
SQL:
Since v0.4.3- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: the generator in SELECT explodes to one row per overlapping
-- quadbin cell, each chip clipped to its cell. 3rd arg is assignment
-- ('covering' shown; default is 'centroid'). Zoom 12 for a city-scale raster.
SELECT gbx_rst_quadbin_tessellate(tile, 12, 'covering') FROM rasters;
-- Lightweight (pyrx): registered as a streaming table function — call with LATERAL.
SELECT t.* FROM rasters, LATERAL gbx_rst_quadbin_tessellate(tile, 12, 'covering') t;
-- Explicit: assignment='centroid' (default) + coverage='complete' (default).
SELECT t.* FROM rasters, LATERAL gbx_rst_quadbin_tessellate(tile, 12, 'centroid', 'complete') t;
+------+-------------------+--------------+
|source|cellid |raster |
+------+-------------------+--------------+
|... |5250127588525215743|<raster bytes>|
+------+-------------------+--------------+
(SELECT t.* expands the v2-tile struct; cellid is the quadbin index)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
single_band_tile_df(spark).createOrReplaceTempView("rasters")
return spark.sql(
"SELECT t.* FROM rasters, LATERAL gbx_rst_quadbin_tessellate(tile, 5) t"
).take(3)
+-------------------+-----------------------------------------------------------+
|cellid |... |
+-------------------+-----------------------------------------------------------+
|5250127588525215743|{..., <raster bytes>, ..., {driver -> GTiff, ...}} |
+-------------------+-----------------------------------------------------------+
(one v2-Tile row per quadbin cell, cellid = the quadbin index)
from pyspark.sql import functions as f
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("rasters")
# Partition into quadbin cells at zoom 12 (covering mode)
result = df.select(
rx.rst_quadbin_tessellate("tile", f.lit(12)).alias("qb_cells")
).first()
+---+
|qb_|
+---+
|[{ |
+---+
(array of tile structs per quadbin cell, zoom 12)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_quadbin_tessellate(col("tile"), lit(12)).alias("qb_cells"))
result.show(truncate = false)
+---+
|qb_|
+---+
|[{ |
+---+
(array of tile structs per quadbin cell, zoom 12)
rst_custom_tessellate
LightweightHeavyweight Streaming UDTFThe lightweight implementation is backed by pygx custom-grid cell math and rasterio. It enumerates the overlapping custom-grid cells for the raster and clips one tile per cell, emitting Long-keyed chips (custom cell IDs as BIGINT). The raster is expected to be in the custom grid's native CRS — no automatic reprojection is performed; reproject upstream with rst_transform if needed.
Tessellate a raster to user-defined custom-grid cells — one row per overlapping cell, each clipped to that cell's extent. Pass a custom-grid struct (from gbx_custom_grid(...)) and a resolution integer that selects the cell size level. Cell IDs are BIGINT encoded by the custom-grid layout. For the covering-mode overlap rule (positive-area overlap, boundary touches excluded), see rst_h3_tessellate.
Signature: rst_custom_tessellate(tile: Column, grid: Column, resolution: Column, [assignment: Column], [coverage: Column]): Column — assignment is "centroid" (default) or "covering"; coverage is "complete" (default) or "sparse".
SQL:
Since v0.5.1- SQL
- Python (light)
- Python (heavy)
- Scala
-- Default: centroid assignment, complete coverage (one chip per cell).
SELECT t.*
FROM (SELECT gbx_rst_fromfile('{_CUSTOM_RASTER_PATH}') AS tile) AS r,
LATERAL gbx_rst_custom_tessellate(
tile,
{_CUSTOM_GRID_LITERAL},
1
) t;
-- Explicit covering assignment (each pixel contributes to all overlapping cells).
SELECT t.*
FROM (SELECT gbx_rst_fromfile('{_CUSTOM_RASTER_PATH}') AS tile) AS r,
LATERAL gbx_rst_custom_tessellate(
tile,
{_CUSTOM_GRID_LITERAL},
1, 'covering', 'sparse'
) t;
+--------------+------------------------------------------------------+
|cellid |raster |
+--------------+------------------------------------------------------+
|<Long cell id>|{<Long>, <raster bytes>, null, {driver -> GTiff, ...}}|
+--------------+------------------------------------------------------+
(one v2 tile struct per cell; cellid is BIGINT encoding the custom cell)
from databricks.labs.gbx.pyrx import functions as rx
from databricks.labs.gbx.pygx import functions as gx
from pyspark.sql import functions as f
rx.register(spark)
gx.register(spark)
# A 2km x 2km square over central London, in EPSG:27700 metres.
london_wkt = (
"POLYGON((529000 179000, 531000 179000, "
"531000 181000, 529000 181000, 529000 179000))"
)
london = spark.range(1).select(
rx.rst_rasterize(
f.lit(london_wkt),
f.lit(1.0),
f.lit(529000.0),
f.lit(179000.0),
f.lit(531000.0),
f.lit(181000.0),
f.lit(200),
f.lit(200),
f.lit(27700),
).alias("tile")
)
london.createOrReplaceTempView("london_rasters")
# Custom grid exactly covering the raster extent in EPSG:27700, resolution 1
# (four 1km cells subdividing the 2km root cell).
custom_grid_sql = "gbx_custom_grid(529000, 531000, 179000, 181000, 2, 2000, 2000, 27700)"
return spark.sql(
f"SELECT t.* FROM london_rasters, "
f"LATERAL gbx_rst_custom_tessellate(tile, {custom_grid_sql}, 1) t"
).take(3)
+--------------------+-----------------------------------------------------------+
|cellid |raster |
+--------------------+-----------------------------------------------------------+
|<custom cell bigint>|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+--------------------+-----------------------------------------------------------+
(one v2-Tile row per custom-grid cell; cellid is BIGINT encoding the custom cell)
if rx is None:
raise ImportError("rasterx not installed")
from pyspark.sql import functions as f
_LONDON_4KM_WKT = (
"POLYGON((529000 179000, 533000 179000, "
"533000 183000, 529000 183000, 529000 179000))"
)
df = spark.range(1).select(
rx.rst_rasterize(
f.lit(_LONDON_4KM_WKT),
f.lit(1.0),
f.lit(529000.0), f.lit(179000.0),
f.lit(533000.0), f.lit(183000.0),
f.lit(40), f.lit(40), f.lit(27700),
).alias("tile")
)
df.createOrReplaceTempView("bng_london_rasters")
# Custom grid covering the same 4km BNG extent; resolution 1 = four 2km cells.
custom_grid_sql = "gbx_custom_grid(529000, 533000, 179000, 183000, 2, 4000, 4000, 27700)"
return spark.sql(
f"SELECT t.* FROM bng_london_rasters, "
f"LATERAL gbx_rst_custom_tessellate(tile, {custom_grid_sql}, 1) t"
).take(3)
+--------------------+-----------------------------------------------------------+
|cellid |raster |
+--------------------+-----------------------------------------------------------+
|<custom cell bigint>|{0, <raster bytes>, null, {driver -> GTiff, ...}} |
+--------------------+-----------------------------------------------------------+
(one v2-Tile row per custom-grid cell; cellid is BIGINT encoding the custom cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Synthetic 2km London raster (EPSG:27700); custom grid with the same CRS and extent.
val londonWkt = "POLYGON((529000 179000, 531000 179000, 531000 181000, 529000 181000, 529000 179000))"
val raster = spark.range(1).select(
rx.rst_rasterize(lit(londonWkt), lit(1.0),
lit(529000.0), lit(179000.0), lit(531000.0), lit(181000.0),
lit(200), lit(200), lit(27700)).alias("tile"))
// Custom grid covering the same EPSG:27700 extent as the raster.
val customGrid = call_function("gbx_custom_grid",
lit(529000), lit(531000), lit(179000), lit(181000), lit(2), lit(2000), lit(2000), lit(27700))
val result = raster.select(rx.rst_custom_tessellate(col("tile"), customGrid, lit(1)).alias("chips"))
result.show(truncate = false)
[Row(cellid=<bigint>, raster=Row(...)), ...]
(one v2-Tile Row per custom-grid cell; cellid is BIGINT encoding the custom cell)
rst_retile
LightweightHeavyweight Streaming UDTFPowered by rasterio. Streams one tile row per retiled region via streaming UDTF (matches the heavyweight generator behavior).
Signature: rst_retile(tile: Column, tileWidth: Column, tileHeight: Column): Column — Retile to uniform dimensions.
SQL:
Since v0.2.0- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT t.*
FROM rasters,
LATERAL gbx_rst_retile(tile, 256, 256) t;
+------+--------------+----+----------------------+
|cellid|raster |path|... |
+------+--------------+----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+----+----------------------+
(one row per sub-tile; t.* expands the v2-Tile struct fields)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
single_band_tile_df(spark).createOrReplaceTempView("rasters")
# Explode raster into 64x64-pixel tiles (returns array of tiles)
return spark.sql(
"SELECT t.* FROM rasters, " "LATERAL gbx_rst_retile(tile, 64, 64) t"
).take(3)
+------+--------------+-----+----------------------+
|cellid|raster |path |... |
+------+--------------+-----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+-----+----------------------+
(one row per sub-tile; t.* expands the v2-Tile struct fields)
if rx is None:
raise ImportError("rasterx not installed")
# The `rasters` view is created in Setup; retile it into 64x64 sub-tiles via SQL LATERAL.
return spark.sql(
"SELECT t.* FROM rasters, " "LATERAL gbx_rst_retile(tile, 64, 64) t"
).take(3)
+------+--------------+-----+----------------------+
|cellid|raster |path |... |
+------+--------------+-----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+-----+----------------------+
(one row per sub-tile; t.* expands the v2-Tile struct fields)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_retile(col("tile"), lit(64), lit(64)).alias("tiles"))
result.show(truncate = false)
+-----+
|tiles|
+-----+
|[{0, |
+-----+
(array of tile structs: [{0, raster, path, metadata}, ...])
rst_separatebands
LightweightHeavyweight Streaming UDTFPowered by rasterio. Streams one band-tile row per band via streaming UDTF — O(1) worker memory regardless of band count (matches the heavyweight generator behavior).
Signature: rst_separatebands(tile: Column): Column — Split multi-band into array of bands.
SQL:
Since v0.2.0- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT t.*
FROM multiband_rasters,
LATERAL gbx_rst_separatebands(tile) t;
+----+-----------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+
|path|red_band |green_band |blue_band |
+----+-----------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+----+-----------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
multiband_tile_df(spark).createOrReplaceTempView("multiband_rasters")
# Explode multiband raster into individual band tiles
return spark.sql(
"SELECT t.* FROM multiband_rasters, " "LATERAL gbx_rst_separatebands(tile) t"
).collect()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(one row per band: 3 rows for a 3-band raster)
if rx is None:
raise ImportError("rasterx not installed")
# The `multiband_rasters` view is created in Setup; separate its bands via SQL LATERAL.
return spark.sql(
"SELECT t.* FROM multiband_rasters, " "LATERAL gbx_rst_separatebands(tile) t"
).collect()
+---+
|z |
+---+
(one row per band: each is a v2-Tile struct)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_separatebands(col("tile")).alias("bands"))
result.show(truncate = false)
+-----+
|bands|
+-----+
|[{0, |
+-----+
(array of tile structs: one per band from the multiband raster)
rst_tooverlappingtiles
LightweightHeavyweight Streaming UDTFPowered by rasterio. Streams one tile row per overlapping region via streaming UDTF (matches the heavyweight generator behavior).
Signature: rst_tooverlappingtiles(tile: Column, tileWidth: Column, tileHeight: Column, overlap: Column): Column — Create overlapping tiles.
SQL:
Since v0.2.0- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT t.*
FROM rasters,
LATERAL gbx_rst_tooverlappingtiles(tile, 256, 256, 10) t;
+------+--------------+----+----------------------+
|cellid|raster |path|... |
+------+--------------+----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+----+----------------------+
(one row per overlapping tile; t.* expands the v2-Tile struct fields)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
single_band_tile_df(spark).createOrReplaceTempView("rasters")
# 64x64 tiles with 8% overlap
return spark.sql(
"SELECT t.* FROM rasters, "
"LATERAL gbx_rst_tooverlappingtiles(tile, 64, 64, 8) t"
).take(3)
+------+--------------+-----+----------------------+
|cellid|raster |path |... |
+------+--------------+-----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+-----+----------------------+
(one row per overlapping tile; t.* expands the v2-Tile struct fields)
if rx is None:
raise ImportError("rasterx not installed")
# The `rasters` view is created in Setup; 64x64 tiles with 8% overlap via SQL LATERAL.
return spark.sql(
"SELECT t.* FROM rasters, "
"LATERAL gbx_rst_tooverlappingtiles(tile, 64, 64, 8) t"
).take(3)
+------+--------------+-----+----------------------+
|cellid|raster |path |... |
+------+--------------+-----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+-----+----------------------+
(one row per overlapping tile; t.* expands the v2-Tile struct fields)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_tooverlappingtiles(col("tile"), lit(64), lit(64), lit(8)).alias("tiles"))
result.show(truncate = false)
+-----+
|tiles|
+-----+
|[{0, |
+-----+
(array of overlapping tile structs with 8% overlap)
Grid Functions (H3)
See Grid raster tessellation for the coverage / assignment / fill model shared by all grids.
Aggregate raster values to H3 grid cells, and utility functions for H3-based canvas setup (9 total).
gbx_h3_cell_bbox
LightweightHeavyweightPowered by h3. Returns a STRUCT<xmin DOUBLE, ymin DOUBLE, xmax DOUBLE, ymax DOUBLE> bounding box for the given H3 cell in the requested srid. In 'centroids' mode the box tightly wraps the centroid point; in 'spatial_envelope' mode it wraps the full hexagon outline. When kring_pad > 0 the k-ring of that radius is computed first and the bounding box covers all cells in the ring. The lightweight SQL function requires all four arguments explicitly; the Python API (rx.h3_cell_bbox(cellid, srid, mode, kring_pad)) honors the same defaults as the Scala implementation.
Scalar function — returns the bounding box STRUCT<xmin, ymin, xmax, ymax> for one H3 cell in the given CRS. Use this to drive the xmin/ymin/xmax/ymax and grid-size parameters of rst_h3_rasterize_agg when you need a consistent per-cell canvas, or to clip and inspect cell extents in downstream queries.
Signature: h3_cell_bbox(cellid: Column, srid: Column, mode: Column, kring_pad: Column): Column
Parameters:
cellid— H3 cell ID (BIGINT or STRING)srid— EPSG code;4326for WGS 84 lon/latmode—'centroids'(centroid point envelope) or'spatial_envelope'(hexagon boundary envelope)kring_pad— expand by this many k-rings before computing the bounding box;0= no expansion
Returns: STRUCT<xmin DOUBLE, ymin DOUBLE, xmax DOUBLE, ymax DOUBLE>
SQL:
Since v0.4.0- SQL
- Python (light)
- Python (heavy)
- Scala
-- Bounding box (STRUCT<xmin, ymin, xmax, ymax>) for each H3 cell in EPSG:4326.
-- Uses 'centroids' mode with no k-ring padding (kring_pad=0).
SELECT
cellid,
gbx_h3_cell_bbox(cellid, 4326, 'centroids', 0) AS bbox
FROM (
VALUES
(617733151020810239),
(617733151085035519),
(617733151021334527)
) AS t(cellid);
+------------------+------------------------------+
|cellid |bbox |
+------------------+------------------------------+
|617733151020810239|{-74.02, 40.70, -74.01, 40.71}|
+------------------+------------------------------+
(STRUCT<xmin, ymin, xmax, ymax> per H3 cell, in EPSG:4326)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.createDataFrame(
[(617733151020810239,), (617733151085035519,), (617733151021334527,)],
["cellid"],
)
return df.select(
"cellid",
rx.gbx_h3_cell_bbox("cellid", f.lit(4326), f.lit("centroids")).alias("bbox"),
).take(3)
+------------------+------------------------------+
|cellid |bbox |
+------------------+------------------------------+
|617733151020810239|{-74.02, 40.70, -74.01, 40.71}|
+------------------+------------------------------+
(STRUCT<xmin, ymin, xmax, ymax> per H3 cell, in EPSG:4326)
if rx is None:
raise ImportError("rasterx not installed")
from pyspark.sql import functions as f
df = spark.createDataFrame(
[(617733151020810239,), (617733151085035519,), (617733151021334527,)],
["cellid"],
)
return df.select(
"cellid",
rx.gbx_h3_cell_bbox("cellid", f.lit(4326), f.lit("centroids"), f.lit(0)).alias(
"bbox"
),
).take(3)
+------------------+------------------------------+
|cellid |bbox |
+------------------+------------------------------+
|617733151020810239|{-74.02, 40.70, -74.01, 40.71}|
+------------------+------------------------------+
(STRUCT<xmin, ymin, xmax, ymax> per H3 cell, in EPSG:4326)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
rx.register(spark)
// H3 res-9 cell ids (as BIGINT); scalar bbox in EPSG:4326, centroids mode.
val schema = StructType(Seq(StructField("cellid", LongType())))
val rows = Seq(617733151020810239L, 617733151085035519L, 617733151021334527L)
val df = spark.createDataFrame(spark.sparkContext.parallelize(rows.map(org.apache.spark.sql.Row(_))), schema)
val result = df.select(col("cellid"), rx.gbx_h3_cell_bbox(col("cellid"), 4326, "centroids", 0).alias("bbox"))
result.show(truncate = false)
+------------------+------------------------------+
|cellid |bbox |
+------------------+------------------------------+
|617733151020810239|{-74.02, 40.70, -74.01, 40.71}|
+------------------+------------------------------+
(STRUCT<xmin, ymin, xmax, ymax> per H3 cell, in EPSG:4326)
rst_h3_gridspec (Python / DataFrame helper)
rst_h3_gridspec is not registered as a SQL function and is not available in the heavyweight tier. It is a pure-Python / PySpark DataFrame helper in the pyrx package.
For the heavyweight tier, compose the equivalent shared canvas using the registered scalar gbx_h3_cell_bbox(cellid, srid, mode, kring_pad) with native Spark min/max aggregates and the same floor/ceil snap arithmetic:
SELECT min(cell_bbox.xmin), min(cell_bbox.ymin),
max(cell_bbox.xmax), max(cell_bbox.ymax)
FROM (SELECT gbx_h3_cell_bbox(cellid, 4326, 'centroids', 1) AS cell_bbox FROM cells)
rst_h3_gridspec computes the shared, snapped canvas (extent + pixel size) that a group of H3 cells should use so that per-threshold rasters align on a common grid and can be stacked via rst_frombands_agg or mosaicked via rst_merge_agg.
Signature:
rx.rst_h3_gridspec(df, cell_col="cellid", *group_cols,
srid=4326, pixel_size=None,
mode="centroids", kring_pad=1)
Typical multi-threshold workflow:
- Call
rx.rst_h3_gridspec(df, cell_col="cellid", srid=4326, mode='centroids', kring_pad=1)once on the distinct cell set. It returns the grouped DataFrame with agridstruct column — one row per group containing the shared canvas. - For each threshold band, run
rst_h3_rasterize_aggwith those fixed bounds — all output tiles share the same origin and pixel grid. - Stack aligned bands with
rst_frombands_agg(ordered byband_index), or mosaic per-cell tiles withrst_merge_agg.
Parameters:
df— input Spark DataFrame containing H3 cell IDscell_col— column name holding H3 cell IDs (integer or string); default"cellid"*group_cols— additional grouping columns (e.g. a transmitter ID, year, month); one grid spec row is produced per groupsrid— EPSG code for the output CRS;4326for WGS 84 (default)pixel_size— ground resolution in CRS units;None= auto-derived from the H3 resolution via an edge-length heuristic (default)mode—'centroids'(default) or'spatial_envelope'kring_pad— k-ring expansion applied per cell before computing its bounding box (default1)
Returns: the grouped DataFrame with a grid column of type:
STRUCT<xmin DOUBLE, ymin DOUBLE, xmax DOUBLE, ymax DOUBLE,
pixel_size DOUBLE, width INT, height INT, srid INT>