Function Reference
RasterX functions run in two execution tiers — lightweight (pure-Python pyrx) and heavyweight (rasterx). As of v0.4.0, every RasterX function is available in both tiers (badged <Tier both/> per function below); per-function notes call out where the lightweight implementation differs. See Choosing an Execution Tier for the comparison and the lightweight install.
Complete reference for all RasterX functions with detailed descriptions, parameters, return values, and examples.
RasterX's lightweight tier is, in large part, distributed rasterio + a best-of-breed raster stack. See Rasterio Distributed for the coverage matrix and honest gaps.
Overview
RasterX is GeoBrix's raster data processing package, providing comprehensive tools for working with raster datasets such as satellite imagery, elevation models, and other gridded spatial data. It is a refactor and improvement of Mosaic raster functions, extended through v0.5.0 with terrain analysis, spectral indices, vector-raster bridging, web-mercator tile output, H3/quadbin/BNG grid aggregations, and — in v0.5.0 — virtual tiles, a COG-preparation lane, and VRT mosaics for memory-safe processing of multi-gigabyte rasters. Since the Databricks product does not (yet) support anything built-in specifically for raster processing, RasterX provides a gap-filling capability for raster operations on the Databricks platform.
Key Features
- GDAL-Powered: Leverages GDAL for robust raster format support
- Distributed Processing: Built on Spark for scalable raster operations
- Multiple Format Support: GeoTIFF, COG, NetCDF, and other GDAL-supported formats
- Metadata Extraction: Comprehensive raster metadata access
- Raster Operations: Clipping, resampling, transformations, map algebra
- Band Operations: Multi-band raster support, single-band extraction
- Terrain Analysis: Slope, aspect, hillshade, TRI, TPI, roughness, color-relief
- Spectral Indices: EVI, SAVI, NDWI, NBR, NDVI, plus a generic dispatcher
- Vector-Raster Bridge: Rasterize geometries, polygonize value regions
- Tile Publishing: Web-mercator XYZ tile generation (PNG / JPEG / WebP)
- Grid Aggregations: H3, CARTO quadbin v0, and BNG cell aggregations
- Virtual Tiles: Bytes-free windowed reads — process multi-gigabyte rasters without materializing them into executor memory
- COG Preparation & VRT Mosaics:
file_gbx+cog_gbxprepare spec-valid Cloud-Optimized GeoTIFFs and portable VRT mosaics for cheap windowed reads
Function Categories
RasterX exposes 129 SQL functions (registered as gbx_rst_*; available in Python and Scala as rst_*), organized into the following categories (see rasterx/functions.scala):

- Accessor Functions: Read raster properties and metadata (bounds, dimensions, CRS, bands, pixel size, georeference, format, type, NoData, subdatasets, summary, etc.)
- Aggregator Functions: Combine or merge rasters in group-by (combineavg_agg, derivedband_agg, merge_agg, h3_rasterize_agg)
- Constructor Functions: Create or load rasters from paths, binary content, or bands
- Generator Functions: Produce multiple tiles or bands (h3_tessellate, maketiles, retile, separatebands, tooverlappingtiles)
- Grid Functions (H3): Aggregate raster values to H3 cells (rastertogrid avg/count/max/min/median/sum/variance/stddev)
- Grid Functions (quadbin): Aggregate raster values to CARTO quadbin v0 cells (rastertogrid avg/count/max/min/median/sum/variance/stddev; tessellate; rasterize_agg)
- Grid Functions (BNG): Aggregate raster values to British National Grid cells (rastertogrid avg/count/max/min/median/sum/variance/stddev; tessellate; rasterize_agg)
- Operations: Transform and analyze rasters (clip, transform, merge, asformat, ndvi, filter, convolve, map algebra, coordinate conversion, isEmpty, tryOpen, initNoData, updateType, combineavg, derivedband)
- Web-Mercator Tile Output: Reproject to EPSG:3857 and emit slippy-map XYZ tiles (to_webmercator, tilexyz, xyzpyramid)
- Vector-raster bridge: Burn polygons into rasters and trace contiguous regions back to polygons (rasterize, polygonize)
- Terrain Analysis: DEM-derived surfaces from
gdal.DEMProcessing(slope, aspect, hillshade, TRI, TPI, roughness, color relief) - Spectral Indices: Multi-band satellite math (EVI, SAVI, NDWI, NBR, plus the generic
rst_indexdispatcher)
Tile payload
Every RasterX (heavyweight) function returns a materialized tile whose raster field is a self-contained, in-memory raster (GTiff by default) — safe to serialize between Spark stages and executors, persist to Delta, hand off to rasterio / gdal, or write back out via the gdal writer. The bytes are never an XML reference to a per-executor /vsimem/ tempfile or to a path that only exists on the producing node.
See Tile structure for the full tile-struct schema (the shared cellid / raster / path / window / … struct), and Virtual Tiles for the bytes-free lightweight-tier variant and how tiles move between materialized and virtual.
Functions that internally build via an intermediate VRT — gbx_rst_merge, gbx_rst_merge_agg, gbx_rst_frombands, gbx_rst_combineavg, gbx_rst_combineavg_agg, gbx_rst_derivedband, gbx_rst_derivedband_agg — materialize the result to GTiff before returning, so downstream stages on different executors see real raster bytes. Inspect a tile's payload format from tile.metadata.driver; for any of the functions above, it will read GTiff (not VRT). See Release Notes for the v0.3.0 correctness fix that introduced this invariant. See Tile structure for the full tile-struct schema.
Virtual-tile force-output params
Tile-returning rst_* functions in the lightweight tier accept three optional keyword arguments
that control whether the produced tile carries raster bytes or a bytes-free virtual reference
(a path + window instead of in-memory pixels):
| Argument | Type | Meaning |
|---|---|---|
virtualize_dir | str (durable path) | Write the produced tile to <dir>/[<prefix>_]<name>.tif and return a virtual tile (path + window, no bytes). The directory must be a durable, executor-readable location such as a Unity Catalog Volume path. Cannot be set together with materialize=True. |
virtualize_prefix | str | Optional filename prefix added before the provenance-based filename — use this to deconflict when two different function outputs share the same virtualize_dir. |
materialize | bool | True — ensure the produced tile carries bytes. False — no-op. Default (unset) — auto: reference/passthrough ops (header reads, rst_clip, rst_setsrid, identity rst_transform) belong to the no-new-pixels class and incur no pixel computation; pixel-producing ops (slope, focal, mapalgebra, reproject, merge, combineavg, frombands, …) materialize. |
Default (auto) behavior: reference/passthrough operations (rst_clip, rst_setsrid, identity
rst_transform) produce no new pixels — they describe a region of or annotation on existing backing
data rather than computing fresh values, so they are the low-cost class. Pixel-producing operations
(slope, focal, mapalgebra, reproject, merge, combineavg, frombands, …) materialize and return bytes;
for these, virtualize_dir is the only way to get a virtual tile back — it writes the computed
result to a durable path and returns a light virtual row.
The lightweight tier is for light (virtual) raster tiles; the heavyweight tier is for heavy
(binary) raster tiles. The heavyweight (rasterx) tier does not accept virtualize_dir,
virtualize_prefix, or materialize, and accepts only materialized tiles (raster bytes
present) as input — both v1 and v2 materialized tiles are accepted; the result is always the v2
tile struct. Passing a virtual tile to a heavyweight function raises a clear error directing
you to materialize it first. For most raster work the lightweight tier covers the full function
set and is the recommended default. If you specifically need a heavyweight function, materialize
the tile first (materialize=True, or write via a writer, which always materializes) before
passing it. See Virtual tiles and the light→heavy bridge
for details.
Setup
With the geobrix library already installed (Installation), pick your execution tier and run this once. Both tiers alias the module as rx, so every example below is identical regardless of tier — only this setup differs. (See Choosing an Execution Tier for the comparison.)
The examples on this page read from four temp views, each a raster loaded as a tile column. Point the four path placeholders at your own rasters:
| Placeholder | Sample file | Reader (light / heavy) — Temp view | Backs |
|---|---|---|---|
GTIFF_SAMPLE_DIR | single-band GeoTIFF | gtiff_gbx / gtiff_gdal — rasters | Default — most accessor, tile-ops, transform, and generator examples |
GTIFF_MULTI_DIR | multi-band GeoTIFF (red/NIR/green) | gtiff_gbx / gtiff_gdal — multiband_rasters | band-math and spectral-index examples, rst_numbands, rst_bandmetadata |
DTM_DIR | digital elevation model | gtiff_gbx / gtiff_gdal — dem_rasters | terrain examples (rst_slope, rst_aspect, …) |
NETCDF_DIR | NetCDF with subdatasets | netcdf_gbx / netcdf_gdal — netcdf_rasters | rst_subdatasets, rst_getsubdataset |
Each format has a named reader in both tiers (light *_gbx / heavy *_gdal); the setup for each tier below uses its own. The generic readers (raster_gbx light, gdal heavy) also read any raster their tier's engine supports.
The four *_DIR placeholders in the code below are wired to small sample rasters committed in the GeoBrix repo under src/test/resources/binary/ (resolved by the _SAMPLE_PATHS[...] / *_path() helpers you'll see in the snippet). To run the examples yourself, stage those sample files — or your own rasters — in a Unity Catalog Volume directory and set each placeholder to its Volume path.
The setup below loads GTIFF_SAMPLE_DIR into rasters and then loads the other three the same way — swap the path and view name (and, for NetCDF, the reader). Every example on the page assumes these four views exist.
- Lightweight (pyrx)
- Heavyweight (rasterx)
from databricks.labs.gbx.pyrx import functions as rx
from databricks.labs.gbx.ds.register import register as register_readers
rx.register(spark)
# Register the lightweight Python DataSource readers (raster_gbx / gtiff_gbx /
# netcdf_gbx / cog_gbx). Pure-Python, no JAR — the light-tier counterpart of the
# heavy gdal / gtiff_gdal / netcdf_gdal readers.
register_readers(spark)
GTIFF_SAMPLE_DIR = _SAMPLE_PATHS["gtiff"]
GTIFF_MULTI_DIR = _SAMPLE_PATHS["gtiff_multi"]
DTM_DIR = _SAMPLE_PATHS["dtm"]
NETCDF_DIR = _SAMPLE_PATHS["netcdf"]
def load_tiles(path, reader, view):
df = spark.read.format(reader).load(path)
df.createOrReplaceTempView(view)
return df
# Load the default single-band raster into the `rasters` view:
rasters = load_tiles(GTIFF_SAMPLE_DIR, "gtiff_gbx", "rasters")
# The other three load the same way — GeoTIFFs use "gtiff_gbx", NetCDF uses "netcdf_gbx":
load_tiles(GTIFF_MULTI_DIR, "gtiff_gbx", "multiband_rasters")
load_tiles(DTM_DIR, "gtiff_gbx", "dem_rasters")
load_tiles(NETCDF_DIR, "netcdf_gbx", "netcdf_rasters")
Four temp views created — `rasters` (single-band), `multiband_rasters`,
`dem_rasters`, and `netcdf_rasters` — each a DataFrame with a `tile` column.
Every example on this page reads from one of these views.
from databricks.labs.gbx.rasterx import functions as rx
from ._fixtures import single_band_path, multiband_path, dem_path, netcdf_path
rx.register(spark)
GTIFF_SAMPLE_DIR = str(single_band_path())
GTIFF_MULTI_DIR = str(multiband_path())
DTM_DIR = str(dem_path())
NETCDF_DIR = str(netcdf_path())
def load_tiles(path, reader, view):
df = spark.read.format(reader).load(path)
df.createOrReplaceTempView(view)
return df
# Load the default single-band raster into the `rasters` view:
rasters = load_tiles(GTIFF_SAMPLE_DIR, "gtiff_gdal", "rasters")
# The other three load the same way — GeoTIFFs use "gtiff_gdal", NetCDF uses "netcdf_gdal":
load_tiles(GTIFF_MULTI_DIR, "gtiff_gdal", "multiband_rasters")
load_tiles(DTM_DIR, "gtiff_gdal", "dem_rasters")
load_tiles(NETCDF_DIR, "netcdf_gdal", "netcdf_rasters")
RasterX registered. Four temp views created — `rasters` (single-band),
`multiband_rasters`, `dem_rasters`, and `netcdf_rasters` — each with a `tile`
column. Every example on this page reads from one of these views.
Scala uses this same heavyweight (rasterx) setup: create the four views with spark.read.format("gtiff_gdal").load(...).createOrReplaceTempView("rasters") (and netcdf_gdal for the NetCDF view), then each Scala example reads val df = spark.table("multiband_rasters"). Registration is only needed for the SQL functions — the Scala rx.* Column API works without it.
Prefer pure SQL? Register RasterX (Python: rx.register(spark)), then create the same four views directly so the SQL examples can use FROM rasters, FROM multiband_rasters, and so on:
-- After registering RasterX (Python: rx.register(spark)), create the views:
CREATE OR REPLACE TEMP VIEW rasters AS
SELECT * FROM gdal.`{SAMPLE_RASTER_PATH}`; -- GTIFF_SAMPLE_DIR (single-band)
CREATE OR REPLACE TEMP VIEW multiband_rasters AS
SELECT * FROM gdal.`{MULTIBAND_RASTER_PATH}`; -- GTIFF_MULTI_DIR
CREATE OR REPLACE TEMP VIEW dem_rasters AS
SELECT * FROM gdal.`{DEM_RASTER_PATH}`; -- DTM_DIR
CREATE OR REPLACE TEMP VIEW netcdf_rasters AS
SELECT * FROM netcdf_gdal.`{NETCDF_RASTER_PATH}`; -- NETCDF_DIR
Views `rasters`, `multiband_rasters`, `dem_rasters`, and `netcdf_rasters` created.
Every example on this page reads from one of these views.
Tier availability
As of v0.4.0, all RasterX functions run in both execution tiers — the lightweight pyrx (pure-Python) and heavyweight rasterx tiers share the same rst_* / gbx_rst_* names, and each function below carries a :::note Lightweight tier (pyrx) admonition with its backing library and any behavioral differences. For the heavyweight VRT Python pixel-function configuration (used by gbx_rst_combineavg / gbx_rst_derivedband), see VRT Python pixel functions at the end of this page.
Examples — Conventions
Every function on this page shows one example, expressed identically across four tabs. This section defines the shared setup so each function tab contains only its invocation — nothing more.
Canonical sample files
Each function uses one of four canonical files, chosen by what the function demonstrates:
| File | What it demonstrates | Used by |
|---|---|---|
nyc_sentinel2_red.tif | Single-band GeoTIFF (Sentinel-2 red band, NYC area) | Default — most accessor, tile-ops, transform, and generator functions |
rgb_nir_small.tif | 3-band GeoTIFF with per-band metadata (red, NIR, green — 8×8 px) | Band-math and spectral-index functions (rst_ndvi, rst_evi, …), rst_numbands, rst_bandmetadata |
srtm_n40w073.tif | Digital Elevation Model (SRTM, NYC area) | Terrain functions (rst_slope, rst_aspect, rst_hillshade, …) |
CMIP5 NetCDF (prAdjust_day_…nc) | Multi-variable NetCDF with two subdatasets | rst_subdatasets, rst_getsubdataset only |
The tile-column convention
In every example on this page:
- SQL:
rastersis a temporary view whosetilecolumn holds the canonical sample loaded via the reader.FROM rastersmeans "the sample as tiles." - Python (light and heavy):
dfis a DataFrame with atilecolumn loaded from the canonical file.df.select(...)means "apply this function to the sample tiles." - Scala:
rastersis a DataFrame with the sametilecolumn.rasters.select(...)is identical in intent to the Python form.
Each function's example therefore shows only the invocation — the load is the convention, not the code.
How to read the four tabs
| Tab | Tier | Color |
|---|---|---|
| SQL | Both (default) | — |
| Python (light) | pyrx lightweight tier | — |
| Python (heavy) | rasterx heavyweight tier | Blue badge |
| Scala | rasterx heavyweight tier | Blue badge |
All four tabs show the same operation on the same file. Where a tier's output genuinely differs in form, a short note explains why — for example:
- Geometry-returning functions (
rst_boundingbox,rst_georeference): SQL returns WKT; Python and Scala return a binary column. The note reads...(WKB binary). - Subdataset maps (
rst_subdatasets): SQL returnsmap<string,string>; rendered as{SUBDATASET_1_NAME -> ..., SUBDATASET_1_DESC -> ...}. A note clarifies the key pattern. - Band-metadata maps (
rst_bandmetadata): similarly noted where the rendering differs between tiers.
Scalar values are identical across all tabs — same fixture, same function, same result.
Tile-returning functions (rst_clip, rst_resample, rst_transform, the aggregators, the constructors, …) return a v2 Tile. Their output is shown as a representative struct — {0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}} — rather than a literal byte dump. Tiles loaded via rst_fromcontent / rst_fromfile (as in Setup) are materialized: the raster field holds the encoded bytes and path is null. Virtual tiles — a populated path with lazily-read bytes — arise from the force-output parameters, not from the default load.
SQL naming and arguments: in the SQL tab, RasterX functions carry the gbx_ prefix (e.g. gbx_rst_boundingbox, gbx_rst_width); Python and Scala use the bare rst_* name via rx. In the lightweight tier the registered gbx_rst_* SQL functions require every argument to be passed explicitly — optional defaults are honored only through the Python rx.* API. See Language Bindings for more.
Per-function notes
Functions that use a non-default fixture (multiband GeoTIFF, DEM, or NetCDF) carry a one-line note immediately before their code example flagging the file. Functions that produce a tile rather than consuming one (constructors: rst_fromfile, rst_fromcontent, rst_frombands, …) show a fuller load/build example with a note — the bare-invocation model does not apply when there is no input tile yet.
Accessor Functions
Functions to read raster properties and metadata (29 total).
rst_avg
LightweightHeavyweightPowered by rasterio + NumPy. Per-band mean over valid (non-NoData) pixels.
Signature: rst_avg(tile: Column): Column — Per-band average pixel values.
Returns NULL for a band with zero valid pixels (all NoData) on both tiers.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands: red, NIR, green); the canonical single-band sentinel2 tile is all-NoData for this function.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_avg(tile) AS band_averages FROM multiband_rasters;
+-----------------------------+
|band_averages |
+-----------------------------+
|[83.59375, 153.125, 114.3125]|
+-----------------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_avg("tile").alias("band_averages")).first()
+------------------------------------+
|band_averages |
+------------------------------------+
|[83.59375, 153.125, 114.3125] |
+------------------------------------+
from databricks.labs.gbx.rasterx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_avg("tile").alias("band_averages")).first()
+------------------------------------+
|band_averages |
+------------------------------------+
|[83.59375, 153.125, 114.3125] |
+------------------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Reads the multiband_rasters Setup view (rgb_nir_small.tif, 3 bands);
// the single-band sentinel2 tile is all-NoData for this function.
val df = spark.table("multiband_rasters")
val result = df.select(rx.rst_avg(col("tile")).alias("band_averages"))
result.show()
+------------------------------------+
|band_averages |
+------------------------------------+
|[83.59375, 153.125, 114.3125] |
+------------------------------------+
rst_bandmetadata
LightweightHeavyweightPowered by rasterio.
Signature: rst_bandmetadata(tile: Column, band: Column): Column — Band metadata map.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands with per-band GDAL metadata tags); plain single-band GeoTIFFs without tags return an empty map {}.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_bandmetadata(tile, 1) AS band_meta FROM multiband_rasters;
+----------------------------------------------+
|band_meta |
+----------------------------------------------+
|{name -> red, wavelength_nm -> 665, band_in...|
+----------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
result = df.select(rx.rst_bandmetadata("tile", f.lit(1)).alias("band_meta")).first()
+----------------------------------------------+
|band_meta |
+----------------------------------------------+
|{name -> red, wavelength_nm -> 665, band_in...|
+----------------------------------------------+
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("multiband_rasters")
result = tile_df.select(
rx.rst_bandmetadata("tile", f.lit(1)).alias("band_meta")
).first()
+----------------------------------------------+
|band_meta |
+----------------------------------------------+
|{name -> red, wavelength_nm -> 665, band_in...|
+----------------------------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif) which carries per-band GDAL metadata tags.
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_bandmetadata(col("tile"), lit(1)).alias("band_meta"))
result.show(truncate = false)
+----------------------------------------------+
|band_meta |
+----------------------------------------------+
|{name -> red, wavelength_nm -> 665, band_in...|
+----------------------------------------------+
rst_boundingbox
LightweightHeavyweightPowered by rasterio.
Signature: rst_boundingbox(tile: Column): Column — Bounding box geometry.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT path, gbx_rst_boundingbox(tile) as bbox FROM rasters;
+--------------------+-----------------+
|path |bbox |
+--------------------+-----------------+
|.../nyc_sentinel2...|POLYGON ((-74....|
+--------------------+-----------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_boundingbox("tile").alias("bbox")).first()
+----+
|bbox|
+----+
|[...|
+----+
(WKB binary — bounding POLYGON of the raster extent in EPSG:32618)
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_boundingbox("tile").alias("bbox")).first()
+----+
|bbox|
+----+
|[...|
+----+
(WKB binary — bounding POLYGON of the raster extent in EPSG:32618)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_boundingbox(col("tile")).alias("bbox"))
result.show(truncate = false)
+----+
|bbox|
+----+
|[...|
+----+
(WKB binary — bounding POLYGON of the raster extent in EPSG:32618)
rst_crs
LightweightHeavyweightPowered by rasterio + pyproj.
Signature: rst_crs(tile: Column): Column — the tile's CRS as a string (authority string like EPSG:4326 / ESRI:54008, else WKT); always returns a value, including for non-EPSG rasters where rst_srid is NULL. See Coordinate Reference Systems.
Returns the authority string (AUTHORITY:CODE, e.g. EPSG:4326 or ESRI:54008) when the CRS has one, otherwise its WKT. Unlike rst_srid (which returns the integer EPSG code, or NULL/0 for a CRS with no EPSG code), rst_crs always returns a value — including for non-EPSG rasters defined by an ESRI code, WKT, or PROJ4 string.
rst_sridreturns the integer EPSG code (e.g.4326);NULL(lightweight) or0(heavyweight) when the CRS has no EPSG code. Use it when you need the numeric SRID for the native ST bridge or an EPSG-only workflow.rst_crsreturns the CRS string and never loses a non-EPSG CRS.rst_setcrs/rst_transformcrstake a CRS string. An int-castable string is treated as an EPSG SRID ('4326'behaves like SRID4326); otherwise the string is parsed as an authority code (EPSG:/ESRI:), WKT, or PROJ4. This is how ESRI codes, WKT, and PROJ4 definitions survive a round trip.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_crs(tile) AS crs FROM rasters;
+----------+
|crs |
+----------+
|EPSG:32618|
+----------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_crs("tile").alias("crs")).first()
+----------+
|crs |
+----------+
|EPSG:32618|
+----------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_crs("tile").alias("crs")).first()
+----------+
|crs |
+----------+
|EPSG:32618|
+----------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_crs(col("tile")).alias("crs"))
result.show(truncate = false)
+----------+
|crs |
+----------+
|EPSG:32618|
+----------+
rst_format
LightweightHeavyweightPowered by rasterio.
Signature: rst_format(tile: Column): Column — GDAL format name.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_format(tile) AS format FROM rasters;
+------+
|format|
+------+
|GTiff |
+------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_format("tile").alias("format")).first()
+------+
|format|
+------+
|GTiff |
+------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_format("tile").alias("format")).first()
+------+
|format|
+------+
|GTiff |
+------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_format(col("tile")).alias("format"))
result.show()
+------+
|format|
+------+
|GTiff |
+------+
rst_georeference
LightweightHeavyweightPowered by rasterio.
Signature: rst_georeference(tile: Column): Column — Georeference parameters as a map.
The result is a MapType with the following keys, corresponding to GDAL's 6-element geotransform:
| Key | Geotransform index | Meaning |
|---|---|---|
upperLeftX | GT(0) | X of the upper-left corner of the upper-left pixel |
upperLeftY | GT(3) | Y of the upper-left corner of the upper-left pixel |
scaleX | GT(1) | Pixel width (west–east resolution) |
scaleY | GT(5) | Pixel height (north–south resolution; often negative for north-up) |
skewX | GT(2) | Row rotation (typically 0) |
skewY | GT(4) | Column rotation (typically 0) |
See the GDAL geotransform tutorial and raster data model for details.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_georeference(tile) AS georeference FROM rasters;
+-------------------------------------------------------------+
|georeference |
+-------------------------------------------------------------+
|{scaleX -> 10.0, scaleY -> -10.0, upperLeftX -> 2121950.0,...|
+-------------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_georeference("tile").alias("georeference")).first()
+--------------------------------------------------------------+
|georeference |
+--------------------------------------------------------------+
|{scaleX -> 10.0, scaleY -> -10.0, upperLeftX -> 2121950.0,... |
+--------------------------------------------------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_georeference("tile").alias("georeference")).first()
+--------------------------------------------------------------+
|georeference |
+--------------------------------------------------------------+
|{scaleX -> 10.0, scaleY -> -10.0, upperLeftX -> 2121950.0,... |
+--------------------------------------------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_georeference(col("tile")).alias("georeference"))
result.show(truncate = false)
+--------------------------------------------------------------+
|georeference |
+--------------------------------------------------------------+
|{scaleX -> 10.0, scaleY -> -10.0, upperLeftX -> 2121950.0,... |
+--------------------------------------------------------------+
rst_getnodata
LightweightHeavyweightPowered by rasterio. Returns the dataset NoData value repeated once per band.
Signature: rst_getnodata(tile: Column): Column — NoData values per band.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_getnodata(tile) AS nodata FROM rasters;
+------+
|nodata|
+------+
|[0.0] |
+------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_getnodata("tile").alias("nodata")).first()
+--------+
|nodata |
+--------+
|[0.0] |
+--------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_getnodata("tile").alias("nodata")).first()
+--------+
|nodata |
+--------+
|[0.0] |
+--------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_getnodata(col("tile")).alias("nodata"))
result.show()
+--------+
|nodata |
+--------+
|[0.0] |
+--------+
rst_getsubdataset
LightweightHeavyweightPowered by rasterio. Subdataset availability depends on rasterio's bundled GDAL driver set.
Signature: rst_getsubdataset(tile: Column, subsetName: Column): Column — Extract subdataset.
Examples use the CMIP5 NetCDF fixture (prAdjust_day_HadGEM2-CC_*.nc) which has two subdatasets: time_bnds and prAdjust. Subdatasets require a multi-layer format such as NetCDF; plain GeoTIFFs return no subdatasets.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- netcdf_rasters view is from the CMIP5 NetCDF fixture (has time_bnds and prAdjust)
SELECT gbx_rst_getsubdataset(tile, 'prAdjust') AS subdataset FROM netcdf_rasters;
+-----------------------------------------------------------+
|subdataset |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(the extracted prAdjust subdataset as a tile — 720x360, 31 bands)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("netcdf_rasters")
result = df.select(
rx.rst_getsubdataset("tile", f.lit("prAdjust")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(light tier returns a materialized v2 Tile; extracted prAdjust subdataset — 720 pixels wide, 31 bands, 360 rows)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
nc_df = spark.table("netcdf_rasters")
result = nc_df.select(
rx.rst_getsubdataset("tile", f.lit("prAdjust")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(extracted prAdjust subdataset — 720 pixels wide, 31 bands, 360 rows)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses committed CMIP5 NetCDF fixture (has time_bnds and prAdjust subdatasets).
// Subdatasets require a multi-layer format such as NetCDF.
// rst_width wraps the result to return a real scalar proving extraction.
val rasters = spark.table("netcdf_rasters")
val result = rasters.select(
rx.rst_width(rx.rst_getsubdataset(col("tile"), lit("prAdjust"))).alias("width")
)
result.show()
+-----+
|width|
+-----+
| 720|
+-----+
(width of the extracted prAdjust subdataset — 720 pixels, 31 bands, 360 rows)
rst_height
LightweightHeavyweightPowered by rasterio.
Signature: rst_height(tile: Column): Column — Height in pixels.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_height(tile) AS height FROM rasters;
+------+
|height|
+------+
|161 |
+------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_height("tile").alias("height")).first()
+------+
|height|
+------+
|161 |
+------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_height("tile").alias("height")).first()
+------+
|height|
+------+
|161 |
+------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_height(col("tile")).alias("height"))
result.show()
+------+
|height|
+------+
|161 |
+------+
rst_max
LightweightHeavyweightPowered by rasterio + NumPy. Per-band maximum over valid (non-NoData) pixels.
Signature: rst_max(tile: Column): Column — Maximum pixel values per band.
Returns NULL for a band with zero valid pixels (all NoData) on both tiers.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands); the canonical single-band sentinel2 tile is all-NoData for this function.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_max(tile) AS band_max FROM multiband_rasters;
+---------------------+
|band_max |
+---------------------+
|[119.0, 197.0, 148.0]|
+---------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_max("tile").alias("band_max")).first()
+---------------------+
|band_max |
+---------------------+
|[119.0, 197.0, 148.0]|
+---------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_max("tile").alias("band_max")).first()
+---------------------+
|band_max |
+---------------------+
|[119.0, 197.0, 148.0]|
+---------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands); single-band sentinel2 is all-NoData.
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_max(col("tile")).alias("band_max"))
result.show()
+---------------------+
|band_max |
+---------------------+
|[119.0, 197.0, 148.0]|
+---------------------+
rst_median
LightweightHeavyweightPowered by rasterio + NumPy. Per-band median over valid (non-NoData) pixels.
Signature: rst_median(tile: Column): Column — Median pixel values per band.
Returns NULL for a band with zero valid pixels (all NoData) on both tiers.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands); the canonical single-band sentinel2 tile is all-NoData for this function.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_median(tile) AS band_median FROM multiband_rasters;
+--------------------+
|band_median |
+--------------------+
|[85.0, 157.5, 111.5]|
+--------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_median("tile").alias("band_median")).first()
+---------------------+
|band_median |
+---------------------+
|[85.0, 157.5, 111.5] |
+---------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_median("tile").alias("band_median")).first()
+---------------------+
|band_median |
+---------------------+
|[85.0, 157.5, 111.5] |
+---------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands); single-band sentinel2 is all-NoData.
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_median(col("tile")).alias("band_median"))
result.show()
+---------------------+
|band_median |
+---------------------+
|[85.0, 157.5, 111.5] |
+---------------------+
rst_memsize
LightweightHeavyweightPowered by rasterio. Returns the serialized raster size in bytes.
Signature: rst_memsize(tile: Column): Column — In-memory size in bytes.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_memsize(tile) AS memsize FROM rasters;
+-------+
|memsize|
+-------+
|71749 |
+-------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_memsize("tile").alias("memsize")).first()
+-------+
|memsize|
+-------+
|71749 |
+-------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_memsize("tile").alias("memsize")).first()
+-------+
|memsize|
+-------+
|71749 |
+-------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_memsize(col("tile")).alias("memsize"))
result.show()
+-------+
|memsize|
+-------+
|71749 |
+-------+
rst_metadata
LightweightHeavyweightPowered by rasterio.
Signature: rst_metadata(tile: Column): Column — Metadata map.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_metadata(tile) as metadata FROM rasters;
+--------------------------------------------------+
|metadata |
+--------------------------------------------------+
|{driver -> GTiff, crs -> EPSG:32618, count -> 1,..|
+--------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_metadata("tile").alias("metadata")).first()
+--------------------------------------------------+
|metadata |
+--------------------------------------------------+
|{driver -> GTiff, crs -> EPSG:32618, count -> 1,..|
+--------------------------------------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_metadata("tile").alias("metadata")).first()
+--------------------------------------------------+
|metadata |
+--------------------------------------------------+
|{driver -> GTiff, crs -> EPSG:32618, count -> 1,..|
+--------------------------------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_metadata(col("tile")).alias("metadata"))
result.show(truncate = false)
+--------------------------------------------------+
|metadata |
+--------------------------------------------------+
|{driver -> GTiff, crs -> EPSG:32618, count -> 1,..|
+--------------------------------------------------+
rst_min
LightweightHeavyweightPowered by rasterio + NumPy. Per-band minimum over valid (non-NoData) pixels.
Signature: rst_min(tile: Column): Column — Minimum pixel values per band.
Returns NULL for a band with zero valid pixels (all NoData) on both tiers.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands); the canonical single-band sentinel2 tile is all-NoData for this function.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_min(tile) AS band_min FROM multiband_rasters;
+-------------------+
|band_min |
+-------------------+
|[50.0, 102.0, 82.0]|
+-------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_min("tile").alias("band_min")).first()
+-------------------+
|band_min |
+-------------------+
|[50.0, 102.0, 82.0]|
+-------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_min("tile").alias("band_min")).first()
+-------------------+
|band_min |
+-------------------+
|[50.0, 102.0, 82.0]|
+-------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands); single-band sentinel2 is all-NoData.
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_min(col("tile")).alias("band_min"))
result.show()
+-------------------+
|band_min |
+-------------------+
|[50.0, 102.0, 82.0]|
+-------------------+
rst_numbands
LightweightHeavyweightPowered by rasterio.
Signature: rst_numbands(tile: Column): Column — Number of bands.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands: red, NIR, green).
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_numbands(tile) AS num_bands FROM multiband_rasters;
+---------+
|num_bands|
+---------+
|3 |
+---------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_numbands("tile").alias("num_bands")).first()
+---------+
|num_bands|
+---------+
|3 |
+---------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_numbands("tile").alias("num_bands")).first()
+---------+
|num_bands|
+---------+
|3 |
+---------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands: red, NIR, green).
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_numbands(col("tile")).alias("num_bands"))
result.show()
+---------+
|num_bands|
+---------+
|3 |
+---------+
rst_pixelcount
LightweightHeavyweightPowered by rasterio + NumPy. Per-band count of valid (non-NoData) pixels.
Signature: rst_pixelcount(tile: Column): Column — Total pixel count.
Examples use the multiband fixture (rgb_nir_small.tif, 8×8, 3 bands, no NoData set), yielding 64 valid pixels per band; the canonical single-band sentinel2 tile has NoData=0 with all pixels equal zero, returning [0].
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_pixelcount(tile) AS pixel_count FROM multiband_rasters;
+------------+
|pixel_count |
+------------+
|[64, 64, 64]|
+------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_pixelcount("tile").alias("pixel_count")).first()
+------------+
|pixel_count |
+------------+
|[64, 64, 64]|
+------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_pixelcount("tile").alias("pixel_count")).first()
+------------+
|pixel_count |
+------------+
|[64, 64, 64]|
+------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 8x8, no NoData); single-band sentinel2 returns [0].
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_pixelcount(col("tile")).alias("pixel_count"))
result.show()
+------------+
|pixel_count |
+------------+
|[64, 64, 64]|
+------------+
rst_pixelheight
LightweightHeavyweightPowered by rasterio.
Signature: rst_pixelheight(tile: Column): Column — Pixel height in ground units.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_pixelheight(tile) AS pixel_height FROM rasters;
+------------+
|pixel_height|
+------------+
|10.0 |
+------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_pixelheight("tile").alias("pixel_height")).first()
+------------+
|pixel_height|
+------------+
|10.0 |
+------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_pixelheight("tile").alias("pixel_height")).first()
+------------+
|pixel_height|
+------------+
|10.0 |
+------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_pixelheight(col("tile")).alias("pixel_height"))
result.show()
+------------+
|pixel_height|
+------------+
|10.0 |
+------------+
rst_pixelwidth
LightweightHeavyweightPowered by rasterio.
Signature: rst_pixelwidth(tile: Column): Column — Pixel width in ground units.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_pixelwidth(tile) AS pixel_width FROM rasters;
+-----------+
|pixel_width|
+-----------+
|10.0 |
+-----------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_pixelwidth("tile").alias("pixel_width")).first()
+-----------+
|pixel_width|
+-----------+
|10.0 |
+-----------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_pixelwidth("tile").alias("pixel_width")).first()
+-----------+
|pixel_width|
+-----------+
|10.0 |
+-----------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_pixelwidth(col("tile")).alias("pixel_width"))
result.show()
+-----------+
|pixel_width|
+-----------+
|10.0 |
+-----------+
rst_rotation
LightweightHeavyweightPowered by rasterio.
Signature: rst_rotation(tile: Column): Column — Rotation in radians.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_rotation(tile) AS rotation FROM rasters;
+--------+
|rotation|
+--------+
|0.0 |
+--------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_rotation("tile").alias("rotation")).first()
+--------+
|rotation|
+--------+
|0.0 |
+--------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_rotation("tile").alias("rotation")).first()
+--------+
|rotation|
+--------+
|0.0 |
+--------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_rotation(col("tile")).alias("rotation"))
result.show()
+--------+
|rotation|
+--------+
|0.0 |
+--------+
rst_scalex / rst_scaley
LightweightHeavyweightPowered by rasterio.
Signature: rst_scalex(tile: Column): Column, rst_scaley(tile: Column): Column — Scale (pixel size) in X/Y.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_scalex(tile) AS scale_x FROM rasters;
+-------+
|scale_x|
+-------+
|10.0 |
+-------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_scalex("tile").alias("scale_x")).first()
+-------+
|scale_x|
+-------+
|10.0 |
+-------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_scalex("tile").alias("scale_x")).first()
+-------+
|scale_x|
+-------+
|10.0 |
+-------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_scalex(col("tile")).alias("scale_x"))
result.show()
+-------+
|scale_x|
+-------+
|10.0 |
+-------+
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_scaley(tile) AS scale_y FROM rasters;
+-------+
|scale_y|
+-------+
|-10.0 |
+-------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_scaley("tile").alias("scale_y")).first()
+-------+
|scale_y|
+-------+
|-10.0 |
+-------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_scaley("tile").alias("scale_y")).first()
+-------+
|scale_y|
+-------+
|-10.0 |
+-------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_scaley(col("tile")).alias("scale_y"))
result.show()
+-------+
|scale_y|
+-------+
|-10.0 |
+-------+
rst_skewx / rst_skewy
LightweightHeavyweightPowered by rasterio.
Signature: rst_skewx(tile: Column): Column, rst_skewy(tile: Column): Column — Skew in X/Y.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_skewx(tile) AS skew_x FROM rasters;
+------+
|skew_x|
+------+
|0.0 |
+------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_skewx("tile").alias("skew_x")).first()
+------+
|skew_x|
+------+
|0.0 |
+------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_skewx("tile").alias("skew_x")).first()
+------+
|skew_x|
+------+
|0.0 |
+------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_skewx(col("tile")).alias("skew_x"))
result.show()
+------+
|skew_x|
+------+
|0.0 |
+------+
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_skewy(tile) AS skew_y FROM rasters;
+------+
|skew_y|
+------+
|0.0 |
+------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_skewy("tile").alias("skew_y")).first()
+------+
|skew_y|
+------+
|0.0 |
+------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_skewy("tile").alias("skew_y")).first()
+------+
|skew_y|
+------+
|0.0 |
+------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_skewy(col("tile")).alias("skew_y"))
result.show()
+------+
|skew_y|
+------+
|0.0 |
+------+
rst_srid
LightweightHeavyweightPowered by rasterio.
Signature: rst_srid(tile: Column): Column — the stored spatial reference ID integer (an EPSG or ESRI code), or NULL when the tile has none. Returns the code as stored; it is classified only when applied. See Coordinate Reference Systems for the SRID-vs-CRS-string model and the epsg→esri resolution rule.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_srid(tile) AS srid FROM rasters;
+-----+
|srid |
+-----+
|32618|
+-----+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_srid("tile").alias("srid")).first()
+-----+
|srid |
+-----+
|32618|
+-----+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_srid("tile").alias("srid")).first()
+-----+
|srid |
+-----+
|32618|
+-----+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_srid(col("tile")).alias("srid"))
result.show()
+-----+
|srid |
+-----+
|32618|
+-----+
rst_subdatasets
LightweightHeavyweightPowered by rasterio. Empty for single-dataset rasters (e.g. a plain GeoTIFF).
Signature: rst_subdatasets(tile: Column): Column — List of subdataset names.
Examples use the CMIP5 NetCDF fixture (prAdjust_day_HadGEM2-CC_*.nc) which has two subdatasets: time_bnds and prAdjust. Subdatasets require a multi-layer format; plain GeoTIFFs return an empty map.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- netcdf_rasters view is from the CMIP5 NetCDF fixture (has time_bnds and prAdjust)
SELECT gbx_rst_subdatasets(tile) AS subdatasets FROM netcdf_rasters;
+------------------------------------------------------+
|subdatasets |
+------------------------------------------------------+
|{SUBDATASET_1_NAME -> ..., SUBDATASET_1_DESC -> [31...|
+------------------------------------------------------+
(map with SUBDATASET_1_NAME/DESC for time_bnds and SUBDATASET_2_NAME/DESC for prAdjust)
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("netcdf_rasters")
result = df.select(rx.rst_subdatasets("tile").alias("subdatasets")).first()
+------------------------------------------------------+
|subdatasets |
+------------------------------------------------------+
|{SUBDATASET_1_NAME -> ..., SUBDATASET_1_DESC -> [31...|
+------------------------------------------------------+
(map with SUBDATASET_1_NAME/DESC for time_bnds and SUBDATASET_2_NAME/DESC for prAdjust)
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("netcdf_rasters")
result = tile_df.select(rx.rst_subdatasets("tile").alias("subdatasets")).first()
+------------------------------------------------------+
|subdatasets |
+------------------------------------------------------+
|{SUBDATASET_1_NAME -> ..., SUBDATASET_1_DESC -> [31...|
+------------------------------------------------------+
(map with SUBDATASET_1_NAME/DESC for time_bnds and SUBDATASET_2_NAME/DESC for prAdjust)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses committed CMIP5 NetCDF fixture (has time_bnds and prAdjust subdatasets).
val rasters = spark.table("netcdf_rasters")
val result = rasters.select(rx.rst_subdatasets(col("tile")).alias("subdatasets"))
result.show(truncate = false)
+------------------------------------------------------+
|subdatasets |
+------------------------------------------------------+
|{SUBDATASET_1_NAME -> ..., SUBDATASET_1_DESC -> [31...|
+------------------------------------------------------+
(map with SUBDATASET_1_NAME/DESC for time_bnds and SUBDATASET_2_NAME/DESC for prAdjust)
rst_summary
LightweightHeavyweightPowered by rasterio + NumPy. Returns a JSON summary (driver, size, CRS, geotransform, per-band statistics); the lightweight JSON shape differs from the heavyweight gdalinfo -json output.
Signature: rst_summary(tile: Column): Column — Statistical summary of values.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands) which has real pixel data.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_summary(tile) AS summary FROM multiband_rasters;
+------------------------------------------------------------+
|summary |
+------------------------------------------------------------+
|{"driverShortName": "GTiff", "size": [8, 8], "coordinateS...|
+------------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_summary("tile").alias("summary")).first()
+------------------------------------------------------------+
|summary |
+------------------------------------------------------------+
|{"driverShortName": "GTiff", "size": [8, 8], "coordinateS...|
+------------------------------------------------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_summary("tile").alias("summary")).first()
+------------------------------------------------------------+
|summary |
+------------------------------------------------------------+
|{"driverShortName": "GTiff", "size": [8, 8], "coordinateS...|
+------------------------------------------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands) which has real pixel data.
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_summary(col("tile")).alias("summary"))
result.show(truncate = false)
+------------------------------------------------------------+
|summary |
+------------------------------------------------------------+
|{"driverShortName": "GTiff", "size": [8, 8], "coordinateS...|
+------------------------------------------------------------+
rst_type
LightweightHeavyweightPowered by rasterio.
Signature: rst_type(tile: Column): Column — Data type per band.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands, UInt16).
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_type(tile) AS band_types FROM multiband_rasters;
+------------------------+
|band_types |
+------------------------+
|[UInt16, UInt16, UInt16]|
+------------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_type("tile").alias("band_types")).first()
+-----------------------------+
|band_types |
+-----------------------------+
|[UInt16, UInt16, UInt16] |
+-----------------------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_type("tile").alias("band_types")).first()
+-----------------------------+
|band_types |
+-----------------------------+
|[UInt16, UInt16, UInt16] |
+-----------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands, UInt16).
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_type(col("tile")).alias("band_types"))
result.show()
+-----------------------------+
|band_types |
+-----------------------------+
|[UInt16, UInt16, UInt16] |
+-----------------------------+
rst_upperleftx / rst_upperlefty
LightweightHeavyweightPowered by rasterio.
Signature: rst_upperleftx(tile: Column): Column, rst_upperlefty(tile: Column): Column — Upper-left corner coordinates.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_upperleftx(tile) AS upper_left_x FROM rasters;
+------------+
|upper_left_x|
+------------+
|2121950.0 |
+------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_upperleftx("tile").alias("upper_left_x")).first()
+------------+
|upper_left_x|
+------------+
|2121950.0 |
+------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_upperleftx("tile").alias("upper_left_x")).first()
+------------+
|upper_left_x|
+------------+
|2121950.0 |
+------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_upperleftx(col("tile")).alias("upper_left_x"))
result.show()
+------------+
|upper_left_x|
+------------+
|2121950.0 |
+------------+
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_upperlefty(tile) AS upper_left_y FROM rasters;
+------------+
|upper_left_y|
+------------+
|-10790470.0 |
+------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_upperlefty("tile").alias("upper_left_y")).first()
+---------------+
|upper_left_y |
+---------------+
|-10790470.0 |
+---------------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_upperlefty("tile").alias("upper_left_y")).first()
+---------------+
|upper_left_y |
+---------------+
|-10790470.0 |
+---------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_upperlefty(col("tile")).alias("upper_left_y"))
result.show()
+---------------+
|upper_left_y |
+---------------+
|-10790470.0 |
+---------------+
rst_width
LightweightHeavyweightPowered by rasterio.
Signature: rst_width(tile: Column): Column — Width in pixels.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_width(tile) AS width FROM rasters;
+-----+
|width|
+-----+
|236 |
+-----+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_width("tile").alias("width")).first()
+-----+
|width|
+-----+
|236 |
+-----+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_width("tile").alias("width")).first()
+-----+
|width|
+-----+
|236 |
+-----+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_width(col("tile")).alias("width"))
result.show()
+-----+
|width|
+-----+
|236 |
+-----+
Aggregator Functions
Combine or merge rasters in group-by (7 total).
rst_bng_rasterize_agg
LightweightHeavyweight Grouped-agg UDFThe lightweight implementation is backed by pygx._bng cell math and rasterio. BNG cell IDs (STRING) are parsed via pygx._bng to EPSG:27700-native cell geometries and burned into the output band — the output canvas is EPSG:27700 throughout with no warp step (this aggregator takes cellid+value rows, not a raster tile).
Pixels not covered by any geometry in the group are set to -9999.0 (band-registered NoData). Filter or mask downstream via gbx_rst_getnodata or IS NULL on the extracted band value.
Heavyweight gbx_rst_bng_rasterize_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_bng_rasterize_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
cell_id AS cellid,
gbx_rst_fromcontent(gbx_rst_bng_rasterize_agg(cellid, burn_value), 'GTiff') AS tile
FROM bng_cell_values
GROUP BY cell_id
Streaming aggregator that burns BNG cell geometry/value pairs (one row per cell) into a single rasterized tile per group. The input raster is automatically reprojected to EPSG:27700 (British National Grid) before rasterization. BNG cell IDs are STRING. The inverse of rst_bng_rastertogrid*: where those functions reduce raster pixels to per-cell statistics, this one synthesizes a raster from per-cell values.
out_crs parameter (no-op for BNG)The lightweight Python binding accepts an optional trailing out_crs (string CRS) argument that the heavyweight/Scala tier does not — but for BNG it is a no-op: the output is always EPSG:27700. Both out_srid and out_crs are ignored here (they exist only for signature parity with the H3/quadbin aggregators).
Signature: rst_bng_rasterize_agg(cellid: Column, value: Column, out_srid: Column, pixel_size: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, mode: Column, kring_pad: Column): Column
Parameters: cellid — BNG cell ID (STRING); value — numeric burn value; out_srid — EPSG code for the output CRS (typically 27700); pixel_size — output raster cell size in metres (usually 1.0 for 1km BNG cells); xmin/ymin/xmax/ymax — output extent in EPSG:27700 metres; width/height — output raster dimensions in pixels; mode — aggregation mode for overlapping values (typically "last"); kring_pad — cell neighbourhood expansion (typically 0)
Multi-row fixture: 3 BNG 1km STRING cell rows near central London (EPSG:27700) with burn values 1.0/2.0/3.0. All tabs use the same grouped-agg rasterize invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Rasterize BNG cells into one raster tile per region. cellid is a STRING
-- (e.g. 'TQ38SW'); the srid argument is a no-op (BNG always forces EPSG:27700,
-- pass 27700 for clarity). The extent auto-derives from the cell set (null
-- canvas args).
SELECT region_id,
gbx_rst_bng_rasterize_agg(
cellid, burn_value,
27700, cast(null as double),
cast(null as double), cast(null as double),
cast(null as double), cast(null as double),
cast(null as int), cast(null as int),
'centroids', cast(0 as int)
) AS tile
FROM bng_cell_values
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|tile |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(tile, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|tile |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
from databricks.labs.gbx.gridx.bng import functions as bngx
bngx.register(spark)
spark.sql("""
CREATE OR REPLACE TEMP VIEW _bng_cells AS
SELECT region,
gbx_bng_eastnorthasbng(e, n, 3) AS cellid,
val AS value
FROM (VALUES
('R1', cast(530000.0 as double), cast(180000.0 as double), cast(1.0 as double)),
('R1', cast(531000.0 as double), cast(181000.0 as double), cast(2.0 as double)),
('R1', cast(529000.0 as double), cast(179000.0 as double), cast(3.0 as double))
) AS t(region, e, n, val)
""")
df = spark.table("_bng_cells")
result = (
df.groupBy("region")
.agg(rx.rst_bng_rasterize_agg("cellid", "value").alias("tile"))
.first()
)
spark.catalog.dropTempView("_bng_cells")
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.gridx.bng import functions as bngx
bngx.register(spark)
spark.sql("""
CREATE OR REPLACE TEMP VIEW _bng_cells_heavy AS
SELECT region,
gbx_bng_eastnorthasbng(e, n, 3) AS cellid,
val AS value
FROM (VALUES
('R1', cast(530000.0 as double), cast(180000.0 as double), cast(1.0 as double)),
('R1', cast(531000.0 as double), cast(181000.0 as double), cast(2.0 as double)),
('R1', cast(529000.0 as double), cast(179000.0 as double), cast(3.0 as double))
) AS t(region, e, n, val)
""")
df = spark.table("_bng_cells_heavy")
# cellid + value are the only required args; the extent/size/mode/kring
# options default to the same values the lightweight tier uses (BNG forces
# EPSG:27700, so out_srid is a no-op).
result = (
df.groupBy("region")
.agg(rx.rst_bng_rasterize_agg(df["cellid"], df["value"]).alias("tile"))
.first()
)
spark.catalog.dropTempView("_bng_cells_heavy")
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import com.databricks.labs.gbx.gridx.bng.{functions => bng}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
rx.register(spark)
bng.register(spark)
// Multi-row fixture: 3 BNG 1km cells near central London (EPSG:27700).
val schema = StructType(Seq(StructField("region", StringType()), StructField("e", DoubleType()), StructField("n", DoubleType()), StructField("val", DoubleType())))
val rows = Seq(("R1",530000.0,180000.0,1.0),("R1",531000.0,181000.0,2.0),("R1",529000.0,179000.0,3.0))
val raw = spark.createDataFrame(spark.sparkContext.parallelize(rows.map(r => org.apache.spark.sql.Row(r._1,r._2,r._3,r._4))), schema)
raw.createOrReplaceTempView("_bng_src")
val df = spark.sql("SELECT region, gbx_bng_eastnorthasbng(e, n, 3) AS cellid, val AS value FROM _bng_src")
val result = df.groupBy("region").agg(rx.rst_bng_rasterize_agg(col("cellid"), col("value")).alias("tile"))
result.show()
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_combineavg_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio + NumPy. Aggregate — groupBy(...).agg(rx.rst_combineavg_agg("tile")) returns the NoData-aware per-pixel mean as one tile per group; input tiles must share the same grid (shape/extent/CRS).
Heavyweight gbx_rst_combineavg_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_combineavg_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(gbx_rst_combineavg_agg(tile), 'GTiff') AS tile
FROM tiles
GROUP BY group_key
Signature: rst_combineavg_agg(tile: Column): Column — Average tiles per group.
Multi-tile fixture: 3 per-band rows from rgb_nir_small.tif (same grid). All tabs use the same grouped-agg invocation on multiple tile rows.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Group by region and average
SELECT
region,
gbx_rst_combineavg_agg(tile) as regional_average
FROM rasters
GROUP BY region;
# Heavyweight SQL — one tile struct per group:
+------+-----------------------------------------------------------+
|region|regional_average |
+------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(<agg>, 'GTiff') to rebuild a tile:
+------+----------------+
|region|regional_average|
+------+----------------+
|... |[B@... (BINARY) |
+------+----------------+
from databricks.labs.gbx.pyrx import functions as rx
df = _get_multi_band_tiles_df(spark)
result = (
df.groupBy("region")
.agg(rx.rst_combineavg_agg("tile").alias("avg_tile"))
.first()
)
+------+-----------------------------------------------------------+
|region|avg_tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
df = _get_multi_band_tiles_df_heavy(spark)
result = (
df.groupBy("region")
.agg(rx.rst_combineavg_agg("tile").alias("avg_tile"))
.first()
)
+------+-----------------------------------------------------------+
|region|avg_tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Multi-tile fixture: load multiband tif and split to 3 per-band rows (same grid).
val mb = spark.read.format("gdal").load("src/test/resources/binary/geotiff-small/rgb_nir_small.tif")
val b1 = mb.select(rx.rst_band(col("tile"), lit(1)).alias("tile")).withColumn("region", lit("R1"))
val b2 = mb.select(rx.rst_band(col("tile"), lit(2)).alias("tile")).withColumn("region", lit("R1"))
val b3 = mb.select(rx.rst_band(col("tile"), lit(3)).alias("tile")).withColumn("region", lit("R1"))
val df = b1.union(b2).union(b3)
val result = df.groupBy("region").agg(rx.rst_combineavg_agg(col("tile")).alias("avg"))
result.show()
+------+-----------------------------------------------------------+
|region|avg |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_derivedband_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio with GDAL VRT Python pixel functions. Aggregate — groupBy(...).agg(rx.rst_derivedband_agg("tile", pyfunc, funcName)) stacks the group's tiles as bands and applies your pixel function, returning one tile per group.
Heavyweight gbx_rst_derivedband_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_derivedband_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(gbx_rst_derivedband_agg(tile, 'def f(a,b): return a+b', 'f'), 'GTiff') AS tile
FROM tiles
GROUP BY group_key
Signature: rst_derivedband_agg(tile: Column, pyfunc: String, funcName: String): Column — Apply Python UDF to tiles per group.
Multi-tile fixture: 3 per-band rows from rgb_nir_small.tif. All tabs use the same grouped-agg invocation; pixel function selects band 0 (identity).
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT region, gbx_rst_derivedband_agg(tile, 'def f(a): return a', 'f') as result FROM rasters GROUP BY region;
# Heavyweight SQL — one v2 tile struct per group:
+------+-----------------------------------------------------------+
|region|result |
+------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(result, 'GTiff') to rebuild a tile struct:
+------+---------------+
|region|result |
+------+---------------+
|... |[B@... (BINARY)|
+------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
pyfunc = (
"def fn(in_ar, out_ar, xoff, yoff, xsize, ysize, "
"raster_xsize, raster_ysize, buf_radius, gt, **kwargs):\n"
" out_ar[:] = in_ar[0]\n"
)
df = _get_multi_band_tiles_df(spark)
result = (
df.groupBy("region")
.agg(rx.rst_derivedband_agg("tile", pyfunc, "fn").alias("derived"))
.first()
)
+------+-----------------------------------------------------------+
|region|derived |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
pyfunc = (
"def fn(in_ar, out_ar, xoff, yoff, xsize, ysize, "
"raster_xsize, raster_ysize, buf_radius, gt, **kwargs):\n"
" out_ar[:] = in_ar[0]\n"
)
df = _get_multi_band_tiles_df_heavy(spark)
result = (
df.groupBy("region")
.agg(
rx.rst_derivedband_agg("tile", f.lit(pyfunc), f.lit("fn")).alias("derived")
)
.first()
)
+------+-----------------------------------------------------------+
|region|derived |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Multi-tile fixture: load multiband tif and split to 3 per-band rows.
val mb = spark.read.format("gdal").load("src/test/resources/binary/geotiff-small/rgb_nir_small.tif")
val b1 = mb.select(rx.rst_band(col("tile"), lit(1)).alias("tile")).withColumn("region", lit("R1"))
val b2 = mb.select(rx.rst_band(col("tile"), lit(2)).alias("tile")).withColumn("region", lit("R1"))
val b3 = mb.select(rx.rst_band(col("tile"), lit(3)).alias("tile")).withColumn("region", lit("R1"))
val df = b1.union(b2).union(b3)
val fn = "def fn(in_ar, out_ar, xoff, yoff, xsize, ysize, raster_xsize, raster_ysize, buf_radius, gt, **kwargs):\n out_ar[:] = in_ar[0]\n"
val result = df.groupBy("region").agg(rx.rst_derivedband_agg(col("tile"), fn, "fn").alias("derived"))
result.show()
+------+-----------------------------------------------------------+
|region|derived |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_dtmfromgeoms_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio + SciPy (scipy.spatial.Delaunay). Aggregate — builds one TIN DTM tile per group from the group's Z-valued points via barycentric interpolation over an unconstrained Delaunay triangulation; breaklines, merge_tolerance, and snap_tolerance are accepted but not enforced (the heavyweight tier builds a constrained TIN). The lightweight Python binding also accepts an optional trailing out_crs (string CRS) argument that the SQL/heavyweight tiers do not — the heavyweight builder takes the int out_srid only. out_crs wins over out_srid when both are given. This is a lightweight superset, not a heavyweight regression.
Heavyweight gbx_rst_dtmfromgeoms_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_dtmfromgeoms_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(
gbx_rst_dtmfromgeoms_agg(point, null, 0.0, 0.0, 0,0,10,10, 8,8, 32633),
'GTiff'
) AS tile
FROM observations
GROUP BY group_key
Streaming aggregator that accepts one Z-valued point WKB per row and produces a TIN/Delaunay DTM raster tile per group; breaklines are supplied as a per-group constant array to enforce hard terrain edges.
Signature: rst_dtmfromgeoms_agg(point: Column, breaklines: Column, mergeTolerance: Column, snapTolerance: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, out_srid: Column): Column
Parameters: point — WKB point geometry with Z coordinate (one per row); breaklines — constant WKB array of breakline geometries per group (pass null or empty array if unused); remaining parameters match rst_dtmfromgeoms
Multi-row fixture: 4 Z-valued WKB POINT rows (elevation 100–250 m) over a [0,0,1,1] EPSG:4326 extent. All tabs use the same grouped-agg invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Stream survey points per region into one TIN DTM tile. Breaklines are a
-- per-group constant array; for 10 m cells over a 1000 m extent use 100 px.
SELECT region_id,
gbx_rst_dtmfromgeoms_agg(
point_wkb, breaklines_wkb_array,
0.0, 0.01,
bbox_xmin, bbox_ymin, bbox_xmax, bbox_ymax,
100, 100, 32633
) AS dtm
FROM survey_points
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|dtm |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(dtm, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|dtm |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
BinaryType,
StringType,
StructField,
StructType,
)
def _wkb_point_z(x, y, z):
"""Build a WKB POINT Z (ISO wkbType 1001) as bytes."""
import struct
return struct.pack("<bIddd", 1, 1001, x, y, z)
rows = [
(_wkb_point_z(0.1, 0.1, 100.0), "R1"),
(_wkb_point_z(0.9, 0.1, 200.0), "R1"),
(_wkb_point_z(0.1, 0.9, 150.0), "R1"),
(_wkb_point_z(0.9, 0.9, 250.0), "R1"),
]
schema = StructType(
[
StructField("pt", BinaryType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_dtmfromgeoms_agg(
"pt",
f.lit(None).cast("array<binary>"),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(1.0),
f.lit(1.0),
f.lit(8),
f.lit(8),
f.lit(4326),
).alias("dtm")
)
.first()
)
+------+-----------------------------------------------------------+
|region|dtm |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
BinaryType,
StringType,
StructField,
StructType,
)
def _wkb_point_z(x, y, z):
import struct
return struct.pack("<bIddd", 1, 1001, x, y, z)
rows = [
(_wkb_point_z(0.1, 0.1, 100.0), "R1"),
(_wkb_point_z(0.9, 0.1, 200.0), "R1"),
(_wkb_point_z(0.1, 0.9, 150.0), "R1"),
(_wkb_point_z(0.9, 0.9, 250.0), "R1"),
]
schema = StructType(
[
StructField("pt", BinaryType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_dtmfromgeoms_agg(
"pt",
f.lit(None).cast("array<binary>"),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(1.0),
f.lit(1.0),
f.lit(8),
f.lit(8),
f.lit(4326),
).alias("dtm")
)
.first()
)
+------+-----------------------------------------------------------+
|region|dtm |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import java.nio.{ByteBuffer, ByteOrder}
rx.register(spark)
// Multi-row fixture: 4 WKB POINT Z rows over [0,0,1,1] EPSG:4326.
def mkPtZ(x: Double, y: Double, z: Double): Array[Byte] = {
val buf = ByteBuffer.allocate(29).order(ByteOrder.LITTLE_ENDIAN)
buf.put(1.toByte); buf.putInt(1001); buf.putDouble(x); buf.putDouble(y); buf.putDouble(z); buf.array()
}
val schema = StructType(Seq(StructField("pt", BinaryType()), StructField("region", StringType())))
val rows = Seq((mkPtZ(0.1,0.1,100.0),"R1"),(mkPtZ(0.9,0.1,200.0),"R1"),(mkPtZ(0.1,0.9,150.0),"R1"),(mkPtZ(0.9,0.9,250.0),"R1"))
val df = spark.createDataFrame(spark.sparkContext.parallelize(rows.map { case (p,r) => org.apache.spark.sql.Row(p,r) }), schema)
val result = df.groupBy("region").agg(
rx.rst_dtmfromgeoms_agg(col("pt"), lit(null).cast("array<binary>"), lit(0.0), lit(0.0), lit(0.0), lit(0.0), lit(1.0), lit(1.0), lit(8), lit(8), lit(4326)).alias("dtm")
)
result.show()
+------+-----------------------------------------------------------+
|region|dtm |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_frombands_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio. Aggregate — groupBy(...).agg(rx.rst_frombands_agg("tile", "band_index")) stacks the group's tiles into one multi-band tile ordered by ascending band_index.
Heavyweight gbx_rst_frombands_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_frombands_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(gbx_rst_frombands_agg(tile, band_index), 'GTiff') AS tile
FROM bands
GROUP BY group_key
Streaming aggregator that collects ordered per-band tiles (one row per band) into a single multi-band raster tile per group; use when bands arrive as separate rows rather than a pre-built array.
Signature: rst_frombands_agg(tile: Column, bandIndex: Column): Column
Parameters: tile — Single-band raster tile; bandIndex — 1-based band position within the output raster
Multi-tile fixture: 3 per-band rows from rgb_nir_small.tif with band_index 1/2/3. All tabs stack via the same grouped-agg invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Collect per-band tiles in acquisition order into one multi-band raster per scene.
SELECT scene_id,
gbx_rst_frombands_agg(tile, band_index) AS multi_band
FROM band_tiles
GROUP BY scene_id;
# Heavyweight SQL — one v2 tile struct per group:
+--------+-----------------------------------------------------------+
|scene_id|multi_band |
+--------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+--------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(multi_band, 'GTiff') to rebuild a tile struct:
+--------+---------------+
|scene_id|multi_band |
+--------+---------------+
|... |[B@... (BINARY)|
+--------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
df = _get_multi_band_tiles_df(spark)
result = (
df.groupBy("region")
.agg(rx.rst_frombands_agg("tile", "band_index").alias("stacked"))
.first()
)
+------+-----------------------------------------------------------+
|region|stacked |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
df = _get_multi_band_tiles_df_heavy(spark)
result = (
df.groupBy("region")
.agg(rx.rst_frombands_agg("tile", "band_index").alias("stacked"))
.first()
)
+------+-----------------------------------------------------------+
|region|stacked |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Multi-tile fixture: load multiband tif and split to 3 per-band rows with band_index.
val mb = spark.read.format("gdal").load("src/test/resources/binary/geotiff-small/rgb_nir_small.tif")
val b1 = mb.select(rx.rst_band(col("tile"), lit(1)).alias("tile")).withColumn("band_index", lit(1)).withColumn("region", lit("R1"))
val b2 = mb.select(rx.rst_band(col("tile"), lit(2)).alias("tile")).withColumn("band_index", lit(2)).withColumn("region", lit("R1"))
val b3 = mb.select(rx.rst_band(col("tile"), lit(3)).alias("tile")).withColumn("band_index", lit(3)).withColumn("region", lit("R1"))
val df = b1.union(b2).union(b3)
val result = df.groupBy("region").agg(rx.rst_frombands_agg(col("tile"), col("band_index")).alias("stacked"))
result.show()
+------+-----------------------------------------------------------+
|region|stacked |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_gridfrompoints_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio + SciPy (cKDTree IDW). Aggregate — groupBy(...).agg(rx.rst_gridfrompoints_agg(...)) inverse-distance-interpolates the group's points into one Float64 grid tile (NoData −9999). The lightweight Python binding also accepts an optional trailing out_crs (string CRS) argument that the SQL/heavyweight tiers do not — the heavyweight builder takes the int out_srid only. out_crs wins over out_srid when both are given. This is a lightweight superset, not a heavyweight regression.
Heavyweight gbx_rst_gridfrompoints_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_gridfrompoints_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(
gbx_rst_gridfrompoints_agg(point, value, 0,0,10,10, 8,8, 32633, 2.0, 12),
'GTiff'
) AS tile
FROM observations
GROUP BY group_key
Streaming IDW-interpolation aggregator that accepts one point geometry and one scalar value per row and produces a Float64 GeoTIFF tile per group; use when observations arrive one per row rather than as pre-built arrays.
Signature: rst_gridfrompoints_agg(point: Column, value: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, widthPx: Column, heightPx: Column, out_srid: Column, power: Column, maxPts: Column): Column
Parameters: point — WKB point geometry (one per row); value — scalar observation for the point; xmin/ymin/xmax/ymax — output extent in CRS units (constant per group); widthPx/heightPx — output dimensions in pixels; srid — EPSG code; power — IDW distance-decay exponent (2.0 is standard); maxPts — maximum nearest neighbours considered per output pixel
Multi-row fixture: 4 WKB POINT rows with observations 10–40 over a [0,0,1,1] EPSG:4326 extent. All tabs use the same grouped-agg IDW invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Aggregate per-station observations into one IDW tile per region.
SELECT region_id,
gbx_rst_gridfrompoints_agg(
station_wkb, observation,
bbox_xmin, bbox_ymin, bbox_xmax, bbox_ymax,
256, 256, 32633
) AS idw
FROM observations
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|idw |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(idw, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|idw |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
BinaryType,
DoubleType,
StringType,
StructField,
StructType,
)
def _wkb_point(x, y):
"""Build a WKB POINT from (x, y) as bytes."""
import struct
return struct.pack("<bIdd", 1, 1, x, y)
rows = [
(_wkb_point(0.1, 0.1), 10.0, "R1"),
(_wkb_point(0.9, 0.1), 20.0, "R1"),
(_wkb_point(0.1, 0.9), 30.0, "R1"),
(_wkb_point(0.9, 0.9), 40.0, "R1"),
]
schema = StructType(
[
StructField("pt", BinaryType()),
StructField("val", DoubleType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_gridfrompoints_agg(
"pt",
"val",
f.lit(0.0),
f.lit(0.0),
f.lit(1.0),
f.lit(1.0),
f.lit(8),
f.lit(8),
f.lit(4326),
).alias("idw")
)
.first()
)
+------+-----------------------------------------------------------+
|region|idw |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
BinaryType,
DoubleType,
StringType,
StructField,
StructType,
)
def _wkb_point(x, y):
import struct
return struct.pack("<bIdd", 1, 1, x, y)
rows = [
(_wkb_point(0.1, 0.1), 10.0, "R1"),
(_wkb_point(0.9, 0.1), 20.0, "R1"),
(_wkb_point(0.1, 0.9), 30.0, "R1"),
(_wkb_point(0.9, 0.9), 40.0, "R1"),
]
schema = StructType(
[
StructField("pt", BinaryType()),
StructField("val", DoubleType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_gridfrompoints_agg(
"pt",
"val",
f.lit(0.0),
f.lit(0.0),
f.lit(1.0),
f.lit(1.0),
f.lit(8),
f.lit(8),
f.lit(4326),
).alias("idw")
)
.first()
)
+------+-----------------------------------------------------------+
|region|idw |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import java.nio.{ByteBuffer, ByteOrder}
rx.register(spark)
// Multi-row fixture: 4 WKB POINT rows with scalar observations, [0,0,1,1] EPSG:4326 extent.
def mkPt(x: Double, y: Double): Array[Byte] = {
val buf = ByteBuffer.allocate(21).order(ByteOrder.LITTLE_ENDIAN)
buf.put(1.toByte); buf.putInt(1); buf.putDouble(x); buf.putDouble(y); buf.array()
}
val schema = StructType(Seq(StructField("pt", BinaryType()), StructField("val", DoubleType()), StructField("region", StringType())))
val rows = Seq((mkPt(0.1,0.1),10.0,"R1"),(mkPt(0.9,0.1),20.0,"R1"),(mkPt(0.1,0.9),30.0,"R1"),(mkPt(0.9,0.9),40.0,"R1"))
val df = spark.createDataFrame(spark.sparkContext.parallelize(rows.map { case (p,v,r) => org.apache.spark.sql.Row(p,v,r) }), schema)
val result = df.groupBy("region").agg(
rx.rst_gridfrompoints_agg(col("pt"), col("val"), lit(0.0), lit(0.0), lit(1.0), lit(1.0), lit(8), lit(8), lit(4326)).alias("idw")
)
result.show()
+------+-----------------------------------------------------------+
|region|idw |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_h3_rasterize_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio + h3. Aggregate — groupBy(...).agg(rx.rst_h3_rasterize_agg("cellid", "value", ...)) burns each H3 cell's centroid pixel (or spatial-envelope pixels with mode='spatial_envelope') into one raster tile per group. When value is omitted or null, all burned pixels carry 1.0 (presence mask). The extent and pixel size are derived automatically from the H3 resolution unless explicit bounds are supplied; kring_pad (default 1) expands the canvas by that many rings to avoid clipping edge cells.
Heavyweight gbx_rst_h3_rasterize_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_h3_rasterize_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
region_id AS cellid,
gbx_rst_fromcontent(
gbx_rst_h3_rasterize_agg(cellid, burn_value, 4326, null, null,null,null,null, null,null, 'centroids', 1),
'GTiff'
) AS tile
FROM h3_cell_values
GROUP BY region_id
Streaming aggregator that burns H3 cell centroid pixels (or spatial-envelope pixels) into one raster tile per group. This is the inverse of rst_h3_rastertogrid*: where those functions reduce raster pixels to per-cell statistics, rst_h3_rasterize_agg reconstructs a raster from per-cell values. Use rst_frombands_agg to stack per-threshold rasters (each produced by one rst_h3_rasterize_agg call) into a single multi-band output.
out_crs parameterThe lightweight Python binding accepts an optional trailing out_crs (string CRS) argument that the heavyweight/Scala tier does not. When supplied it overrides the integer out_srid for the output CRS; the heavyweight tier accepts only the integer out_srid (its builder() is strictly 12-argument). This is a lightweight superset, not a heavyweight regression.
The H3 Rasterize notebook walks this through end to end on a San Francisco Bay Area DEM: elevation isobands → H3 polyfill → a shared canvas from rst_h3_gridspec → per-band rst_h3_rasterize_agg → multi-band stack via rst_frombands_agg, visualized with the gbx.vizx helpers. The same pattern maps directly to a telco multi-threshold signal-coverage stack.
Signature: rst_h3_rasterize_agg(cellid: Column, value: Column, out_srid: Column, pixel_size: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, mode: Column, kring_pad: Column): Column
Parameters:
cellid— H3 cell ID (BIGINT or STRING) to burn (one per row)value— numeric burn value; passnullfor a presence mask (all pixels →1.0)srid— EPSG code for the output CRS; defaults to4326(WGS 84)pixel_size— ground resolution in CRS units (derives from H3 resolution whennull)xmin/ymin/xmax/ymax— output canvas extent; auto-computed from cell bounds +kring_padwhennullwidth/height— output raster dimensions in pixels; auto-derived from extent + pixel_size whennullmode—'centroids'(default, burns the cell-centroid pixel only) or'spatial_envelope'(burns all pixels inside the hexagon envelope)kring_pad— ring count by which to expand the auto-computed canvas (default1)
Multi-row fixture: 3 H3 resolution-9 cell rows with burn values 1.0/2.0/3.0. All tabs use the same grouped-agg rasterize invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Rasterize H3 cells into one raster tile per region. Each cell's value is
-- burned at the cell centroid pixel.
SELECT region_id,
gbx_rst_h3_rasterize_agg(
cellid, burn_value,
4326, cast(null as double),
cast(null as double), cast(null as double),
cast(null as double), cast(null as double),
cast(null as int), cast(null as int),
'centroids', cast(1 as int)
) AS tile
FROM h3_cell_values
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|tile |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(tile, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|tile |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
import h3
res = 9
cell_strs = [
h3.latlng_to_cell(0.01, 0.01, res),
h3.latlng_to_cell(0.02, 0.01, res),
h3.latlng_to_cell(0.01, 0.02, res),
]
cells = [h3.str_to_int(c) for c in cell_strs]
rows = [
(int(cells[0]), 1.0, "R1"),
(int(cells[1]), 2.0, "R1"),
(int(cells[2]), 3.0, "R1"),
]
from pyspark.sql.types import (
DoubleType,
LongType,
StringType,
StructField,
StructType,
)
schema = StructType(
[
StructField("cellid", LongType()),
StructField("value", DoubleType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(rx.rst_h3_rasterize_agg("cellid", "value").alias("tile"))
.first()
)
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
import h3
res = 9
cell_strs = [
h3.latlng_to_cell(0.01, 0.01, res),
h3.latlng_to_cell(0.02, 0.01, res),
h3.latlng_to_cell(0.01, 0.02, res),
]
cells = [h3.str_to_int(c) for c in cell_strs]
rows = [
(int(cells[0]), 1.0, "R1"),
(int(cells[1]), 2.0, "R1"),
(int(cells[2]), 3.0, "R1"),
]
from pyspark.sql.types import (
DoubleType,
LongType,
StringType,
StructField,
StructType,
)
schema = StructType(
[
StructField("cellid", LongType()),
StructField("value", DoubleType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
# cellid + value are the only required args; extent/size/mode/kring and the
# EPSG:4326 out_srid all take their defaults (same as the lightweight tier).
result = (
df.groupBy("region")
.agg(rx.rst_h3_rasterize_agg(df["cellid"], df["value"]).alias("tile"))
.first()
)
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
rx.register(spark)
// Multi-row fixture: 3 H3 resolution-9 cell ids (as BIGINT) with burn values.
val schema = StructType(Seq(
StructField("cellid", LongType()), StructField("value", DoubleType()), StructField("region", StringType())))
// H3 res-9 cells near origin
val rows = Seq((617733151020810239L, 1.0, "R1"), (617733151021334527L, 2.0, "R1"), (617733151085035519L, 3.0, "R1"))
val df = spark.createDataFrame(spark.sparkContext.parallelize(rows.map(r => org.apache.spark.sql.Row(r._1,r._2,r._3))), schema)
val result = df.groupBy("region").agg(rx.rst_h3_rasterize_agg(col("cellid"), col("value")).alias("tile"))
result.show()
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_merge_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio (rasterio.merge). Aggregate — groupBy(...).agg(rx.rst_merge_agg("tile")) merges the group's tiles into one mosaic tile (output spans the union extent).
Heavyweight gbx_rst_merge_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_merge_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(gbx_rst_merge_agg(tile), 'GTiff') AS tile
FROM tiles
GROUP BY group_key
Signature: rst_merge_agg(tile: Column): Column — Merge tiles per group.
Multi-tile fixture: 3 per-band rows from rgb_nir_small.tif. All tabs use the same grouped-agg mosaic invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT
scene_id,
gbx_rst_merge_agg(tile) as merged_scene
FROM satellite_tiles
GROUP BY scene_id;
# Heavyweight SQL — one v2 tile struct per group:
+--------+-----------------------------------------------------------+
|scene_id|merged_scene |
+--------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+--------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(merged_scene, 'GTiff') to rebuild a tile struct:
+--------+---------------+
|scene_id|merged_scene |
+--------+---------------+
|... |[B@... (BINARY)|
+--------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
df = _get_multi_band_tiles_df(spark)
result = df.groupBy("region").agg(rx.rst_merge_agg("tile").alias("mosaic")).first()
+------+-----------------------------------------------------------+
|region|mosaic |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
df = _get_multi_band_tiles_df_heavy(spark)
result = df.groupBy("region").agg(rx.rst_merge_agg("tile").alias("mosaic")).first()
+------+-----------------------------------------------------------+
|region|mosaic |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Multi-tile fixture: load multiband tif and split to 3 per-band rows.
val mb = spark.read.format("gdal").load("src/test/resources/binary/geotiff-small/rgb_nir_small.tif")
val b1 = mb.select(rx.rst_band(col("tile"), lit(1)).alias("tile")).withColumn("region", lit("R1"))
val b2 = mb.select(rx.rst_band(col("tile"), lit(2)).alias("tile")).withColumn("region", lit("R1"))
val b3 = mb.select(rx.rst_band(col("tile"), lit(3)).alias("tile")).withColumn("region", lit("R1"))
val df = b1.union(b2).union(b3)
val result = df.groupBy("region").agg(rx.rst_merge_agg(col("tile")).alias("mosaic"))
result.show()
+------+-----------------------------------------------------------+
|region|mosaic |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_quadbin_rasterize_agg
LightweightHeavyweight Grouped-agg UDFThe lightweight implementation is backed by pygx._quadbin cell math and rasterio — it burns each quadbin cell's geometry/value into the output band via rasterio.features.rasterize, matching the heavyweight cell set and burn values exactly.
Pixels not covered by any geometry in the group are set to -9999.0 (band-registered NoData). Filter or mask downstream via gbx_rst_getnodata or IS NULL on the extracted band value.
Heavyweight gbx_rst_quadbin_rasterize_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_quadbin_rasterize_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
cell_id AS cellid,
gbx_rst_fromcontent(gbx_rst_quadbin_rasterize_agg(cellid, burn_value), 'GTiff') AS tile
FROM quadbin_cell_values
GROUP BY cell_id
Streaming aggregator that burns quadbin cell geometry/value pairs (one row per cell) into a single rasterized tile per group. The input raster is interpreted as EPSG:4326 (lon/lat); resolution is the quadbin zoom level (0..26). The inverse of rst_quadbin_rastertogrid*: where those functions reduce raster pixels to per-cell statistics, this one synthesizes a raster from per-cell values.
out_crs parameterThe lightweight Python binding accepts an optional trailing out_crs (string CRS) argument that the heavyweight/Scala tier does not. When supplied it overrides the integer out_srid for the output CRS; the heavyweight tier accepts only the integer out_srid (its builder() is strictly 12-argument). This is a lightweight superset, not a heavyweight regression.
Signature: rst_quadbin_rasterize_agg(cellid: Column, value: Column, out_srid: Column, pixel_size: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, mode: Column, kring_pad: Column): Column
Parameters: cellid — quadbin cell ID (BIGINT); value — numeric burn value; out_srid — EPSG code for the output CRS; pixel_size — output raster cell size in the target CRS; xmin/ymin/xmax/ymax — output extent in the target CRS; width/height — output raster dimensions in pixels; mode — aggregation mode for overlapping values (typically "last"); kring_pad — cell neighbourhood expansion (typically 0)
Multi-row fixture: 3 quadbin zoom-12 cell rows near central London with burn values 1.0/2.0/3.0. All tabs use the same grouped-agg rasterize invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Rasterize quadbin cells into one raster tile per region. Each cell's value
-- is burned at the cell centroid pixel; the extent auto-derives from the cell
-- set (null canvas args). cellid is BIGINT.
SELECT region_id,
gbx_rst_quadbin_rasterize_agg(
cellid, burn_value,
4326, cast(null as double),
cast(null as double), cast(null as double),
cast(null as double), cast(null as double),
cast(null as int), cast(null as int),
'centroids', cast(0 as int)
) AS tile
FROM quadbin_cell_values
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|tile |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(tile, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|tile |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
from databricks.labs.gbx.gridx.quadbin import functions as qbx
qbx.register(spark)
spark.sql("""
CREATE OR REPLACE TEMP VIEW _qb_cells AS
SELECT region,
gbx_quadbin_pointascell(cast(lon as double), cast(lat as double), 12) AS cellid,
cast(val as double) AS value
FROM (VALUES
('R1', -0.10, 51.50, 1.0),
('R1', -0.11, 51.51, 2.0),
('R1', -0.09, 51.49, 3.0)
) AS t(region, lon, lat, val)
""")
df = spark.table("_qb_cells")
result = (
df.groupBy("region")
.agg(rx.rst_quadbin_rasterize_agg("cellid", "value").alias("tile"))
.first()
)
spark.catalog.dropTempView("_qb_cells")
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
from databricks.labs.gbx.gridx.quadbin import functions as qbx
qbx.register(spark)
spark.sql("""
CREATE OR REPLACE TEMP VIEW _qb_cells_heavy AS
SELECT region,
gbx_quadbin_pointascell(cast(lon as double), cast(lat as double), 12) AS cellid,
cast(val as double) AS value
FROM (VALUES
('R1', -0.10, 51.50, 1.0),
('R1', -0.11, 51.51, 2.0),
('R1', -0.09, 51.49, 3.0)
) AS t(region, lon, lat, val)
""")
df = spark.table("_qb_cells_heavy")
# cellid + value are the only required args; extent/size/mode/kring and the
# EPSG:4326 out_srid all take their defaults (same as the lightweight tier).
result = (
df.groupBy("region")
.agg(rx.rst_quadbin_rasterize_agg(df["cellid"], df["value"]).alias("tile"))
.first()
)
spark.catalog.dropTempView("_qb_cells_heavy")
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import com.databricks.labs.gbx.gridx.{functions => gx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
rx.register(spark)
gx.register(spark)
// Multi-row fixture: 3 quadbin zoom-12 cells near central London.
val schema = StructType(Seq(StructField("region", StringType()), StructField("lon", DoubleType()), StructField("lat", DoubleType()), StructField("val", DoubleType())))
val rows = Seq(("R1",-0.10,51.50,1.0),("R1",-0.11,51.51,2.0),("R1",-0.09,51.49,3.0))
val raw = spark.createDataFrame(spark.sparkContext.parallelize(rows.map(r => org.apache.spark.sql.Row(r._1,r._2,r._3,r._4))), schema)
raw.createOrReplaceTempView("_qb_src")
val df = spark.sql("SELECT region, gbx_quadbin_pointascell(lon, lat, 12) AS cellid, val AS value FROM _qb_src")
val result = df.groupBy("region").agg(rx.rst_quadbin_rasterize_agg(col("cellid"), col("value")).alias("tile"))
result.show()
+------+-----------------------------------------------------------+
|region|tile |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
rst_rasterize_agg
LightweightHeavyweight Grouped-agg UDFPowered by rasterio (rasterio.features). Aggregate — burns the group's (geom, value) rows into one tile over the given extent/size/SRID (last-wins on overlap).
Heavyweight gbx_rst_rasterize_agg returns a tile STRUCT<cellid, raster, path, window, clip_polygon, clip_crs, crs, metadata> (the v2 8-field tile); the lightweight SQL function returns BINARY (the raster bytes). A PySpark grouped-aggregate pandas_udf cannot return a StructType, so the lightweight SQL aggregate returns the raster payload as BINARY. The lightweight Python wrapper rx.rst_rasterize_agg(...) returns the full tile struct (it composes the aggregate with a tile-wrapping step), so only raw SQL differs. To rebuild the tile-struct equivalent in SQL, select the group key as cellid and wrap the BINARY with gbx_rst_fromcontent:
-- Lightweight SQL: rebuild the (cellid, raster) the heavyweight struct would carry
SELECT
group_key AS cellid,
gbx_rst_fromcontent(
gbx_rst_rasterize_agg(geom, value, 0,0,10,10, 8,8, 32633),
'GTiff'
) AS tile
FROM features
GROUP BY group_key
Streaming aggregator that burns geometry/value pairs (one row per feature) into a single rasterized tile per group; use when features arrive as individual rows rather than as a pre-built collection.
Signature: rst_rasterize_agg(geom: Column, value: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width: Column, height: Column, out_srid: Column): Column
Parameters: geom — WKB geometry to burn; value — numeric burn value; xmin/ymin/xmax/ymax — output extent (in the target CRS); width/height — output raster dimensions in pixels; srid — EPSG code for the output CRS
Multi-row fixture: 3 polygon rows burned at values 1.0/2.0/3.0 over a [0,0,4,4] EPSG:4326 extent. All tabs use the same grouped-agg burn invocation.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Aggregate per-feature burn values into one rasterized tile per region.
SELECT region_id,
gbx_rst_rasterize_agg(
geom_wkb, burn_value,
bbox_xmin, bbox_ymin, bbox_xmax, bbox_ymax,
256, 256, 4326
) AS tile
FROM features
GROUP BY region_id;
# Heavyweight SQL — one v2 tile struct per group:
+---------+-----------------------------------------------------------+
|region_id|tile |
+---------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+---------+-----------------------------------------------------------+
# Lightweight SQL — raster bytes as BINARY (see the note above); wrap with
# gbx_rst_fromcontent(tile, 'GTiff') to rebuild a tile struct:
+---------+---------------+
|region_id|tile |
+---------+---------------+
|... |[B@... (BINARY)|
+---------+---------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
BinaryType,
DoubleType,
StringType,
StructField,
StructType,
)
# WKB POLYGON((0 0, 4 0, 4 4, 0 4, 0 0)) in EPSG:4326
poly = bytes.fromhex(
"0103000000010000000500000000000000000000000000000000000000"
"0000000000001040000000000000000000000000000010400000000000001040"
"000000000000000000000000000010400000000000000000"
"0000000000000000"
)
rows = [(poly, 1.0, "R1"), (poly, 2.0, "R1"), (poly, 3.0, "R1")]
schema = StructType(
[
StructField("geom", BinaryType()),
StructField("value", DoubleType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_rasterize_agg(
"geom",
"value",
f.lit(0.0),
f.lit(0.0),
f.lit(4.0),
f.lit(4.0),
f.lit(8),
f.lit(8),
f.lit(4326),
).alias("burned")
)
.first()
)
+------+-----------------------------------------------------------+
|region|burned |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
BinaryType,
DoubleType,
StringType,
StructField,
StructType,
)
# WKB POLYGON((0 0, 4 0, 4 4, 0 4, 0 0)) in EPSG:4326
poly = bytes.fromhex(
"0103000000010000000500000000000000000000000000000000000000"
"0000000000001040000000000000000000000000000010400000000000001040"
"000000000000000000000000000010400000000000000000"
"0000000000000000"
)
rows = [(poly, 1.0, "R1"), (poly, 2.0, "R1"), (poly, 3.0, "R1")]
schema = StructType(
[
StructField("geom", BinaryType()),
StructField("value", DoubleType()),
StructField("region", StringType()),
]
)
df = spark.createDataFrame(rows, schema)
result = (
df.groupBy("region")
.agg(
rx.rst_rasterize_agg(
"geom",
"value",
f.lit(0.0),
f.lit(0.0),
f.lit(4.0),
f.lit(4.0),
f.lit(8),
f.lit(8),
f.lit(4326),
).alias("burned")
)
.first()
)
+------+-----------------------------------------------------------+
|region|burned |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Multi-row fixture: 3 rows with WKB polygons + burn values over a 4x4 EPSG:4326 canvas.
spark.sql("CREATE OR REPLACE TEMP VIEW _rst_agg_src AS SELECT 'R1' AS region, unhex('010300000001000000050000000000000000000000000000000000000000000000000000000000000000001040000000000000104000000000000000000000000000001040000000000000104000000000000000000000000000000000') AS geom, 1.0 AS value UNION ALL SELECT 'R1', unhex('010300000001000000050000000000000000000000000000000000000000000000000000000000000000001040000000000000104000000000000000000000000000001040000000000000104000000000000000000000000000000000'), 2.0 UNION ALL SELECT 'R1', unhex('010300000001000000050000000000000000000000000000000000000000000000000000000000000000001040000000000000104000000000000000000000000000001040000000000000104000000000000000000000000000000000'), 3.0")
val df = spark.table("_rst_agg_src")
val result = df.groupBy("region").agg(
rx.rst_rasterize_agg(col("geom"), col("value"), lit(0.0), lit(0.0), lit(4.0), lit(4.0), lit(8), lit(8), lit(4326)).alias("burned")
)
result.show()
+------+-----------------------------------------------------------+
|region|burned |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2 Tile per group — raster bytes populated, path null)
Constructor Functions
Create or load rasters from path, binary content, or bands (4 total).
rst_dtmfromgeoms
LightweightHeavyweightPowered by rasterio + SciPy (scipy.spatial.Delaunay). Barycentric interpolation over an unconstrained Delaunay TIN; cells outside the convex hull are NoData. breaklines, merge_tolerance, and snap_tolerance are accepted for signature parity but not enforced — the heavyweight tier builds a constrained TIN that honors them.
Create a DTM raster tile via TIN/Delaunay interpolation from an array of Z-valued point WKB geometries, with an optional array of breakline WKB geometries to preserve sharp terrain transitions.
out_crs parameterThe lightweight Python binding accepts an optional trailing out_crs (string CRS) argument that the SQL and heavyweight/Scala tiers do not — the heavyweight RST_DTMFromGeoms builder is strictly ≤12-argument (…, out_srid, [no_data]) and rejects a 13th argument. In lightweight Python, out_crs (string) wins over the int out_srid. This is a lightweight superset, not a heavyweight regression.
Signature: rst_dtmfromgeoms(points_array: Column, breaklines_array: Column, merge_tolerance: Column, snap_tolerance: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width_px: Column, height_px: Column, out_srid: Column, [no_data: Column = null]): Column — output CRS via out_srid (int); input points assumed already in the output CRS. See Coordinate Reference Systems.
Parameters: points_array — Array of WKB point geometries with Z coordinates; breaklines_array — Array of WKB line/polygon geometries enforcing hard edges (pass null or empty array if unused); merge_tolerance/snap_tolerance — Delaunay triangulation tolerances (vertex-merge distance and snapping distance; small values such as 0.0 and 0.01 are typical); xmin/ymin/xmax/ymax — output extent in CRS units; width_px/height_px — output raster dimensions in pixels (for N-metre cells set width_px = round((xmax-xmin)/N)); out_srid — EPSG code for the output CRS. An optional trailing no_data argument overrides the default fill for cells outside the triangulated hull.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- TIN interpolation from arrays of Z-valued point WKB and breakline WKB.
-- Output is a 100 x 100 Float64 GTiff over the extent. For N-metre cells set
-- width_px = round((xmax-xmin)/N): here a 1000 m extent at 10 m cells -> 100 px.
SELECT gbx_rst_dtmfromgeoms(
points_wkb_array, breaklines_wkb_array,
0.0, 0.01,
0.0, 0.0, 1000.0, 1000.0,
100, 100, 32633
) AS dtm
FROM survey_points;
+-----------------------------------------------------------+
|dtm |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import StructType, StructField, ArrayType, BinaryType
rx.register(spark)
# Create a synthetic survey point DataFrame with 4 points inset from the grid
# edges (Delaunay needs the hull to cover interior cells).
# POINT Z geometries: (100,100,50), (900,100,80), (900,900,120), (100,900,60)
# Using explicit schema to avoid type inference errors
survey_data = [
(
[
bytes.fromhex(
"0101000080000000000000594000000000000059400000000000004940"
),
bytes.fromhex(
"01010000800000000000208c4000000000000059400000000000005440"
),
bytes.fromhex(
"01010000800000000000208c400000000000208c400000000000005e40"
),
bytes.fromhex(
"010100008000000000000059400000000000208c400000000000004e40"
),
],
[],
)
]
schema = StructType(
[
StructField("points_wkb_array", ArrayType(BinaryType()), True),
StructField("breaklines_wkb_array", ArrayType(BinaryType()), True),
]
)
df = spark.createDataFrame(survey_data, schema=schema)
result = df.select(
rx.rst_dtmfromgeoms(
f.col("points_wkb_array"),
f.col("breaklines_wkb_array"),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(1000.0),
f.lit(1000.0),
f.lit(100),
f.lit(100),
f.lit(32618),
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TIN-interpolated DTM over specified extent and pixel count)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import (
StructType,
StructField,
ArrayType,
BinaryType,
)
rx.register(spark)
# Create synthetic survey point data with 4 points inset from the grid edges
# POINT Z geometries: (100,100,50), (900,100,80), (900,900,120), (100,900,60)
# Using explicit schema to avoid type inference errors
survey_data = [
(
[
bytes.fromhex(
"0101000080000000000000594000000000000059400000000000004940"
),
bytes.fromhex(
"01010000800000000000208c4000000000000059400000000000005440"
),
bytes.fromhex(
"01010000800000000000208c400000000000208c400000000000005e40"
),
bytes.fromhex(
"010100008000000000000059400000000000208c400000000000004e40"
),
],
[],
)
]
schema = StructType(
[
StructField("points_wkb_array", ArrayType(BinaryType()), True),
StructField("breaklines_wkb_array", ArrayType(BinaryType()), True),
]
)
df = spark.createDataFrame(survey_data, schema=schema)
result = df.select(
rx.rst_dtmfromgeoms(
f.col("points_wkb_array"),
f.col("breaklines_wkb_array"),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(0.0),
f.lit(1000.0),
f.lit(1000.0),
f.lit(100),
f.lit(100),
f.lit(32618),
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TIN-interpolated DTM over specified extent and pixel count)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Create synthetic survey points with Z-valued WKB geometries
val survey = spark.createDataFrame(Seq(
(Array[Array[Byte]](/* WKB point 1 */, /* WKB point 2 */), Array[Array[Byte]]())
)).toDF("points_wkb", "breaklines_wkb")
val result = survey.select(
rx.rst_dtmfromgeoms(col("points_wkb"), col("breaklines_wkb"), lit(0.0), lit(0.01),
lit(0.0), lit(0.0), lit(1000.0), lit(1000.0), lit(100), lit(100), lit(32633)).alias("tin")
)
result.show(truncate = false)
+-----------------------------------------------------------+
|tin |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TIN-interpolated DTM over specified extent and pixel count)
rst_frombands
LightweightHeavyweightPowered by rasterio. Stacks an ARRAY of single-band tiles into one multi-band tile in array order (element 0 → band 1), preserving georeference/CRS/dtype/NoData from the first.
Create a raster from an array of band tiles.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_frombands(bands: Column): Column
Constructor — stacks per-band tiles into a multi-band tile. Example splits the multiband fixture into per-band tiles via rst_band, then re-stacks them into a 3-band tile.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT
gbx_rst_frombands(array(band1, band2, band3)) as multi_band
FROM separated_bands;
+-----------------------------------------------------------+
|multi_band |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
# Build an array of per-band tiles (band 1, band 2, band 3) then stack
with_bands = df.select(
f.array(
rx.rst_band("tile", f.lit(1)),
rx.rst_band("tile", f.lit(2)),
rx.rst_band("tile", f.lit(3)),
).alias("bands")
)
result = with_bands.select(rx.rst_frombands("bands").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(3 per-band tiles stacked back into a 3-band tile; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("multiband_rasters")
with_bands = tile_df.select(
f.array(
rx.rst_band("tile", f.lit(1)),
rx.rst_band("tile", f.lit(2)),
rx.rst_band("tile", f.lit(3)),
).alias("bands")
)
result = with_bands.select(rx.rst_frombands("bands").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(3 per-band tiles stacked back into a 3-band tile)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Constructor: extract per-band tiles from the multiband fixture, then stack them back.
val rasters = spark.table("multiband_rasters")
val withBands = rasters.select(
array(
rx.rst_band(col("tile"), lit(1)),
rx.rst_band(col("tile"), lit(2)),
rx.rst_band(col("tile"), lit(3))
).alias("bands")
)
val result = withBands.select(rx.rst_frombands(col("bands")).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(3 per-band tiles stacked back into a 3-band tile)
rst_fromcontent
LightweightHeavyweightPowered by rasterio. The driver defaults to GTiff when unspecified and outputs are re-encoded as GeoTIFF; readable input formats are limited to the GDAL build bundled with rasterio.
Create a raster from binary content.
Signature: rst_fromcontent(content: Column, driver: Column): Column
Parameters: content — Binary column; driver — GDAL driver name
Returns: Binary raster tile data
Constructor — builds a tile from binary content. Example uses Spark's binaryFile reader to load bytes, then rst_fromcontent to decode them — this is the canonical tier-agnostic pattern (works on any compute including Serverless).
rst_fromcontent materializes bytes that are already in a column. Those bytes are in executor memory before this call; GeoBrix does not guard their size. On Serverless / Spark Connect the connect-aware cap is 64 MiB per tile — keep individual content values at or under that limit. For larger sources, use the virtual-tile raster readers (the default) and let the runtime page bytes lazily. See Serverless & Memory.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Load from binary table
SELECT
path,
gbx_rst_fromcontent(content, 'GTiff') as tile
FROM binary_raster_table;
+----+-----------------------------------------------------------+
|path|tile |
+----+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+----+-----------------------------------------------------------+
from ._fixtures import single_band_path
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
path = str(single_band_path())
binary_df = spark.read.format("binaryFile").load(path)
tile_df = binary_df.select(
rx.rst_fromcontent(f.col("content"), f.lit("GTiff")).alias("tile")
)
result = tile_df.select(rx.rst_format("tile").alias("format")).first()
+------+
|format|
+------+
|GTiff |
+------+
(format of the tile loaded from binary content via binaryFile reader)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
from path_config import SAMPLE_DATA_BASE
path = f"{SAMPLE_DATA_BASE}/nyc/sentinel2/nyc_sentinel2_red.tif"
binary_df = spark.read.format("binaryFile").load(path)
tile_df = binary_df.select(
rx.rst_fromcontent(f.col("content"), f.lit("GTiff")).alias("tile")
)
result = tile_df.select(rx.rst_format("tile").alias("format")).first()
+------+
|format|
+------+
|GTiff |
+------+
(format of the tile loaded from binary content via binaryFile reader)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Constructor: binaryFile reader + rst_fromcontent is the canonical tier-agnostic pattern.
// binaryFile runs in Spark (holds the Volume credential) and works on any compute.
val binary = spark.read.format("binaryFile")
.load("/Volumes/main/default/test-data/geobrix-examples/nyc/sentinel2/nyc_sentinel2_red.tif")
val tiles = binary.select(rx.rst_fromcontent(col("content"), lit("GTiff")).alias("tile"))
val result = tiles.select(rx.rst_format(col("tile")).alias("format"))
result.show()
+------+
|format|
+------+
|GTiff |
+------+
(format of the tile loaded from binary content via binaryFile reader)
rst_fromfile
LightweightReference a raster on disk as a tile, by path.
The lightweight tier returns a virtual tile by default — bytes-free, pointing at path over its whole-file window, with width/height/CRS read from the header. No pixels are read until a downstream op needs them, so this is the lazy way to load rasters by path. There is no virtualize_dir/virtualize_prefix argument here: the source file already is the durable backing store, so nothing needs writing. A path that cannot be opened returns null.
Signature: rst_fromfile(path: Column, driver: Column = 'GTiff'[, materialize: bool = False]) — materialize is a lightweight-Python argument; from Python, materialize=True reads the pixels now and returns a materialized tile.
Parameters: path — raster file path; driver — GDAL driver-name hint carried into metadata (rasterio auto-detects the real format on open); materialize (light Python only) — True reads pixels now and returns a materialized tile, default False returns a virtual tile.
gbx_rst_fromfile is a Python UDF (requires a light-tier extra, e.g. geobrix[light_env6]; no Scala/JVM form — the executor JVM cannot read a UC Volume /Volumes/... FUSE path, whose credential is held only by Spark's managed Python worker). The SQL call is the same 2-argument form in both tiers, but the tier that registered decides the result: the lightweight registration (pyrx) returns a virtual tile, the heavyweight registration (rasterx) returns a materialized tile (JVM/heavy callers cannot use a virtual path-only tile). Whichever register() ran last wins. From Python, pass materialize=True to force bytes regardless of tier. Without a light-tier extra the function is not registered and the Python binding raises with guidance.
materialize=True errors over the Serverless capCalling rst_fromfile(..., materialize=True) on a file larger than the connect-aware stream cap (64 MiB on Serverless / Spark Connect, 256 MiB on classic) raises a ValueError. The error is intentional — silently materializing an oversized tile would OOM the executor. The virtual default (materialize=False, which is the lightweight default) is Serverless-safe: pixels are read lazily, size-gated, at each downstream operation. See Serverless & Memory.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- gbx_rst_fromfile is a Python UDF (no JVM form; requires a light-tier extra, e.g. geobrix[light_env6]). The
-- SQL call is the same 2-argument form in both tiers — the tier you register
-- decides the result (whichever register() ran last wins):
-- Lightweight (rx.register / pyrx): returns a VIRTUAL tile (bytes-free,
-- path + whole-file window; lazy).
-- Heavyweight (rasterx register): returns a MATERIALIZED tile (raster
-- bytes present) — JVM/heavy callers
-- cannot use a virtual path-only tile.
SELECT
gbx_rst_fromfile('/Volumes/main/geobrix_samples/nyc/sentinel2.tif', 'GTiff') AS tile;
-- Either way, accessors read what they need — width/height come from the header
-- (no pixel read even for the virtual tile):
SELECT
path,
gbx_rst_width(gbx_rst_fromfile(path, 'GTiff')) as width,
gbx_rst_height(gbx_rst_fromfile(path, 'GTiff')) as height
FROM raster_paths;
# Both tiers return the SAME v2 tile struct (cellid, raster, path, window, ...);
# only the field values differ — virtual carries the path, materialized the bytes.
# Lightweight registration — a VIRTUAL v2 tile (raster null; path + window set):
+---------------------------------------------------------------+
|tile |
+---------------------------------------------------------------+
|{0, null, /Volumes/..., {0, 0, 10980, 10980}, ..., null, {...}}|
+---------------------------------------------------------------+
# Heavyweight registration — a MATERIALIZED v2 tile (raster bytes; path null):
+------------------------------------------------------------+
|tile |
+------------------------------------------------------------+
|{0, <raster bytes>, null, null, ..., {driver -> GTiff, ...}}|
+------------------------------------------------------------+
# width/height (either tier) read from the header:
+----+-----+------+
|path|width|height|
+----+-----+------+
|... |10980|10980 |
+----+-----+------+
from ._fixtures import single_band_path
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
path = str(single_band_path())
path_df = spark.createDataFrame([(path,)], ["path"])
# Default — a VIRTUAL v2 tile (same struct as always, just bytes-free):
+------------------------------------------------------------+
|tile |
+------------------------------------------------------------+
|{0, null, /Volumes/..., {0, 0, 236, 161}, ..., null, {...}} |
+------------------------------------------------------------+
(raster is null → bytes-free; path + whole-file window carry the reference)
# rst_fromfile("path", materialize=True) instead returns a MATERIALIZED v2 tile:
# {0, <raster bytes>, null, null, ..., {driver -> GTiff, ...}} (raster set)
from ._fixtures import single_band_path
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
path = str(single_band_path())
path_df = spark.createDataFrame([(path,)], ["path"])
# Heavy rst_fromfile delegates to pyrx with materialize=True → bytes present.
return path_df.select(
rx.rst_fromfile("path", f.lit("GTiff")).alias("tile")
).first()["tile"]
+------------------------------------------------------------+
|tile |
+------------------------------------------------------------+
|{0, <raster bytes>, null, null, ..., {driver -> GTiff, ...}}|
+------------------------------------------------------------+
(a MATERIALIZED v2 tile: raster bytes present, path/window null)
Not available in this tier.
rst_fromfile pulls columnar raster paths into a tile column inline. To ingest rasters as a normal Spark job — partitioned parallel reads, optional tiling (sizeInMB), FUSE-safe Volume staging — use the Raster Reader (raster_gbx / gtiff_gbx, or heavyweight gdal / gtiff_gdal): spark.read.format("raster_gbx").load(path). It, too, emits virtual tiles by default.
rst_fromcontentIf you already hold raster bytes in a column (e.g. from Spark's built-in binaryFile reader), build the tile with gbx_rst_fromcontent(content, driver). Note this is not a lazy equivalent of rst_fromfile in the lightweight tier: rst_fromcontent takes bytes you have already read, so it always produces a materialized tile. (In the heavyweight tier all tiles are materialized, so there the two are interchangeable.) It also works without a light-tier extra and on any compute, since the binaryFile reader runs in Spark, which holds the Volume credential:
df = (
spark.read.format("binaryFile")
.load("/Volumes/main/geobrix_samples/geobrix-examples/nyc/*.tif")
.selectExpr("path", "gbx_rst_fromcontent(content, 'GTiff') AS tile")
)
rst_gridfrompoints
LightweightHeavyweightPowered by rasterio + SciPy (cKDTree IDW). Inverse-distance interpolation (power, max_pts) to a single-band Float64 grid; NoData −9999. Matches the heavyweight invdist defaults (power=2.0, max_pts=12).
IDW-interpolate an array of Z-valued point geometries to a Float64 GeoTIFF tile covering an explicit bounding box and pixel grid. Supply the points and their scalar values as arrays in a single row; use rst_gridfrompoints_agg when points arrive one per row.
out_crs parameterThe lightweight Python binding accepts an optional trailing out_crs (string CRS) argument that the SQL and heavyweight/Scala tiers do not — the heavyweight RST_GridFromPoints builder is strictly ≤11-argument (…, out_srid, [power, [max_pts]]) and rejects a 12th argument. In lightweight Python, out_crs (string) wins over the int out_srid. This is a lightweight superset, not a heavyweight regression.
Signature: rst_gridfrompoints(points_array: Column, values_array: Column, xmin: Column, ymin: Column, xmax: Column, ymax: Column, width_px: Column, height_px: Column, out_srid: Column, [power: Column, [max_pts: Column]]): Column — output CRS via out_srid (int); input points are assumed already in the output CRS. See Coordinate Reference Systems.
Parameters: points_array — ARRAY<BINARY> of WKB point geometries; values_array — ARRAY<DOUBLE> of scalar observations, one per point; xmin/ymin/xmax/ymax — output extent in CRS units; width_px/height_px — output dimensions in pixels; out_srid — EPSG code; power — IDW distance-decay exponent (2.0 is the standard); max_pts — maximum nearest neighbours considered per output pixel
- SQL
- Python (light)
- Python (heavy)
- Scala
-- IDW (power=2, max_points=12) from arrays of point WKB and values.
-- Output is a 256 x 256 Float64 GTiff covering the requested extent.
SELECT gbx_rst_gridfrompoints(
points_wkb_array, values_array,
0.0, 0.0, 1000.0, 1000.0,
256, 256, 32633
) AS idw
FROM point_clouds;
+-----------------------------------------------------------+
|idw |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(IDW-interpolated tile over specified extent)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from pyspark.sql.types import ArrayType, BinaryType, DoubleType
rx.register(spark)
# Create a synthetic point cloud DataFrame with WKB points and values
# Using simple WKB POINT(2.0, 2.0) and POINT(4.0, 4.0)
point_data = [
(
[
bytes.fromhex("010100000000000000000000400000000000000040"),
bytes.fromhex("010100000000000000000008400000000000000840"),
],
[100.0, 110.0],
)
]
df = spark.createDataFrame(
point_data,
[
"points_wkb_array",
"values_array",
],
)
result = df.select(
rx.rst_gridfrompoints(
f.col("points_wkb_array"),
f.col("values_array"),
f.lit(0.0),
f.lit(0.0),
f.lit(1000.0),
f.lit(1000.0),
f.lit(256),
f.lit(256),
f.lit(32633),
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(IDW-interpolated tile over specified extent)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
# Create synthetic point cloud: 2 points with WKB and values
point_data = [
(
[
bytes.fromhex("010100000000000000000004400000000000000040"),
bytes.fromhex("010100000000000000000008400000000000000040"),
],
[100.0, 110.0],
)
]
df = spark.createDataFrame(point_data, ["points_wkb_array", "values_array"])
result = df.select(
rx.rst_gridfrompoints(
f.col("points_wkb_array"),
f.col("values_array"),
f.lit(0.0),
f.lit(0.0),
f.lit(1000.0),
f.lit(1000.0),
f.lit(256),
f.lit(256),
f.lit(32633),
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(IDW-interpolated tile over specified extent)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
// Create synthetic point cloud with WKB-encoded points and values
val points = spark.createDataFrame(Seq(
(Array[Array[Byte]](/* WKB point 1 */, /* WKB point 2 */), Array[Double](100.0, 110.0))
)).toDF("points_wkb", "values")
val result = points.select(
rx.rst_gridfrompoints(col("points_wkb"), col("values"), lit(0.0), lit(0.0),
lit(1000.0), lit(1000.0), lit(256), lit(256), lit(32633)).alias("idw")
)
result.show(truncate = false)
+-----------------------------------------------------------+
|idw |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(IDW-interpolated tile over specified extent)
Generator Functions
Produce multiple tiles or bands (7 total).
rst_bng_tessellate
LightweightHeavyweight Streaming UDTFThe lightweight implementation is backed by pygx._bng cell math and rasterio. It auto-warps the input to EPSG:27700 via rasterio.warp, enumerates the overlapping BNG cells, and clips one tile per cell — rendering STRING BNG cell IDs at the row boundary and matching the heavyweight cell set and clip windows.
Tessellate a raster to British National Grid (BNG) cells — one row per overlapping cell, each clipped to that cell's extent. The raster is automatically reprojected to EPSG:27700 (British National Grid) before tessellation, so any projected or geographic input is handled transparently. Resolution accepts integer indices ±1..±6 (1 = 100 km, 2 = 10 km, 3 = 1 km, 4 = 100 m, 5 = 10 m, 6 = 1 m; negative indices address quadrant sub-cells) or string keys from BNG.resolutionMap (e.g. "1km", "100m"). Cell IDs are STRING (e.g. "TQ28", "SU3412").
Natural fit for CV image-tiling workflows that require Ordnance Survey–scale cells aligned to the national grid — for example tiling aerial or satellite imagery into 1 km BNG cells for object-detection model inference. See issue #49 for the background.
Signature: rst_bng_tessellate(tile: Column, resolution: Column): Column
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: the generator in SELECT explodes to one row per overlapping BNG
-- cell (covering mode, default; pass 'centroid' as the 3rd arg for
-- pixel-centroid single-assignment). '1km' == integer resolution 3; a raster in
-- any CRS is warped to EPSG:27700 first.
SELECT gbx_rst_bng_tessellate(tile, '1km', 'covering') FROM rasters;
-- Lightweight (pyrx): registered as a streaming table function — call with LATERAL.
SELECT t.* FROM rasters, LATERAL gbx_rst_bng_tessellate(tile, '1km', 'covering') t;
+------+------+--------------+
|source|cellid|raster |
+------+------+--------------+
|... |TQ2979|<raster bytes>|
+------+------+--------------+
(SELECT t.* expands the v2-tile struct; cellid is the BNG grid-square STRING)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
# A 2km x 2km square over central London, in EPSG:27700 metres.
london_wkt = (
"POLYGON((529000 179000, 531000 179000, "
"531000 181000, 529000 181000, 529000 179000))"
)
london = spark.range(1).select(
rx.rst_rasterize(
f.lit(london_wkt),
f.lit(1.0),
f.lit(529000.0),
f.lit(179000.0),
f.lit(531000.0),
f.lit(181000.0),
f.lit(200),
f.lit(200),
f.lit(27700),
).alias("tile")
)
london.createOrReplaceTempView("rasters")
return spark.sql(
"SELECT t.* FROM rasters, LATERAL gbx_rst_bng_tessellate(tile, 3) t"
).take(3)
+------+-----------------------------------------------------------+
|cellid|... |
+------+-----------------------------------------------------------+
|TQ2979|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
|TQ2980|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
|TQ3079|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(one v2-Tile row per 1km BNG cell overlapping the London raster; cellid is the
BNG grid-square STRING)
from pyspark.sql import functions as f
if rx is None:
raise ImportError("rasterx not installed")
london_wkt = (
"POLYGON((529000 179000, 531000 179000, "
"531000 181000, 529000 181000, 529000 179000))"
)
df = spark.range(1).select(
rx.rst_rasterize(
f.lit(london_wkt),
f.lit(1.0),
f.lit(529000.0),
f.lit(179000.0),
f.lit(531000.0),
f.lit(181000.0),
f.lit(200),
f.lit(200),
f.lit(27700),
).alias("tile")
)
+-----------------------------------------------------------+
|bng_cell |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(one v2-tile-struct row per 1km BNG cell overlapping the London raster; the
generator explodes to rows in a plain select)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_bng_tessellate(col("tile"), lit(3)).alias("bng_cells"))
result.show(truncate = false)
+---+
|bng|
+---+
|[{ |
+---+
(array of tile structs per BNG cell; raster rewarped to EPSG:27700)
rst_h3_tessellate
LightweightHeavyweight Streaming UDTFPowered by rasterio + h3. Clips the raster to each overlapping H3 cell at resolution, returning one row per cell via streaming UDTF (matches the heavyweight generator behavior).
Signature: rst_h3_tessellate(tile: Column, resolution: Column, mode: Column = "covering"): Column — Tessellate raster to H3 cells. mode is "covering" (default — every overlapping cell, clipped) or "centroid" (pixel-centroid single-assignment partition). See H3 Raster Tessellation for the full mode guide.
Covering mode uses a positive-area overlap rule (shared by all three grids — rst_h3_tessellate, rst_quadbin_tessellate, rst_bng_tessellate): a cell is emitted iff its geometry has greater-than-zero area overlap with the raster. A cell that merely touches the raster along a boundary edge or corner (zero pixel overlap — common on grid-aligned tiles where the raster edges land on cell boundaries) is excluded. A within-extent cell whose pixels are all NoData is emitted — NoData renders in place, it does not punch a gap into the mosaic. This rule is identical on the lightweight and heavyweight tiers, so covering-mode cell sets match exactly across tiers.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: the generator in SELECT explodes to one row per overlapping H3
-- cell, clipped to its hexagon (covering mode, default; pass 'centroid' as the
-- 3rd arg for pixel-centroid single-assignment).
SELECT gbx_rst_h3_tessellate(tile, 7, 'covering') FROM rasters;
-- Lightweight (pyrx): registered as a streaming table function — call with LATERAL.
SELECT t.* FROM rasters, LATERAL gbx_rst_h3_tessellate(tile, 7, 'covering') t;
+------+------------------+--------------+
|source|cellid |raster |
+------+------------------+--------------+
|... |599686042433355775|<raster bytes>|
+------+------------------+--------------+
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
single_band_tile_df(spark).createOrReplaceTempView("rasters")
return spark.sql(
"SELECT t.* FROM rasters, LATERAL gbx_rst_h3_tessellate(tile, 3) t"
).take(3)
+-------------------+-----------------------------------------------------------+
|cellid |... |
+-------------------+-----------------------------------------------------------+
|577586652210266111 |{..., <raster bytes>, ..., {driver -> GTiff, ...}} |
+-------------------+-----------------------------------------------------------+
(one v2-Tile row per H3 cell, cellid = the H3 index)
from pyspark.sql import functions as f
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("rasters")
# Partition into H3 cells (covering mode, returns array of structs)
result = df.select(rx.rst_h3_tessellate("tile", f.lit(7)).alias("h3_cells")).first()
+---+
|h3_|
+---+
|[{ |
+---+
(array of structs: [{cellid: LONG, raster: BINARY}, ...])
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_h3_tessellate(col("tile"), lit(7)).alias("h3_cells"))
result.show(truncate = false)
+---+
|h3_|
+---+
|[{ |
+---+
(array of structs: [{cellid: LONG, raster: BINARY}, ...])
rst_maketiles
LightweightHeavyweight Streaming UDTFPowered by rasterio. Streams one tile row per subdivided region via streaming UDTF. It derives a square tile size from the MB budget and always partitions; it does not honor the heavyweight size_in_mb = -1 (single tile) or 0 (64 MB) sentinels or the power-of-four split, so tile counts and dimensions differ.
Signature: rst_maketiles(tile: Column, sizeInMB: Column): Column — Subdivide into smaller tiles by approximate size in MB.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Subdivide into MB-sized tiles using LATERAL. The second argument is a target
-- size in MB, not pixel dimensions; the tile grid is derived from the MB budget.
SELECT t.*
FROM rasters,
LATERAL gbx_rst_maketiles(tile, 4) t;
+------+--------------+----+----------------------+
|cellid|raster |path|... |
+------+--------------+----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+----+----------------------+
(one row per sub-tile; t.* expands the v2-Tile struct fields)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
single_band_tile_df(spark).createOrReplaceTempView("rasters")
# Subdivide raster into approximately 1.0 MB tiles
return spark.sql(
"SELECT t.* FROM rasters, " "LATERAL gbx_rst_maketiles(tile, 1.0) t"
).collect()
+------+--------------+-----+----------------------+
|cellid|raster |path |... |
+------+--------------+-----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+-----+----------------------+
(one row per sub-tile; t.* expands the v2-Tile struct fields)
if rx is None:
raise ImportError("rasterx not installed")
# The `rasters` view is created in Setup; subdivide into ~1.0 MB tiles via SQL LATERAL.
return spark.sql(
"SELECT t.* FROM rasters, " "LATERAL gbx_rst_maketiles(tile, 1.0) t"
).collect()
+------+--------------+-----+----------------------+
|cellid|raster |path |... |
+------+--------------+-----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+-----+----------------------+
(one row per sub-tile; t.* expands the v2-Tile struct fields)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_maketiles(col("tile"), lit(1.0)).alias("tiles"))
result.show(truncate = false)
+-----+
|tiles|
+-----+
|[{0, |
+-----+
(array of tile structs per MB subdivision)
rst_quadbin_tessellate
LightweightHeavyweight Streaming UDTFThe lightweight implementation is backed by pygx._quadbin cell math and rasterio. It enumerates the overlapping quadbin cells for the raster bbox and clips one tile per cell, matching the heavyweight cell set and clip windows.
Tessellate a raster to CARTO quadbin v0 cells — one row per overlapping cell, each clipped to that cell's extent. The input raster must be in EPSG:4326 (lon/lat); reproject upstream with rst_transform if your source CRS differs. Resolution is the quadbin zoom level (0..26). Each output row carries the quadbin cell ID (BIGINT) and the clipped tile struct.
Signature: rst_quadbin_tessellate(tile: Column, resolution: Column): Column
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: the generator in SELECT explodes to one row per overlapping
-- quadbin cell, each chip clipped to its cell (covering mode, default; pass
-- 'centroid' as the 3rd arg for pixel-centroid single-assignment). Zoom 12 for
-- a city-scale raster.
SELECT gbx_rst_quadbin_tessellate(tile, 12, 'covering') FROM rasters;
-- Lightweight (pyrx): registered as a streaming table function — call with LATERAL.
SELECT t.* FROM rasters, LATERAL gbx_rst_quadbin_tessellate(tile, 12, 'covering') t;
+------+-------------------+--------------+
|source|cellid |raster |
+------+-------------------+--------------+
|... |5250127588525215743|<raster bytes>|
+------+-------------------+--------------+
(SELECT t.* expands the v2-tile struct; cellid is the quadbin index)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
single_band_tile_df(spark).createOrReplaceTempView("rasters")
return spark.sql(
"SELECT t.* FROM rasters, LATERAL gbx_rst_quadbin_tessellate(tile, 5) t"
).take(3)
+-------------------+-----------------------------------------------------------+
|cellid |... |
+-------------------+-----------------------------------------------------------+
|5250127588525215743|{..., <raster bytes>, ..., {driver -> GTiff, ...}} |
+-------------------+-----------------------------------------------------------+
(one v2-Tile row per quadbin cell, cellid = the quadbin index)
from pyspark.sql import functions as f
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("rasters")
# Partition into quadbin cells at zoom 12 (covering mode)
result = df.select(
rx.rst_quadbin_tessellate("tile", f.lit(12)).alias("qb_cells")
).first()
+---+
|qb_|
+---+
|[{ |
+---+
(array of tile structs per quadbin cell, zoom 12)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_quadbin_tessellate(col("tile"), lit(12)).alias("qb_cells"))
result.show(truncate = false)
+---+
|qb_|
+---+
|[{ |
+---+
(array of tile structs per quadbin cell, zoom 12)
rst_retile
LightweightHeavyweight Streaming UDTFPowered by rasterio. Streams one tile row per retiled region via streaming UDTF (matches the heavyweight generator behavior).
Signature: rst_retile(tile: Column, tileWidth: Column, tileHeight: Column): Column — Retile to uniform dimensions.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT t.*
FROM rasters,
LATERAL gbx_rst_retile(tile, 256, 256) t;
+------+--------------+----+----------------------+
|cellid|raster |path|... |
+------+--------------+----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+----+----------------------+
(one row per sub-tile; t.* expands the v2-Tile struct fields)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
single_band_tile_df(spark).createOrReplaceTempView("rasters")
# Explode raster into 64x64-pixel tiles (returns array of tiles)
return spark.sql(
"SELECT t.* FROM rasters, " "LATERAL gbx_rst_retile(tile, 64, 64) t"
).take(3)
+------+--------------+-----+----------------------+
|cellid|raster |path |... |
+------+--------------+-----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+-----+----------------------+
(one row per sub-tile; t.* expands the v2-Tile struct fields)
if rx is None:
raise ImportError("rasterx not installed")
# The `rasters` view is created in Setup; retile it into 64x64 sub-tiles via SQL LATERAL.
return spark.sql(
"SELECT t.* FROM rasters, " "LATERAL gbx_rst_retile(tile, 64, 64) t"
).take(3)
+------+--------------+-----+----------------------+
|cellid|raster |path |... |
+------+--------------+-----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+-----+----------------------+
(one row per sub-tile; t.* expands the v2-Tile struct fields)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_retile(col("tile"), lit(64), lit(64)).alias("tiles"))
result.show(truncate = false)
+-----+
|tiles|
+-----+
|[{0, |
+-----+
(array of tile structs: [{0, raster, path, metadata}, ...])
rst_separatebands
LightweightHeavyweight Streaming UDTFPowered by rasterio. Streams one band-tile row per band via streaming UDTF — O(1) worker memory regardless of band count (matches the heavyweight generator behavior).
Signature: rst_separatebands(tile: Column): Column — Split multi-band into array of bands.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT t.*
FROM multiband_rasters,
LATERAL gbx_rst_separatebands(tile) t;
+----+-----------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+
|path|red_band |green_band |blue_band |
+----+-----------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+----+-----------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
multiband_tile_df(spark).createOrReplaceTempView("multiband_rasters")
# Explode multiband raster into individual band tiles
return spark.sql(
"SELECT t.* FROM multiband_rasters, " "LATERAL gbx_rst_separatebands(tile) t"
).collect()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(one row per band: 3 rows for a 3-band raster)
if rx is None:
raise ImportError("rasterx not installed")
# The `multiband_rasters` view is created in Setup; separate its bands via SQL LATERAL.
return spark.sql(
"SELECT t.* FROM multiband_rasters, " "LATERAL gbx_rst_separatebands(tile) t"
).collect()
+---+
|z |
+---+
(one row per band: each is a v2-Tile struct)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_separatebands(col("tile")).alias("bands"))
result.show(truncate = false)
+-----+
|bands|
+-----+
|[{0, |
+-----+
(array of tile structs: one per band from the multiband raster)
rst_tooverlappingtiles
LightweightHeavyweight Streaming UDTFPowered by rasterio. Streams one tile row per overlapping region via streaming UDTF (matches the heavyweight generator behavior).
Signature: rst_tooverlappingtiles(tile: Column, tileWidth: Column, tileHeight: Column, overlap: Column): Column — Create overlapping tiles.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT t.*
FROM rasters,
LATERAL gbx_rst_tooverlappingtiles(tile, 256, 256, 10) t;
+------+--------------+----+----------------------+
|cellid|raster |path|... |
+------+--------------+----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+----+----------------------+
(one row per overlapping tile; t.* expands the v2-Tile struct fields)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
single_band_tile_df(spark).createOrReplaceTempView("rasters")
# 64x64 tiles with 8% overlap
return spark.sql(
"SELECT t.* FROM rasters, "
"LATERAL gbx_rst_tooverlappingtiles(tile, 64, 64, 8) t"
).take(3)
+------+--------------+-----+----------------------+
|cellid|raster |path |... |
+------+--------------+-----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+-----+----------------------+
(one row per overlapping tile; t.* expands the v2-Tile struct fields)
if rx is None:
raise ImportError("rasterx not installed")
# The `rasters` view is created in Setup; 64x64 tiles with 8% overlap via SQL LATERAL.
return spark.sql(
"SELECT t.* FROM rasters, "
"LATERAL gbx_rst_tooverlappingtiles(tile, 64, 64, 8) t"
).take(3)
+------+--------------+-----+----------------------+
|cellid|raster |path |... |
+------+--------------+-----+----------------------+
|0 |<raster bytes>|... |{driver -> GTiff, ...}|
+------+--------------+-----+----------------------+
(one row per overlapping tile; t.* expands the v2-Tile struct fields)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_tooverlappingtiles(col("tile"), lit(64), lit(64), lit(8)).alias("tiles"))
result.show(truncate = false)
+-----+
|tiles|
+-----+
|[{0, |
+-----+
(array of overlapping tile structs with 8% overlap)
Grid Functions (H3)
Aggregate raster values to H3 grid cells, and utility functions for H3-based canvas setup (9 total).
gbx_h3_cell_bbox
LightweightHeavyweightPowered by h3. Returns a STRUCT<xmin DOUBLE, ymin DOUBLE, xmax DOUBLE, ymax DOUBLE> bounding box for the given H3 cell in the requested srid. In 'centroids' mode the box tightly wraps the centroid point; in 'spatial_envelope' mode it wraps the full hexagon outline. When kring_pad > 0 the k-ring of that radius is computed first and the bounding box covers all cells in the ring. The lightweight SQL function requires all four arguments explicitly; the Python API (rx.h3_cell_bbox(cellid, srid, mode, kring_pad)) honors the same defaults as the Scala implementation.
Scalar function — returns the bounding box STRUCT<xmin, ymin, xmax, ymax> for one H3 cell in the given CRS. Use this to drive the xmin/ymin/xmax/ymax and grid-size parameters of rst_h3_rasterize_agg when you need a consistent per-cell canvas, or to clip and inspect cell extents in downstream queries.
Signature: h3_cell_bbox(cellid: Column, srid: Column, mode: Column, kring_pad: Column): Column
Parameters:
cellid— H3 cell ID (BIGINT or STRING)srid— EPSG code;4326for WGS 84 lon/latmode—'centroids'(centroid point envelope) or'spatial_envelope'(hexagon boundary envelope)kring_pad— expand by this many k-rings before computing the bounding box;0= no expansion
Returns: STRUCT<xmin DOUBLE, ymin DOUBLE, xmax DOUBLE, ymax DOUBLE>
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Bounding box (STRUCT<xmin, ymin, xmax, ymax>) for each H3 cell in EPSG:4326.
-- Uses 'centroids' mode with no k-ring padding (kring_pad=0).
SELECT
cellid,
gbx_h3_cell_bbox(cellid, 4326, 'centroids', 0) AS bbox
FROM (
VALUES
(617733151020810239),
(617733151085035519),
(617733151021334527)
) AS t(cellid);
+------------------+------------------------------+
|cellid |bbox |
+------------------+------------------------------+
|617733151020810239|{-74.02, 40.70, -74.01, 40.71}|
+------------------+------------------------------+
(STRUCT<xmin, ymin, xmax, ymax> per H3 cell, in EPSG:4326)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.createDataFrame(
[(617733151020810239,), (617733151085035519,), (617733151021334527,)],
["cellid"],
)
return df.select(
"cellid",
rx.gbx_h3_cell_bbox("cellid", f.lit(4326), f.lit("centroids")).alias("bbox"),
).take(3)
+------------------+------------------------------+
|cellid |bbox |
+------------------+------------------------------+
|617733151020810239|{-74.02, 40.70, -74.01, 40.71}|
+------------------+------------------------------+
(STRUCT<xmin, ymin, xmax, ymax> per H3 cell, in EPSG:4326)
if rx is None:
raise ImportError("rasterx not installed")
from pyspark.sql import functions as f
df = spark.createDataFrame(
[(617733151020810239,), (617733151085035519,), (617733151021334527,)],
["cellid"],
)
return df.select(
"cellid",
rx.gbx_h3_cell_bbox("cellid", f.lit(4326), f.lit("centroids"), f.lit(0)).alias(
"bbox"
),
).take(3)
+------------------+------------------------------+
|cellid |bbox |
+------------------+------------------------------+
|617733151020810239|{-74.02, 40.70, -74.01, 40.71}|
+------------------+------------------------------+
(STRUCT<xmin, ymin, xmax, ymax> per H3 cell, in EPSG:4326)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
rx.register(spark)
// H3 res-9 cell ids (as BIGINT); scalar bbox in EPSG:4326, centroids mode.
val schema = StructType(Seq(StructField("cellid", LongType())))
val rows = Seq(617733151020810239L, 617733151085035519L, 617733151021334527L)
val df = spark.createDataFrame(spark.sparkContext.parallelize(rows.map(org.apache.spark.sql.Row(_))), schema)
val result = df.select(col("cellid"), rx.gbx_h3_cell_bbox(col("cellid"), 4326, "centroids", 0).alias("bbox"))
result.show(truncate = false)
+------------------+------------------------------+
|cellid |bbox |
+------------------+------------------------------+
|617733151020810239|{-74.02, 40.70, -74.01, 40.71}|
+------------------+------------------------------+
(STRUCT<xmin, ymin, xmax, ymax> per H3 cell, in EPSG:4326)
rst_h3_gridspec (Python / DataFrame helper)
rst_h3_gridspec is not registered as a SQL function and is not available in the heavyweight tier. It is a pure-Python / PySpark DataFrame helper in the pyrx package.
For the heavyweight tier, compose the equivalent shared canvas using the registered scalar gbx_h3_cell_bbox(cellid, srid, mode, kring_pad) with native Spark min/max aggregates and the same floor/ceil snap arithmetic:
SELECT min(cell_bbox.xmin), min(cell_bbox.ymin),
max(cell_bbox.xmax), max(cell_bbox.ymax)
FROM (SELECT gbx_h3_cell_bbox(cellid, 4326, 'centroids', 1) AS cell_bbox FROM cells)
rst_h3_gridspec computes the shared, snapped canvas (extent + pixel size) that a group of H3 cells should use so that per-threshold rasters align on a common grid and can be stacked via rst_frombands_agg or mosaicked via rst_merge_agg.
Signature:
rx.rst_h3_gridspec(df, cell_col="cellid", *group_cols,
srid=4326, pixel_size=None,
mode="centroids", kring_pad=1)
Typical multi-threshold workflow:
- Call
rx.rst_h3_gridspec(df, cell_col="cellid", srid=4326, mode='centroids', kring_pad=1)once on the distinct cell set. It returns the grouped DataFrame with agridstruct column — one row per group containing the shared canvas. - For each threshold band, run
rst_h3_rasterize_aggwith those fixed bounds — all output tiles share the same origin and pixel grid. - Stack aligned bands with
rst_frombands_agg(ordered byband_index), or mosaic per-cell tiles withrst_merge_agg.
Parameters:
df— input Spark DataFrame containing H3 cell IDscell_col— column name holding H3 cell IDs (integer or string); default"cellid"*group_cols— additional grouping columns (e.g. a transmitter ID, year, month); one grid spec row is produced per groupsrid— EPSG code for the output CRS;4326for WGS 84 (default)pixel_size— ground resolution in CRS units;None= auto-derived from the H3 resolution via an edge-length heuristic (default)mode—'centroids'(default) or'spatial_envelope'kring_pad— k-ring expansion applied per cell before computing its bounding box (default1)
Returns: the grouped DataFrame with a grid column of type:
STRUCT<xmin DOUBLE, ymin DOUBLE, xmax DOUBLE, ymax DOUBLE,
pixel_size DOUBLE, width INT, height INT, srid INT>
rst_h3_rastertogridavg
LightweightHeavyweight Streaming UDTFPowered by rasterio + h3. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_h3_rastertogridavg as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_h3_rastertogridavg(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridavg(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_h3_rastertogridavg(tile: Column, resolution: Column): Column
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_h3_rastertogridavg(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridavg(tile, 4) t;
+----+------------------+-------+
|band|cellID |measure|
+----+------------------+-------+
|1 |599686042433355775|123.45 |
|1 |599686043374559743|98.12 |
|2 |599686042433355775|210.67 |
+----+------------------+-------+
(one row per band×cell. The lightweight LATERAL form yields these [band, cellID,
measure] rows directly; the heavyweight scalar form returns a nested ARRAY that
explode(...) flattens to the same rows.)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
# LATERAL UDTF returns [band, cellID, measure] columns directly
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridavg(tile, 4) t"
).take(5)
+----+------------------+-------+
|band|cellID |measure|
+----+------------------+-------+
|1 |599686042433355775|123.45 |
|1 |599686042433355776|124.20 |
+----+------------------+-------+
(one row per band × H3 cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
# Heavy tier returns ARRAY<ARRAY<struct(cellID, measure)>> per band
from pyspark.sql import functions as f
result = df.select(
rx.rst_h3_rastertogridavg("tile", f.lit(4)).alias("h3_grid")
).first()["h3_grid"]
[[{cellID: 599686042433355775, measure: 123.45}, {cellID: 599686042433355776, measure: 124.20}], ...]
(ARRAY<ARRAY<struct(cellID, measure)>>: one inner array per band, one struct per cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_h3_rastertogridavg(col("tile"), lit(4)).alias("h3_grid")).first()
result.getAs[Seq[Seq[Row]]]("h3_grid")
Vector(Vector(Row(cellID: 599686042433355775, measure: 123.45), ...), ...)
(Seq[Seq[Row]]: outer seq = bands, inner seq = cells per band)
rst_h3_rastertogridcount
LightweightHeavyweight Streaming UDTFPowered by rasterio + h3. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>, where measure is the per-cell pixel count (integer). The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_h3_rastertogridcount as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_h3_rastertogridcount(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridcount(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_h3_rastertogridcount(tile: Column, resolution: Column): Column — Pixel count per H3 cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_h3_rastertogridcount(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridcount(tile, 4) t;
+----+------------------+-------+
|band|cellID |measure|
+----+------------------+-------+
|1 |599686042433355775|256 |
|1 |599686043374559743|240 |
|2 |599686042433355775|256 |
+----+------------------+-------+
(pixel count per band×cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridcount(tile, 4) t"
).take(5)
+----+------------------------------+-------+
|band|cellID |measure|
+----+------------------------------+-------+
|1 |599686042433355775 |256 |
|1 |599686042433355776 |240 |
+----+------------------------------+-------+
(pixel count per band × H3 cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_h3_rastertogridcount("tile", f.lit(4)).alias("h3_grid")
).first()["h3_grid"]
[[{cellID: 599686042433355775, measure: 256}, ...], ...]
(pixel count per band × H3 cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_h3_rastertogridcount(col("tile"), lit(4)).alias("h3_grid")).first()
result.getAs[Seq[Seq[Row]]]("h3_grid")
Vector(Vector(Row(cellID: 599686042433355775, measure: 256), ...), ...)
(pixel count per band×cell)
rst_h3_rastertogridmax
LightweightHeavyweight Streaming UDTFPowered by rasterio + h3. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_h3_rastertogridmax as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_h3_rastertogridmax(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridmax(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_h3_rastertogridmax(tile: Column, resolution: Column): Column — Max value per H3 cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_h3_rastertogridmax(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridmax(tile, 4) t;
+----+------------------+-------+
|band|cellID |measure|
+----+------------------+-------+
|1 |599686042433355775|255.0 |
|1 |599686043374559743|254.0 |
|2 |599686042433355775|240.0 |
+----+------------------+-------+
(max value per band×cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridmax(tile, 4) t"
).take(5)
+----+------------------------------+-------+
|band|cellID |measure|
+----+------------------------------+-------+
|1 |599686042433355775 |255.0 |
|1 |599686042433355776 |254.0 |
+----+------------------------------+-------+
(max value per band × H3 cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_h3_rastertogridmax("tile", f.lit(4)).alias("h3_grid")
).first()["h3_grid"]
[[{cellID: 599686042433355775, measure: 255.0}, ...], ...]
(max value per band × H3 cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_h3_rastertogridmax(col("tile"), lit(4)).alias("h3_grid")).first()
result.getAs[Seq[Seq[Row]]]("h3_grid")
Vector(Vector(Row(cellID: 599686042433355775, measure: 255.0), ...), ...)
(max value per band×cell)
rst_h3_rastertogridmedian
LightweightHeavyweight Streaming UDTFPowered by rasterio + h3. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_h3_rastertogridmedian as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_h3_rastertogridmedian(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridmedian(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_h3_rastertogridmedian(tile: Column, resolution: Column): Column — Median value per H3 cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_h3_rastertogridmedian(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridmedian(tile, 4) t;
+----+------------------+-------+
|band|cellID |measure|
+----+------------------+-------+
|1 |599686042433355775|120.5 |
|1 |599686043374559743|122.0 |
|2 |599686042433355775|115.0 |
+----+------------------+-------+
(median value per band×cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridmedian(tile, 4) t"
).take(5)
+----+------------------------------+-------+
|band|cellID |measure|
+----+------------------------------+-------+
|1 |599686042433355775 |120.5 |
|1 |599686042433355776 |122.0 |
+----+------------------------------+-------+
(median value per band × H3 cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_h3_rastertogridmedian("tile", f.lit(4)).alias("h3_grid")
).first()["h3_grid"]
[[{cellID: 599686042433355775, measure: 120.5}, ...], ...]
(median value per band × H3 cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_h3_rastertogridmedian(col("tile"), lit(4)).alias("h3_grid")).first()
result.getAs[Seq[Seq[Row]]]("h3_grid")
Vector(Vector(Row(cellID: 599686042433355775, measure: 120.5), ...), ...)
(median value per band×cell)
rst_h3_rastertogridmin
LightweightHeavyweight Streaming UDTFPowered by rasterio + h3. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_h3_rastertogridmin as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_h3_rastertogridmin(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridmin(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_h3_rastertogridmin(tile: Column, resolution: Column): Column — Min value per H3 cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_h3_rastertogridmin(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridmin(tile, 4) t;
+----+------------------+-------+
|band|cellID |measure|
+----+------------------+-------+
|1 |599686042433355775|0.0 |
|1 |599686043374559743|10.0 |
|2 |599686042433355775|5.0 |
+----+------------------+-------+
(min value per band×cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridmin(tile, 4) t"
).take(5)
+----+------------------------------+-------+
|band|cellID |measure|
+----+------------------------------+-------+
|1 |599686042433355775 |0.0 |
|1 |599686042433355776 |10.0 |
+----+------------------------------+-------+
(min value per band × H3 cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_h3_rastertogridmin("tile", f.lit(4)).alias("h3_grid")
).first()["h3_grid"]
[[{cellID: 599686042433355775, measure: 0.0}, ...], ...]
(min value per band × H3 cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_h3_rastertogridmin(col("tile"), lit(4)).alias("h3_grid")).first()
result.getAs[Seq[Seq[Row]]]("h3_grid")
Vector(Vector(Row(cellID: 599686042433355775, measure: 0.0), ...), ...)
(min value per band×cell)
rst_h3_rastertogridstddev
LightweightHeavyweight Streaming UDTFPowered by rasterio + h3. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>, where measure is the population standard deviation (sqrt of the population variance) of the valid pixel values in each cell — a single-pixel cell yields 0.0. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_h3_rastertogridstddev as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_h3_rastertogridstddev(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridstddev(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_h3_rastertogridstddev(tile: Column, resolution: Column): Column — Population standard deviation of pixel values per H3 cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_h3_rastertogridstddev(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridstddev(tile, 4) t;
+----+------------------+-------+
|band|cellID |measure|
+----+------------------+-------+
|1 |599686042433355775|35.29 |
|1 |599686043374559743|37.27 |
|2 |599686042433355775|34.01 |
+----+------------------+-------+
(standard deviation per band×cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridstddev(tile, 4) t"
).take(5)
+----+------------------+-------+
|band|cellID |measure|
+----+------------------+-------+
|1 |599686042433355775|35.29 |
|1 |599686042433355776|37.27 |
+----+------------------+-------+
(standard deviation per band × H3 cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_h3_rastertogridstddev("tile", f.lit(4)).alias("h3_grid")
).first()["h3_grid"]
[[{cellID: 599686042433355775, measure: 35.29}, ...], ...]
(standard deviation per band × H3 cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_h3_rastertogridstddev(col("tile"), lit(4)).alias("h3_grid")).first()
result.getAs[Seq[Seq[Row]]]("h3_grid")
Vector(Vector(Row(cellID: 599686042433355775, measure: 35.29), ...), ...)
(standard deviation per band×cell)
rst_h3_rastertogridsum
LightweightHeavyweight Streaming UDTFPowered by rasterio + h3. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>, where measure is the total of the valid pixel values in each cell. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_h3_rastertogridsum as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_h3_rastertogridsum(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridsum(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_h3_rastertogridsum(tile: Column, resolution: Column): Column — Sum of pixel values per H3 cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_h3_rastertogridsum(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridsum(tile, 4) t;
+----+------------------+-------+
|band|cellID |measure|
+----+------------------+-------+
|1 |599686042433355775|31563.0|
|1 |599686043374559743|29488.0|
|2 |599686042433355775|28672.0|
+----+------------------+-------+
(sum of pixel values per band×cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridsum(tile, 4) t"
).take(5)
+----+------------------------------+--------+
|band|cellID |measure |
+----+------------------------------+--------+
|1 |599686042433355775 |31563.0 |
|1 |599686042433355776 |29488.0 |
+----+------------------------------+--------+
(sum of pixel values per band × H3 cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_h3_rastertogridsum("tile", f.lit(4)).alias("h3_grid")
).first()["h3_grid"]
[[{cellID: 599686042433355775, measure: 31563.0}, ...], ...]
(sum of pixel values per band × H3 cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_h3_rastertogridsum(col("tile"), lit(4)).alias("h3_grid")).first()
result.getAs[Seq[Seq[Row]]]("h3_grid")
Vector(Vector(Row(cellID: 599686042433355775, measure: 31563.0), ...), ...)
(sum of pixel values per band×cell)
rst_h3_rastertogridvariance
LightweightHeavyweight Streaming UDTFPowered by rasterio + h3. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>, where measure is the population variance (÷ n, two-pass) of the valid pixel values in each cell — a single-pixel cell yields 0.0. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_h3_rastertogridvariance as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_h3_rastertogridvariance(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridvariance(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_h3_rastertogridvariance(tile: Column, resolution: Column): Column — Population variance of pixel values per H3 cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_h3_rastertogridvariance(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridvariance(tile, 4) t;
+----+------------------+-------+
|band|cellID |measure|
+----+------------------+-------+
|1 |599686042433355775|1245.5 |
|1 |599686043374559743|1389.2 |
|2 |599686042433355775|1156.0 |
+----+------------------+-------+
(variance per band×cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_h3_rastertogridvariance(tile, 4) t"
).take(5)
+----+------------------------------+-------+
|band|cellID |measure|
+----+------------------------------+-------+
|1 |599686042433355775 |1245.5 |
|1 |599686042433355776 |1389.2 |
+----+------------------------------+-------+
(variance per band × H3 cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_h3_rastertogridvariance("tile", f.lit(4)).alias("h3_grid")
).first()["h3_grid"]
[[{cellID: 599686042433355775, measure: 1245.5}, ...], ...]
(variance per band × H3 cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_h3_rastertogridvariance(col("tile"), lit(4)).alias("h3_grid")).first()
result.getAs[Seq[Seq[Row]]]("h3_grid")
Vector(Vector(Row(cellID: 599686042433355775, measure: 1245.5), ...), ...)
(variance per band×cell)
Grid Functions (quadbin)
Aggregate raster values to CARTO quadbin v0 grid cells (8 reducers). Each reducer returns an array (one entry per band) of struct<cellID: BIGINT, measure: DOUBLE> rows; explode the array element you want to drive per-cell rows. Resolution is the quadbin zoom (0..26). See also rst_quadbin_tessellate (Generator Functions) and rst_quadbin_rasterize_agg (Aggregator Functions) for the remaining quadbin raster-grid surface.
rst_quadbin_rastertogridavg
LightweightHeavyweight Streaming UDTFPowered by rasterio + quadbin. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_quadbin_rastertogridavg as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_quadbin_rastertogridavg(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridavg(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_quadbin_rastertogridavg(tile: Column, resolution: Column): Column — Mean pixel value per quadbin cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_quadbin_rastertogridavg(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridavg(tile, 4) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |123.45 |
|1 |12346 |124.20 |
|2 |12345 |210.67 |
+----+------+-------+
(one row per band×Quadbin cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridavg(tile, 4) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |123.45 |
|1 |12346 |124.20 |
+----+------+-------+
(one row per band × Quadbin cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_quadbin_rastertogridavg("tile", f.lit(4)).alias("quadbin_grid")
).first()["quadbin_grid"]
[[{cellID: 12345, measure: 123.45}, ...], ...]
(ARRAY<ARRAY<struct(cellID, measure)>> per band × Quadbin cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_quadbin_rastertogridavg(col("tile"), lit(4)).alias("quadbin_grid")).first()
result.getAs[Seq[Seq[Row]]]("quadbin_grid")
Vector(Vector(Row(cellID: 12345, measure: 123.45), ...), ...)
(Seq[Seq[Row]]: outer seq = bands, inner seq = cells per band)
rst_quadbin_rastertogridcount
LightweightHeavyweight Streaming UDTFPowered by rasterio + quadbin. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>, where measure is the per-cell pixel count (integer). The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_quadbin_rastertogridcount as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_quadbin_rastertogridcount(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridcount(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_quadbin_rastertogridcount(tile: Column, resolution: Column): Column — Pixel count per quadbin cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_quadbin_rastertogridcount(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridcount(tile, 4) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |256 |
|1 |12346 |240 |
|2 |12345 |256 |
+----+------+-------+
(pixel count per band×Quadbin cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridcount(tile, 4) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |256 |
|1 |12346 |240 |
+----+------+-------+
(pixel count per band × Quadbin cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_quadbin_rastertogridcount("tile", f.lit(4)).alias("quadbin_grid")
).first()["quadbin_grid"]
[[{cellID: 12345, measure: 256}, ...], ...]
(pixel count per band × Quadbin cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_quadbin_rastertogridcount(col("tile"), lit(4)).alias("quadbin_grid")).first()
result.getAs[Seq[Seq[Row]]]("quadbin_grid")
Vector(Vector(Row(cellID: 12345, measure: 256), ...), ...)
(pixel count per band×cell)
rst_quadbin_rastertogridmax
LightweightHeavyweight Streaming UDTFPowered by rasterio + quadbin. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_quadbin_rastertogridmax as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_quadbin_rastertogridmax(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridmax(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_quadbin_rastertogridmax(tile: Column, resolution: Column): Column — Max pixel value per quadbin cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_quadbin_rastertogridmax(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridmax(tile, 4) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |255.0 |
|1 |12346 |254.0 |
|2 |12345 |240.0 |
+----+------+-------+
(max value per band×Quadbin cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridmax(tile, 4) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |255.0 |
|1 |12346 |254.0 |
+----+------+-------+
(max value per band × Quadbin cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_quadbin_rastertogridmax("tile", f.lit(4)).alias("quadbin_grid")
).first()["quadbin_grid"]
[[{cellID: 12345, measure: 255.0}, ...], ...]
(max value per band × Quadbin cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_quadbin_rastertogridmax(col("tile"), lit(4)).alias("quadbin_grid")).first()
result.getAs[Seq[Seq[Row]]]("quadbin_grid")
Vector(Vector(Row(cellID: 12345, measure: 255.0), ...), ...)
(max value per band×cell)
rst_quadbin_rastertogridmedian
LightweightHeavyweight Streaming UDTFPowered by rasterio + quadbin. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_quadbin_rastertogridmedian as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_quadbin_rastertogridmedian(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridmedian(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_quadbin_rastertogridmedian(tile: Column, resolution: Column): Column — Median pixel value per quadbin cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_quadbin_rastertogridmedian(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridmedian(tile, 4) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |120.5 |
|1 |12346 |122.0 |
|2 |12345 |115.0 |
+----+------+-------+
(median value per band×Quadbin cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridmedian(tile, 4) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |120.5 |
|1 |12346 |122.0 |
+----+------+-------+
(median value per band × Quadbin cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_quadbin_rastertogridmedian("tile", f.lit(4)).alias("quadbin_grid")
).first()["quadbin_grid"]
[[{cellID: 12345, measure: 120.5}, ...], ...]
(median value per band × Quadbin cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_quadbin_rastertogridmedian(col("tile"), lit(4)).alias("quadbin_grid")).first()
result.getAs[Seq[Seq[Row]]]("quadbin_grid")
Vector(Vector(Row(cellID: 12345, measure: 120.5), ...), ...)
(median value per band×cell)
rst_quadbin_rastertogridmin
LightweightHeavyweight Streaming UDTFPowered by rasterio + quadbin. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_quadbin_rastertogridmin as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_quadbin_rastertogridmin(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridmin(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_quadbin_rastertogridmin(tile: Column, resolution: Column): Column — Min pixel value per quadbin cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_quadbin_rastertogridmin(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridmin(tile, 4) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |0.0 |
|1 |12346 |10.0 |
|2 |12345 |5.0 |
+----+------+-------+
(min value per band×Quadbin cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridmin(tile, 4) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |0.0 |
|1 |12346 |10.0 |
+----+------+-------+
(min value per band × Quadbin cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_quadbin_rastertogridmin("tile", f.lit(4)).alias("quadbin_grid")
).first()["quadbin_grid"]
[[{cellID: 12345, measure: 0.0}, ...], ...]
(min value per band × Quadbin cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_quadbin_rastertogridmin(col("tile"), lit(4)).alias("quadbin_grid")).first()
result.getAs[Seq[Seq[Row]]]("quadbin_grid")
Vector(Vector(Row(cellID: 12345, measure: 0.0), ...), ...)
(min value per band×cell)
rst_quadbin_rastertogridstddev
LightweightHeavyweight Streaming UDTFPowered by rasterio + quadbin. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>, where measure is the population standard deviation (sqrt of the population variance) of the valid pixel values in each cell — a single-pixel cell yields 0.0. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_quadbin_rastertogridstddev as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_quadbin_rastertogridstddev(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridstddev(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_quadbin_rastertogridstddev(tile: Column, resolution: Column): Column — Population standard deviation of pixel values per quadbin cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_quadbin_rastertogridstddev(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridstddev(tile, 4) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |35.29 |
|1 |12346 |37.27 |
|2 |12345 |34.01 |
+----+------+-------+
(standard deviation per band×Quadbin cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridstddev(tile, 4) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |35.29 |
|1 |12346 |37.27 |
+----+------+-------+
(standard deviation per band × Quadbin cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_quadbin_rastertogridstddev("tile", f.lit(4)).alias("quadbin_grid")
).first()["quadbin_grid"]
[[{cellID: 12345, measure: 35.29}, ...], ...]
(standard deviation per band × Quadbin cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_quadbin_rastertogridstddev(col("tile"), lit(4)).alias("quadbin_grid")).first()
result.getAs[Seq[Seq[Row]]]("quadbin_grid")
Vector(Vector(Row(cellID: 12345, measure: 35.29), ...), ...)
(standard deviation per band×cell)
rst_quadbin_rastertogridsum
LightweightHeavyweight Streaming UDTFPowered by rasterio + quadbin. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>, where measure is the total of the valid pixel values in each cell. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_quadbin_rastertogridsum as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_quadbin_rastertogridsum(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridsum(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_quadbin_rastertogridsum(tile: Column, resolution: Column): Column — Sum of pixel values per quadbin cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_quadbin_rastertogridsum(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridsum(tile, 4) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |31563.0|
|1 |12346 |29488.0|
|2 |12345 |28672.0|
+----+------+-------+
(sum of pixel values per band×Quadbin cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridsum(tile, 4) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |31563.0|
|1 |12346 |29488.0|
+----+------+-------+
(sum of pixel values per band × Quadbin cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_quadbin_rastertogridsum("tile", f.lit(4)).alias("quadbin_grid")
).first()["quadbin_grid"]
[[{cellID: 12345, measure: 31563.0}, ...], ...]
(sum of pixel values per band × Quadbin cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_quadbin_rastertogridsum(col("tile"), lit(4)).alias("quadbin_grid")).first()
result.getAs[Seq[Seq[Row]]]("quadbin_grid")
Vector(Vector(Row(cellID: 12345, measure: 31563.0), ...), ...)
(sum of pixel values per band×cell)
rst_quadbin_rastertogridvariance
LightweightHeavyweight Streaming UDTFPowered by rasterio + quadbin. Returns an ARRAY (one element per band) of ARRAY<struct(cellID, measure)>, where measure is the population variance (÷ n, two-pass) of the valid pixel values in each cell — a single-pixel cell yields 0.0. The raster is interpreted as EPSG:4326 lon/lat — reproject upstream with rst_transform if your source CRS differs.
Heavyweight registers gbx_rst_quadbin_rastertogridvariance as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_quadbin_rastertogridvariance(tile, 4) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridvariance(tile, 4) t. Both forms appear in the SQL tab below.
Signature: rst_quadbin_rastertogridvariance(tile: Column, resolution: Column): Column — Population variance of pixel values per quadbin cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_quadbin_rastertogridvariance(tile, 4) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridvariance(tile, 4) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |1245.5 |
|1 |12346 |1389.2 |
|2 |12345 |1156.0 |
+----+------+-------+
(variance per band×Quadbin cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
return spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_quadbin_rastertogridvariance(tile, 4) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |12345 |1245.5 |
|1 |12346 |1389.2 |
+----+------+-------+
(variance per band × Quadbin cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_quadbin_rastertogridvariance("tile", f.lit(4)).alias("quadbin_grid")
).first()["quadbin_grid"]
[[{cellID: 12345, measure: 1245.5}, ...], ...]
(variance per band × Quadbin cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_quadbin_rastertogridvariance(col("tile"), lit(4)).alias("quadbin_grid")).first()
result.getAs[Seq[Seq[Row]]]("quadbin_grid")
Vector(Vector(Row(cellID: 12345, measure: 1245.5), ...), ...)
(variance per band×cell)
Grid Functions (BNG)
Aggregate raster values to British National Grid (BNG) cells (8 total). BNG functions automatically reproject the input raster to EPSG:27700 before sampling — no rst_transform needed upstream. Resolution accepts integer indices ±1..±6 (1 = 100 km, 2 = 10 km, 3 = 1 km, 4 = 100 m, 5 = 10 m, 6 = 1 m; negative indices address quadrant sub-cells) or string keys (e.g. "1km", "100m"). Cell IDs are STRING.
The BNG raster-grid reducers are available in both tiers. The lightweight implementation is backed by pygx._bng cell math and rasterio: it auto-warps the input to EPSG:27700 via rasterio.warp, samples pixels per cell, and renders STRING BNG cell IDs — matching the heavyweight cell set and measures exactly.
A cell that covers only NoData pixels returns NULL for the measure (not 0 or NaN) — the same behavior as the H3 and quadbin reducers. NULL is safe to filter (WHERE measure IS NULL) and is excluded by SQL aggregates like MAX/MIN/AVG.
rst_bng_rastertogridavg
LightweightHeavyweight Streaming UDTFMean pixel value per BNG cell. The raster is reprojected to EPSG:27700 internally before sampling.
Heavyweight registers gbx_rst_bng_rastertogridavg as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_bng_rastertogridavg(tile, 3) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridavg(tile, 3) t. Both forms appear in the SQL tab below.
Signature: rst_bng_rastertogridavg(tile: Column, resolution: Column): Column — Mean pixel value per BNG cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_bng_rastertogridavg(tile, 3) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridavg(tile, 3) t;
+----+------+------------------+
|band|cellID|measure |
+----+------+------------------+
|1 |OW5574|77.22222222222223 |
|1 |OW5575|80.66666666666667 |
|2 |OW5574|144.33333333333334|
+----+------+------------------+
(one row per band × BNG cell; cellID is a STRING grid-square label)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
result = spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridavg(tile, 3) t"
).take(5)
+----+------+------------------+
|band|cellID|measure |
+----+------+------------------+
|1 |OW5574|77.22222222222223 |
|1 |OW5575|80.66666666666667 |
|2 |OW5574|144.33333333333334|
+----+------+------------------+
(one row per band × BNG cell; cellID is a STRING grid-square label)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_bng_rastertogridavg("tile", f.lit(3)).alias("bng_grid")
).first()["bng_grid"]
[[Row(cellID='OW5575', measure=...), Row(cellID='OW5574', measure=...), ...], # band 1
[...], # band 2
[...]] # band 3
(ARRAY<ARRAY<struct(cellID STRING, measure)>> — outer per band, inner per BNG cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_bng_rastertogridavg(col("tile"), lit(3)).alias("bng_grid")).first()
result.getAs[Seq[Seq[Row]]]("bng_grid")
Vector(Vector(Row(OW5575, ...), Row(OW5574, ...), ...), Vector(...), Vector(...))
(Seq[Seq[Row]] — outer per band, inner per BNG cell; cellID is a STRING grid-square label)
rst_bng_rastertogridcount
LightweightHeavyweight Streaming UDTFValid (non-NoData) pixel count per BNG cell.
Heavyweight registers gbx_rst_bng_rastertogridcount as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_bng_rastertogridcount(tile, 3) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridcount(tile, 3) t. Both forms appear in the SQL tab below.
Signature: rst_bng_rastertogridcount(tile: Column, resolution: Column): Column — Pixel count per BNG cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_bng_rastertogridcount(tile, 3) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridcount(tile, 3) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |OW5574|9 |
|1 |OW5575|21 |
|2 |OW5574|9 |
+----+------+-------+
(pixel count per band × BNG cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
result = spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridcount(tile, 3) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |OW5574|9 |
|1 |OW5575|21 |
|2 |OW5574|9 |
+----+------+-------+
(pixel count per band × BNG cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_bng_rastertogridcount("tile", f.lit(3)).alias("bng_grid")
).first()["bng_grid"]
[[Row(cellID='OW5575', measure=...), Row(cellID='OW5574', measure=...), ...], # band 1
[...], # band 2
[...]] # band 3
(ARRAY<ARRAY<struct(cellID STRING, measure)>> — outer per band, inner per BNG cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_bng_rastertogridcount(col("tile"), lit(3)).alias("bng_grid")).first()
result.getAs[Seq[Seq[Row]]]("bng_grid")
Vector(Vector(Row(OW5575, ...), Row(OW5574, ...), ...), Vector(...), Vector(...))
(Seq[Seq[Row]] — outer per band, inner per BNG cell; cellID is a STRING grid-square label)
rst_bng_rastertogridmax
LightweightHeavyweight Streaming UDTFMaximum pixel value per BNG cell.
Heavyweight registers gbx_rst_bng_rastertogridmax as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_bng_rastertogridmax(tile, 3) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridmax(tile, 3) t. Both forms appear in the SQL tab below.
Signature: rst_bng_rastertogridmax(tile: Column, resolution: Column): Column — Max pixel value per BNG cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_bng_rastertogridmax(tile, 3) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridmax(tile, 3) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |OW5574|106.0 |
|1 |OW5575|118.0 |
|1 |OW5674|107.0 |
+----+------+-------+
(max value per band × BNG cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
result = spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridmax(tile, 3) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |OW5574|106.0 |
|1 |OW5575|118.0 |
|1 |OW5674|107.0 |
+----+------+-------+
(max value per band × BNG cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_bng_rastertogridmax("tile", f.lit(3)).alias("bng_grid")
).first()["bng_grid"]
[[Row(cellID='OW5575', measure=...), Row(cellID='OW5574', measure=...), ...], # band 1
[...], # band 2
[...]] # band 3
(ARRAY<ARRAY<struct(cellID STRING, measure)>> — outer per band, inner per BNG cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_bng_rastertogridmax(col("tile"), lit(3)).alias("bng_grid")).first()
result.getAs[Seq[Seq[Row]]]("bng_grid")
Vector(Vector(Row(OW5575, ...), Row(OW5574, ...), ...), Vector(...), Vector(...))
(Seq[Seq[Row]] — outer per band, inner per BNG cell; cellID is a STRING grid-square label)
rst_bng_rastertogridmedian
LightweightHeavyweight Streaming UDTFMedian pixel value per BNG cell.
Heavyweight registers gbx_rst_bng_rastertogridmedian as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_bng_rastertogridmedian(tile, 3) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridmedian(tile, 3) t. Both forms appear in the SQL tab below.
Signature: rst_bng_rastertogridmedian(tile: Column, resolution: Column): Column — Median pixel value per BNG cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_bng_rastertogridmedian(tile, 3) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridmedian(tile, 3) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |OW5574|88.0 |
|1 |OW5575|80.0 |
|1 |OW5674|81.0 |
+----+------+-------+
(median value per band × BNG cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
result = spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridmedian(tile, 3) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |OW5574|88.0 |
|1 |OW5575|80.0 |
|1 |OW5674|81.0 |
+----+------+-------+
(median value per band × BNG cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_bng_rastertogridmedian("tile", f.lit(3)).alias("bng_grid")
).first()["bng_grid"]
[[Row(cellID='OW5575', measure=...), Row(cellID='OW5574', measure=...), ...], # band 1
[...], # band 2
[...]] # band 3
(ARRAY<ARRAY<struct(cellID STRING, measure)>> — outer per band, inner per BNG cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_bng_rastertogridmedian(col("tile"), lit(3)).alias("bng_grid")).first()
result.getAs[Seq[Seq[Row]]]("bng_grid")
Vector(Vector(Row(OW5575, ...), Row(OW5574, ...), ...), Vector(...), Vector(...))
(Seq[Seq[Row]] — outer per band, inner per BNG cell; cellID is a STRING grid-square label)
rst_bng_rastertogridmin
LightweightHeavyweight Streaming UDTFMinimum pixel value per BNG cell.
Heavyweight registers gbx_rst_bng_rastertogridmin as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_bng_rastertogridmin(tile, 3) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridmin(tile, 3) t. Both forms appear in the SQL tab below.
Signature: rst_bng_rastertogridmin(tile: Column, resolution: Column): Column — Min pixel value per BNG cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_bng_rastertogridmin(tile, 3) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridmin(tile, 3) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |OW5574|0.0 |
|1 |OW5575|54.0 |
|1 |OW5674|65.0 |
+----+------+-------+
(min value per band × BNG cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
result = spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridmin(tile, 3) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |OW5574|0.0 |
|1 |OW5575|54.0 |
|1 |OW5674|65.0 |
+----+------+-------+
(min value per band × BNG cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_bng_rastertogridmin("tile", f.lit(3)).alias("bng_grid")
).first()["bng_grid"]
[[Row(cellID='OW5575', measure=...), Row(cellID='OW5574', measure=...), ...], # band 1
[...], # band 2
[...]] # band 3
(ARRAY<ARRAY<struct(cellID STRING, measure)>> — outer per band, inner per BNG cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_bng_rastertogridmin(col("tile"), lit(3)).alias("bng_grid")).first()
result.getAs[Seq[Seq[Row]]]("bng_grid")
Vector(Vector(Row(OW5575, ...), Row(OW5574, ...), ...), Vector(...), Vector(...))
(Seq[Seq[Row]] — outer per band, inner per BNG cell; cellID is a STRING grid-square label)
rst_bng_rastertogridstddev
LightweightHeavyweight Streaming UDTFPopulation standard deviation (sqrt of the population variance) of pixel values per BNG cell — a single-pixel cell yields 0.0.
Heavyweight registers gbx_rst_bng_rastertogridstddev as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_bng_rastertogridstddev(tile, 3) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridstddev(tile, 3) t. Both forms appear in the SQL tab below.
Signature: rst_bng_rastertogridstddev(tile: Column, resolution: Column): Column — Population standard deviation of pixel values per BNG cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_bng_rastertogridstddev(tile, 3) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridstddev(tile, 3) t;
+----+------+------------------+
|band|cellID|measure |
+----+------+------------------+
|1 |OW5574|31.043975181373415|
|1 |OW5575|21.543606571950388|
|1 |OW5674|14.023789311975086|
+----+------+------------------+
(population standard deviation per band × BNG cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
result = spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridstddev(tile, 3) t"
).take(5)
+----+------+------------------+
|band|cellID|measure |
+----+------+------------------+
|1 |OW5574|31.043975181373415|
|1 |OW5575|21.543606571950388|
|1 |OW5674|14.023789311975086|
+----+------+------------------+
(population standard deviation per band × BNG cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_bng_rastertogridstddev("tile", f.lit(3)).alias("bng_grid")
).first()["bng_grid"]
[[Row(cellID='OW5575', measure=...), Row(cellID='OW5574', measure=...), ...], # band 1
[...], # band 2
[...]] # band 3
(ARRAY<ARRAY<struct(cellID STRING, measure)>> — outer per band, inner per BNG cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_bng_rastertogridstddev(col("tile"), lit(3)).alias("bng_grid")).first()
result.getAs[Seq[Seq[Row]]]("bng_grid")
Vector(Vector(Row(OW5575, ...), Row(OW5574, ...), ...), Vector(...), Vector(...))
(Seq[Seq[Row]] — outer per band, inner per BNG cell; cellID is a STRING grid-square label)
rst_bng_rastertogridsum
LightweightHeavyweight Streaming UDTFSum of pixel values per BNG cell.
Heavyweight registers gbx_rst_bng_rastertogridsum as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_bng_rastertogridsum(tile, 3) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridsum(tile, 3) t. Both forms appear in the SQL tab below.
Signature: rst_bng_rastertogridsum(tile: Column, resolution: Column): Column — Sum of pixel values per BNG cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_bng_rastertogridsum(tile, 3) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridsum(tile, 3) t;
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |OW5574|695.0 |
|1 |OW5575|1694.0 |
|1 |OW5674|774.0 |
+----+------+-------+
(sum of pixel values per band × BNG cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
result = spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridsum(tile, 3) t"
).take(5)
+----+------+-------+
|band|cellID|measure|
+----+------+-------+
|1 |OW5574|695.0 |
|1 |OW5575|1694.0 |
|1 |OW5674|774.0 |
+----+------+-------+
(sum of pixel values per band × BNG cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_bng_rastertogridsum("tile", f.lit(3)).alias("bng_grid")
).first()["bng_grid"]
[[Row(cellID='OW5575', measure=...), Row(cellID='OW5574', measure=...), ...], # band 1
[...], # band 2
[...]] # band 3
(ARRAY<ARRAY<struct(cellID STRING, measure)>> — outer per band, inner per BNG cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_bng_rastertogridsum(col("tile"), lit(3)).alias("bng_grid")).first()
result.getAs[Seq[Seq[Row]]]("bng_grid")
Vector(Vector(Row(OW5575, ...), Row(OW5574, ...), ...), Vector(...), Vector(...))
(Seq[Seq[Row]] — outer per band, inner per BNG cell; cellID is a STRING grid-square label)
rst_bng_rastertogridvariance
LightweightHeavyweight Streaming UDTFPopulation variance (÷ n, two-pass) of pixel values per BNG cell — a single-pixel cell yields 0.0.
Heavyweight registers gbx_rst_bng_rastertogridvariance as a scalar (ARRAY-returning) function — call it directly and explode(...) to flatten to rows: SELECT gbx_rst_bng_rastertogridvariance(tile, 3) AS grid FROM multiband_rasters. The lightweight (pyrx) tier registers it as a streaming table function, so lightweight SQL must use LATERAL: SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridvariance(tile, 3) t. Both forms appear in the SQL tab below.
Signature: rst_bng_rastertogridvariance(tile: Column, resolution: Column): Column — Population variance of pixel values per BNG cell.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return (one element per band) — call directly;
-- explode to flatten the per-band arrays to rows.
SELECT gbx_rst_bng_rastertogridvariance(tile, 3) AS grid FROM multiband_rasters;
-- Lightweight (pyrx): streaming table function — must use LATERAL.
SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridvariance(tile, 3) t;
+----+------+------------------+
|band|cellID|measure |
+----+------+------------------+
|1 |OW5574|963.7283950617285 |
|1 |OW5575|464.126984126984 |
|1 |OW5674|196.66666666666666|
+----+------+------------------+
(population variance per band × BNG cell)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = _get_multiband_df(spark)
df.createOrReplaceTempView("multiband_rasters")
result = spark.sql(
"SELECT t.* FROM multiband_rasters, LATERAL gbx_rst_bng_rastertogridvariance(tile, 3) t"
).take(5)
+----+------+------------------+
|band|cellID|measure |
+----+------+------------------+
|1 |OW5574|963.7283950617285 |
|1 |OW5575|464.126984126984 |
|1 |OW5674|196.66666666666666|
+----+------+------------------+
(population variance per band × BNG cell)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("multiband_rasters")
from pyspark.sql import functions as f
result = df.select(
rx.rst_bng_rastertogridvariance("tile", f.lit(3)).alias("bng_grid")
).first()["bng_grid"]
[[Row(cellID='OW5575', measure=...), Row(cellID='OW5574', measure=...), ...], # band 1
[...], # band 2
[...]] # band 3
(ARRAY<ARRAY<struct(cellID STRING, measure)>> — outer per band, inner per BNG cell)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("multiband_rasters")
val result = raster.select(rx.rst_bng_rastertogridvariance(col("tile"), lit(3)).alias("bng_grid")).first()
result.getAs[Seq[Seq[Row]]]("bng_grid")
Vector(Vector(Row(OW5575, ...), Row(OW5574, ...), ...), Vector(...), Vector(...))
(Seq[Seq[Row]] — outer per band, inner per BNG cell; cellID is a STRING grid-square label)
Operations
Transform and analyze rasters (20 total).
rst_asformat
LightweightHeavyweightPowered by rasterio. Output formats are limited to rasterio's bundled-GDAL writable driver set; the tile is re-encoded in the requested format.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_asformat(tile: Column, newFormat: Column): Column — Convert to another format.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Convert NetCDF to GeoTIFF
SELECT
path,
gbx_rst_asformat(tile, 'GTiff') as geotiff_tile
FROM netcdf_rasters;
-- Convert to PNG
SELECT
path,
gbx_rst_asformat(tile, 'PNG') as png_tile
FROM visualization_tiles;
+----+-----------------------------------------------------------+
|path|geotiff_tile |
+----+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+----+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(rx.rst_asformat("tile", f.lit("GTiff")).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(re-encoded tile in the requested GDAL format; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(
rx.rst_asformat("tile", f.lit("GTiff")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(re-encoded tile in the requested GDAL format)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_asformat(col("tile"), lit("GTiff")).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(re-encoded tile in the requested GDAL format)
rst_clip
LightweightHeavyweightPowered by rasterio (rasterio.mask). The clip geometry is assumed to be in the raster's CRS; the heavyweight tier has additional SRID-inheritance fallbacks.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_clip(tile: Column, clip: Column, cutlineAllTouched: Column, [clip_crs: Column = null]): Column — Clip by geometry. The clip argument must be WKT (string), EWKT (SRID-prefixed string), WKB (binary), or EWKB (SRID-embedded binary); do not use st_geomfromtext() or other DBR native geometry. Optional clip_crs (string, source role) declares the CRS of a plain WKB/WKT cutline — an EWKB/EWKT embedded SRID wins, absent → assumed already in the raster CRS. See Coordinate Reference Systems.
CRS handling:
- EWKT (
SRID=4326;POLYGON(...)) or EWKB (SRID encoded in the byte header) — the SRID is read, and if it differs from the raster's CRS the cutline is reprojected before clipping. Use this form whenever the geometry and raster may be in different CRSs. - Plain WKT / WKB (no SRID) — the geometry is assumed to already be in the raster's CRS. If that assumption is wrong (for example, lon/lat polygons against a UTM raster), the cutline will land outside the raster and you'll get an empty or blank output. Either switch to EWKT/EWKB, or reproject the geometry to the raster's CRS first.
Example clip geometry: a WKT polygon in the raster's native CRS (EPSG:32618, no SRID prefix). To clip with a WGS84 geometry use EWKT: SRID=4326;POLYGON(...) — the embedded SRID triggers auto-reprojection to the raster's CRS.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Clip with WKT geometry
SELECT
path,
gbx_rst_clip(
tile,
'POLYGON((-122 37, -122 38, -121 38, -121 37, -122 37))',
true
) as clipped
FROM rasters;
+----+-----------------------------------------------------------+
|path|clipped |
+----+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+----+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
# WKT polygon in raster's native CRS (upper-left quadrant of the tile extent)
clip_geom = "POLYGON((2121950 -10791280, 2123140 -10791280, 2123140 -10790470, 2121950 -10790470, 2121950 -10791280))"
result = df.select(
rx.rst_clip("tile", f.lit(clip_geom), f.lit(True)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(clipped tile; polygon is in the raster's native CRS (no SRID = no reprojection); light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
clip_geom = "POLYGON((2121950 -10791280, 2123140 -10791280, 2123140 -10790470, 2121950 -10790470, 2121950 -10791280))"
result = tile_df.select(
rx.rst_clip("tile", f.lit(clip_geom), f.lit(True)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(clipped tile; polygon is in the raster's native CRS (no SRID = no reprojection))
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// WKT polygon in raster's native CRS (EPSG:32618); no SRID prefix = no reprojection.
// To clip with a WGS84 geometry use EWKT: "SRID=4326;POLYGON(...)".
val rasters = spark.table("rasters")
val clipGeom = "POLYGON((2121950 -10791280, 2123140 -10791280, 2123140 -10790470, 2121950 -10790470, 2121950 -10791280))"
val result = rasters.select(rx.rst_clip(col("tile"), lit(clipGeom), lit(true)).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(clipped tile; polygon is in the raster's native CRS (no SRID = no reprojection))
rst_combineavg
LightweightHeavyweightPowered by rasterio + NumPy. Takes an ARRAY<tile> and returns the NoData-aware per-pixel mean; input tiles must share the same grid (shape/extent/CRS). cellid is preserved when all inputs share one, else −1.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_combineavg(tiles: Column): Column — Average multiple tiles (e.g. temporal composite).
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_combineavg(array(tile)) AS combined FROM multiband_rasters;
+-----------------------------------------------------------+
|combined |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from _fixtures import multi_band_tiles_df
df = multi_band_tiles_df(spark)
# Aggregate the 3 per-band tiles into one array, then average per-pixel.
result = (
df.groupBy("region")
.agg(rx.rst_combineavg(f.collect_list("tile")).alias("tile"))
.first()
)
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(averaged combined raster from 3 input tiles)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
from ._fixtures import multi_band_tiles_df_heavy
df = multi_band_tiles_df_heavy(spark)
result = (
df.groupBy("region")
.agg(rx.rst_combineavg(f.collect_list("tile")).alias("tile"))
.first()
)
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(averaged combined raster from 3 input tiles)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
// Multi-row fixture: 3 single-band tiles (one per band from multiband GeoTIFF)
val schema = StructType(Seq(StructField("tile", BinaryType()), StructField("band_index", IntegerType()), StructField("region", StringType())))
// Load multiband, extract 3 bands, collect into array, then average
val multiband = spark.table("multiband_rasters")
val b1 = multiband.select(rx.rst_band(col("tile"), lit(1)).alias("tile")).withColumn("band_index", lit(1))
val b2 = multiband.select(rx.rst_band(col("tile"), lit(2)).alias("tile")).withColumn("band_index", lit(2))
val b3 = multiband.select(rx.rst_band(col("tile"), lit(3)).alias("tile")).withColumn("band_index", lit(3))
val bands = b1.union(b2).union(b3).withColumn("region", lit("R1"))
val result = bands.groupBy("region").agg(rx.rst_combineavg(collect_list("tile")).alias("combined"))
result.show(truncate = false)
+------+-----------------------------------------------------------+
|region|combined |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(averaged combined raster)
rst_convolve
LightweightHeavyweightPowered by SciPy (scipy.ndimage). Output is Float64 and border pixels are filled by edge-replication; the heavyweight tier preserves the input dtype and leaves border pixels unchanged.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_convolve(tile: Column, kernel: Column): Column — Apply convolution kernel.
Example kernel: 3×3 identity ([[0,0,0],[0,1,0],[0,0,0]]), which passes pixels through unchanged.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Apply 3x3 kernel (e.g. blur); kernel format is driver-specific
SELECT path, gbx_rst_convolve(tile, kernel) as filtered FROM rasters_with_kernels;
+----+-----------------------------------------------------------+
|path|filtered |
+----+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+----+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
kernel = f.array(
f.array(f.lit(0.0), f.lit(0.0), f.lit(0.0)),
f.array(f.lit(0.0), f.lit(1.0), f.lit(0.0)),
f.array(f.lit(0.0), f.lit(0.0), f.lit(0.0)),
)
result = df.select(rx.rst_convolve("tile", kernel).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(convolved tile; kernel is a 3x3 identity; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
kernel = f.array(
f.array(f.lit(0.0), f.lit(0.0), f.lit(0.0)),
f.array(f.lit(0.0), f.lit(1.0), f.lit(0.0)),
f.array(f.lit(0.0), f.lit(0.0), f.lit(0.0)),
)
result = tile_df.select(rx.rst_convolve("tile", kernel).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(convolved tile; kernel is a 3x3 identity)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val kernel = array(
array(lit(0.0), lit(0.0), lit(0.0)),
array(lit(0.0), lit(1.0), lit(0.0)),
array(lit(0.0), lit(0.0), lit(0.0))
)
val result = rasters.select(rx.rst_convolve(col("tile"), kernel).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(convolved tile; kernel is a 3x3 identity)
rst_derivedband
LightweightHeavyweightPowered by rasterio with GDAL VRT Python pixel functions. A pixel function authored for one tier runs unchanged in the other.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_derivedband(tile: Column, pyfunc: String, funcName: String): Column — Apply Python UDF to derive band.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_derivedband(tile, 'def double(in_ar, out_ar, xoff, yoff, xsize, ysize, raster_xsize, raster_ysize, buf_radius, gt, **kwargs):\\n out_ar[:] = in_ar[0] * 2\\n', 'double') AS result FROM multiband_rasters;
+-----------------------------------------------------------+
|result |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
# GDAL VRT pixel-function that doubles band 1's pixel values in place.
python_func = (
"def double(in_ar, out_ar, xoff, yoff, xsize, ysize, "
"raster_xsize, raster_ysize, buf_radius, gt, **kwargs):\n"
" out_ar[:] = in_ar[0] * 2\n"
)
result = df.select(
rx.rst_derivedband("tile", f.lit(python_func), f.lit("double")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(raster with derived band from Python UDF)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
python_func = (
"def double(in_ar, out_ar, xoff, yoff, xsize, ysize, "
"raster_xsize, raster_ysize, buf_radius, gt, **kwargs):\n"
" out_ar[:] = in_ar[0] * 2\n"
)
result = df.select(
rx.rst_derivedband("tile", f.lit(python_func), f.lit("double")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(raster with derived band from Python UDF)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Apply a GDAL VRT Python pixel-function that doubles band 1
val rasters = spark.table("multiband_rasters")
val pythonFunc = "def double(in_ar, out_ar, xoff, yoff, xsize, ysize, raster_xsize, raster_ysize, buf_radius, gt, **kwargs):\n out_ar[:] = in_ar[0] * 2\n"
val result = rasters.select(rx.rst_derivedband(col("tile"), lit(pythonFunc), lit("double")).alias("derived"))
result.show(truncate = false)
+-----------------------------------------------------------+
|derived |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(derived band raster from Python UDF)
rst_filter
LightweightHeavyweightPowered by SciPy (scipy.ndimage). The averaging filter is named 'mean' (not 'avg') and 'mode' is unavailable; the averaging output is Float32 and near-edge values may differ slightly from the heavyweight tier.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_filter(tile: Column, kernelSize: Column, operation: Column): Column — Spatial filter (e.g. median, avg).
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Median filter (3x3 window)
SELECT
path,
gbx_rst_filter(tile, 3, 'median') as denoised
FROM noisy_rasters;
-- Average smoothing (5x5 window)
SELECT
path,
gbx_rst_filter(tile, 5, 'avg') as smoothed
FROM rasters;
+----+-----------------------------------------------------------+
|path|denoised |
+----+-----------------------------------------------------------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+----+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(
rx.rst_filter("tile", f.lit(3), f.lit("median")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(filtered tile; 3x3 median filter applied; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(
rx.rst_filter("tile", f.lit(3), f.lit("median")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(filtered tile; 3x3 median filter applied)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_filter(col("tile"), lit(3), lit("median")).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(filtered tile; 3x3 median filter applied)
rst_initnodata
LightweightHeavyweightPowered by rasterio. When no NoData is set it assigns -9999.0; the heavyweight tier assigns a data-type-appropriate sentinel per band, so the NoData value can differ for integer or byte rasters.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_initnodata(tile: Column): Column — Initialize NoData values.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_initnodata(tile) as tile FROM rasters;
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(tile with NoData initialized)
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_initnodata("tile").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(tile with NoData initialized; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_initnodata("tile").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(tile with NoData initialized)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_initnodata(col("tile")).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(tile with NoData initialized)
rst_isempty
LightweightHeavyweightPowered by rasterio.
Signature: rst_isempty(tile: Column): Column — Check if raster is empty.
Returns true when the raster has no size or every band is entirely NoData.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands with real pixel data); the canonical single-band sentinel2 tile has NoData=0 with all pixels equal zero, returning true.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_isempty(tile) AS is_empty FROM multiband_rasters;
+--------+
|is_empty|
+--------+
|false |
+--------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_isempty("tile").alias("is_empty")).first()
+--------+
|is_empty|
+--------+
|false |
+--------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_isempty("tile").alias("is_empty")).first()
+--------+
|is_empty|
+--------+
|false |
+--------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif) which carries real pixel data.
// The single-band sentinel2 tile has NoData=0 with all pixels equal zero (isempty=true).
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_isempty(col("tile")).alias("is_empty"))
result.show()
+--------+
|is_empty|
+--------+
|false |
+--------+
rst_mapalgebra
LightweightHeavyweightPowered by NumExpr. Bands map to A, B, C, … and the calc expression is evaluated with NumExpr (no gdal_calc NumPy builtins); single-band Float32 output.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_mapalgebra(tiles: Column, expression: Column): Column — Map algebra expression (e.g. A - B).
How the inputs bind. tiles is an ARRAY<tile>. By default, band 1 of each tile, in array order, binds to the variables A, B, C, … (A = first tile, B = second, …). The result is a single-band Float32 tile on the first input's georeference.
Spec format (same on both tiers). The expression is accepted in either of two forms — identical across SQL, Scala, and both Python tiers:
- JSON envelope (recommended; the
gdal_calcspec shape): a JSON object with acalcexpression —'{"calc": "A * 2"}'. - Bare expression string: the calc expression on its own —
"A * 2".
Selecting a specific band or raster per variable (A_index / A_band). You do not need to decompose a multiband raster to do band math. The JSON envelope's per-variable keys — A_index / A_band (and B_, C_, …) — map each variable to a chosen raster (0-based, into the tiles array) and a chosen 1-based band, mirroring gdal_calc. For example, NDVI from bands 4 (NIR) and 3 (Red) of a single raster:
{"calc": "(A - B) / (A + B)", "A_index": 0, "B_index": 0, "A_band": 4, "B_band": 3}
Both A and B read raster 0 (the one tile in the array), with A = band 4 and B = band 3 — the direct equivalent of gdal_calc -A in.tif --A_band=4 -B in.tif --B_band=3 --calc="(A - B) / (A + B)". A variable with no *_index/*_band keeps the default (its ordinal raster, band 1). This works on both tiers.
The only gdal_calc envelope key the lightweight tier does not support is extra_options (raw CLI flags with no NumExpr equivalent); it raises a clear error rather than dropping it silently. The heavy tier honors extra_options as well.
The envelope is portable, but the calc expression language is not fully portable because each tier uses a different evaluator: the heavy tier shells out to GDAL's gdal_calc (NumPy expression syntax); the lightweight tier evaluates with NumExpr. They agree on ordinary arithmetic and comparisons (A * 2, (A - B) / (A + B), A > 0), so those expressions run unchanged on both tiers. They diverge on function spellings — e.g. gdal_calc accepts NumPy calls like numpy.where(A > 0, A, 0), whereas NumExpr wants where(A > 0, A, 0). Keep expressions to basic arithmetic for cross-tier portability. For the heavy tier's full syntax and options, see the GDAL raster calculator reference: gdal_raster_calc.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_mapalgebra(
array(tile),
'{"calc": "(A - B) / (A + B)", "A_index": 0, "B_index": 0, "A_band": 2, "B_band": 1}'
) AS ndvi
FROM multiband_rasters;
+-----------------------------------------------------------+
|ndvi |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
ndvi_spec = (
'{"calc": "(A - B) / (A + B)", '
'"A_index": 0, "B_index": 0, "A_band": 2, "B_band": 1}'
)
result = df.select(
rx.rst_mapalgebra(f.array("tile"), f.lit(ndvi_spec)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band Float32 NDVI tile; per-pixel (NIR-Red)/(NIR+Red) in [-1, 1])
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
ndvi_spec = (
'{"calc": "(A - B) / (A + B)", '
'"A_index": 0, "B_index": 0, "A_band": 2, "B_band": 1}'
)
result = df.select(
rx.rst_mapalgebra(f.array("tile"), f.lit(ndvi_spec)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band Float32 NDVI tile; per-pixel (NIR-Red)/(NIR+Red) in [-1, 1])
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Map algebra: scale band values by factor of 2
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_mapalgebra(array(col("tile")), lit("A * 2")).alias("scaled"))
result.show(truncate = false)
+-----------------------------------------------------------+
|scaled |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(result raster from map algebra: A * 2)
rst_merge
LightweightHeavyweightPowered by rasterio (rasterio.merge). Takes an ARRAY<tile> (in one row) and mosaics them into a single tile spanning the union extent (first-tile-wins on overlap, in array order).
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_merge(tiles: Column): Column — Merge tiles into mosaic.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_merge(array(tile)) AS merged FROM multiband_rasters;
+-----------------------------------------------------------+
|merged |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from _fixtures import multi_band_tiles_df
df = multi_band_tiles_df(spark)
result = (
df.groupBy("region")
.agg(rx.rst_merge(f.collect_list("tile")).alias("tile"))
.first()
)
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(merged raster from co-registered input tiles)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
from ._fixtures import multi_band_tiles_df_heavy
df = multi_band_tiles_df_heavy(spark)
result = (
df.groupBy("region")
.agg(rx.rst_merge(f.collect_list("tile")).alias("tile"))
.first()
)
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(merged raster from co-registered input tiles)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
// Multi-row fixture: 3 single-band tiles (one per band from multiband GeoTIFF)
val multiband = spark.table("multiband_rasters")
val b1 = multiband.select(rx.rst_band(col("tile"), lit(1)).alias("tile")).withColumn("band_index", lit(1))
val b2 = multiband.select(rx.rst_band(col("tile"), lit(2)).alias("tile")).withColumn("band_index", lit(2))
val b3 = multiband.select(rx.rst_band(col("tile"), lit(3)).alias("tile")).withColumn("band_index", lit(3))
val bands = b1.union(b2).union(b3).withColumn("region", lit("R1"))
val result = bands.groupBy("region").agg(rx.rst_merge(collect_list("tile")).alias("merged"))
result.show(truncate = false)
+------+-----------------------------------------------------------+
|region|merged |
+------+-----------------------------------------------------------+
|R1 |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+------+-----------------------------------------------------------+
(merged raster from aligned tiles)
rst_ndvi
LightweightHeavyweightPowered by rasterio + NumPy. Valid pixels match the heavyweight tier; pixels with a zero denominator are set to NoData (-9999) rather than left as non-finite values.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_ndvi(tile: Column, redBand: Column, nirBand: Column): Column — NDVI from band indices.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_ndvi(tile, 1, 2) AS ndvi FROM multiband_rasters;
+-----------------------------------------------------------+
|ndvi |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band NDVI raster: (NIR-Red)/(NIR+Red))
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
result = df.select(rx.rst_ndvi("tile", f.lit(1), f.lit(2)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band NDVI raster: (NIR-Red)/(NIR+Red))
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
result = df.select(rx.rst_ndvi("tile", f.lit(1), f.lit(2)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band NDVI raster: (NIR-Red)/(NIR+Red))
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands: red=1, NIR=2, green=3)
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_ndvi(col("tile"), lit(1), lit(2)).alias("ndvi"))
result.show(truncate = false)
+-----------------------------------------------------------+
|ndvi |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band NDVI raster: (NIR-Red)/(NIR+Red))
rst_rastertoworldcoord
LightweightHeavyweightPowered by rasterio.
Signature: rst_rastertoworldcoord(tile: Column, pixelX: Column, pixelY: Column): Column — Pixel to world coordinates as a struct with .x and .y fields.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT
gbx_rst_rastertoworldcoord(tile, 100, 80) as world_coord,
gbx_rst_rastertoworldcoord(tile, 100, 80).x as easting,
gbx_rst_rastertoworldcoord(tile, 100, 80).y as northing
FROM rasters;
+---------------+-------+--------+
|world_coord |easting|northing|
+---------------+-------+--------+
|{500980.0, ...}|500980 |4599220 |
+---------------+-------+--------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
df = single_band_tile_df(spark)
# Pixel (100, 80) → world struct {x: <easting>, y: <northing>}
result = df.select(
rx.rst_rastertoworldcoord("tile", f.lit(100), f.lit(80)).alias("world_coord")
).first()
+-----------------------------+
|world_coord |
+-----------------------------+
|{2122955.0, -10791275.0} |
+-----------------------------+
(struct with x: DOUBLE, y: DOUBLE)
from pyspark.sql import functions as f
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("rasters")
# Pixel (100, 80) → world struct {x: <easting>, y: <northing>}
result = df.select(
rx.rst_rastertoworldcoord("tile", f.lit(100), f.lit(80)).alias("world_coord")
).first()
+-------------------+
|world_coord |
+-------------------+
|{500980.0, ...} |
+-------------------+
(struct with x: DOUBLE, y: DOUBLE)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_rastertoworldcoord(col("tile"), lit(100), lit(80)).alias("world_coord"))
result.show(truncate = false)
+-------------------+
|world_coord |
+-------------------+
|{500980.0, ...} |
+-------------------+
(struct with x: DOUBLE, y: DOUBLE)
rst_rastertoworldcoordx / rst_rastertoworldcoordy
LightweightHeavyweightPowered by rasterio.
Signature: rst_rastertoworldcoordx(tile: Column, pixelX: Column, pixelY: Column): Column, rst_rastertoworldcoordy(tile: Column, pixelX: Column, pixelY: Column): Column — World X / Y coordinate of a pixel.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT
gbx_rst_rastertoworldcoordx(tile, 100, 80) as easting
FROM rasters;
+-------+
|easting|
+-------+
|500980 |
+-------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
df = single_band_tile_df(spark)
# Pixel col=100 → easting
result = df.select(
rx.rst_rastertoworldcoordx("tile", f.lit(100), f.lit(80)).alias("easting")
).first()
+---------+
|easting |
+---------+
|2122955.0|
+---------+
from pyspark.sql import functions as f
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("rasters")
# Pixel col=100 → easting
result = df.select(
rx.rst_rastertoworldcoordx("tile", f.lit(100), f.lit(80)).alias("easting")
).first()
+-------+
|easting|
+-------+
|500980 |
+-------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_rastertoworldcoordx(col("tile"), lit(100), lit(80)).alias("easting"))
result.show(truncate = false)
+-------+
|easting|
+-------+
|500980 |
+-------+
rst_resample
LightweightHeavyweightPowered by rasterio (rasterio.warp).
Resample a raster tile by a multiplicative factor via gdal.Warp -r, scaling pixel dimensions up or down relative to the source.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_resample(tile: Column, factor: Column, algorithm: Column): Column
Parameters: factor — multiplicative scale factor applied to both width and height (e.g. 2.0 doubles the pixel grid); algorithm — gdalwarp resampling method name (e.g. bilinear, near, cubic, cubicspline, lanczos, average)
The rst_resample* family is GeoBrix's equivalent of the ArcGIS Resample tool — producing a new raster at a different cell size with a chosen resampling method (algorithm: near, bilinear, cubic, cubicspline, lanczos, average). Pick the variant by how you specify the target:
rst_resample— by a multiplicative factor (e.g.2.0doubles the grid).rst_resample_to_res— by target ground resolution in CRS units (e.g. metres per pixel).rst_resample_to_size— by target pixel dimensions (width × height).
Unlike rst_sample — which reads a value at a point and is nearest-pixel only — resampling rewrites the whole grid. So to sample with bilinear or cubic interpolation, resample first with the desired algorithm, then call rst_sample.
See Raster Sampling for the full workflow guide.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Upsample 2x with bilinear interpolation. Output dims = source dims * 2.
SELECT gbx_rst_resample(tile, 2.0, 'bilinear') AS upsampled FROM rasters;
+-----------------------------------------------------------+
|upsampled |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(
rx.rst_resample("tile", f.lit(2.0), f.lit("bilinear")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(2x bilinear upsampled tile; source is 236x161 px; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(
rx.rst_resample("tile", f.lit(2.0), f.lit("bilinear")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(2x bilinear upsampled tile; source is 236x161 px)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_resample(col("tile"), lit(2.0), lit("bilinear")).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(2x bilinear upsampled tile; source is 236x161 px)
rst_resample_to_res
LightweightHeavyweightPowered by rasterio (rasterio.warp).
Resample a raster tile to an explicit ground resolution in CRS units via gdal.Warp -tr.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_resample_to_res(tile: Column, xRes: Column, yRes: Column, algorithm: Column): Column
Parameters: xRes — target pixel width in CRS units (e.g. metres for a metric projection); yRes — target pixel height in CRS units; algorithm — gdalwarp resampling method name (e.g. average, bilinear, near)
Part of the rst_resample* family — see the Esri Resample note for how it relates to the ArcGIS Resample tool and rst_sample.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Downsample to a 100 m grid (metric CRS). 'average' weights cells by area.
SELECT gbx_rst_resample_to_res(tile, 100.0, 100.0, 'average') AS coarse
FROM rasters;
+-----------------------------------------------------------+
|coarse |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(
rx.rst_resample_to_res(
"tile", f.lit(20.0), f.lit(20.0), f.lit("average")
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(downsampled tile; 10 m to 20 m resolution; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(
rx.rst_resample_to_res(
"tile", f.lit(20.0), f.lit(20.0), f.lit("average")
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(downsampled tile; 10 m to 20 m resolution)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_resample_to_res(col("tile"), lit(20.0), lit(20.0), lit("average")).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(downsampled tile; 10 m to 20 m resolution)
rst_resample_to_size
LightweightHeavyweightPowered by rasterio (rasterio.warp).
Resample a raster tile to an explicit pixel grid size via gdal.Warp -ts.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_resample_to_size(tile: Column, widthPx: Column, heightPx: Column, algorithm: Column): Column
Parameters: widthPx — target output width in pixels; heightPx — target output height in pixels; algorithm — gdalwarp resampling method name (e.g. near for categorical rasters, bilinear for continuous)
Part of the rst_resample* family — see the Esri Resample note for how it relates to the ArcGIS Resample tool and rst_sample.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Force a 512 x 512 tile, near-neighbour for categorical rasters.
SELECT gbx_rst_resample_to_size(tile, 512, 512, 'near') AS sized FROM rasters;
+-----------------------------------------------------------+
|sized |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(
rx.rst_resample_to_size("tile", f.lit(100), f.lit(100), f.lit("near")).alias(
"tile"
)
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(resampled tile forced to 100x100 pixels; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(
rx.rst_resample_to_size("tile", f.lit(100), f.lit(100), f.lit("near")).alias(
"tile"
)
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(resampled tile forced to 100x100 pixels)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_resample_to_size(col("tile"), lit(100), lit(100), lit("near")).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(resampled tile forced to 100x100 pixels)
rst_transform
LightweightHeavyweightPowered by rasterio (rasterio.warp). Reprojection uses the GDAL build bundled with rasterio, whose projection database and driver set may be narrower than the heavyweight tier.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_transform(tile: Column, targetSrid: Column): Column — Reproject to a target SRID (a positive EPSG or ESRI code, classified at apply time); 0 or a code in neither registry is rejected with a clear error. Use rst_transformcrs to reproject to a target given as a CRS string. See Coordinate Reference Systems.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Reproject to WGS84
SELECT
path,
gbx_rst_transform(tile, 4326) as wgs84_tile,
gbx_rst_srid(gbx_rst_transform(tile, 4326)) as new_srid
FROM rasters;
-- Reproject and clip
SELECT
path,
gbx_rst_clip(gbx_rst_transform(tile, 4326), boundary, true) as result
FROM rasters;
+----+-----------------------------------------------------------+--------+
|path|wgs84_tile |new_srid|
+----+-----------------------------------------------------------+--------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|4326 |
+----+-----------------------------------------------------------+--------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(rx.rst_transform("tile", f.lit(4326)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(reprojected tile; source EPSG:32618 (UTM Zone 18N) to EPSG:4326; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_transform("tile", f.lit(4326)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(reprojected tile; source EPSG:32618 (UTM Zone 18N) to EPSG:4326)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_transform(col("tile"), lit(4326)).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(reprojected tile; source EPSG:32618 (UTM Zone 18N) to EPSG:4326)
rst_transformcrs
LightweightHeavyweightPowered by rasterio (rasterio.warp) + pyproj. Reprojection uses the GDAL build bundled with rasterio, whose projection database and driver set may be narrower than the heavyweight tier.
Signature: rst_transformcrs(tile: Column, targetCrs: Column): Column — reproject to a target CRS given as a string (EPSG:x / ESRI:x / WKT / PROJ4; an int-castable string is treated as a SRID). See Coordinate Reference Systems.
Distinct from rst_transform (integer EPSG only): rst_transformcrs accepts any CRS string — an authority code (EPSG:3857, ESRI:54008), WKT, or PROJ4 — so you can reproject to a non-EPSG target. An int-castable string ('3857') is treated as an EPSG SRID. See the CRS: SRID int vs CRS string note above.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Reproject to Web Mercator using a CRS string.
-- Unlike rst_transform (int EPSG only), rst_transformcrs also accepts
-- ESRI codes, WKT, or PROJ4 targets. An int-castable string ('3857')
-- is treated as an EPSG SRID.
SELECT
path,
gbx_rst_transformcrs(tile, 'EPSG:3857') as webmercator_tile,
gbx_rst_crs(gbx_rst_transformcrs(tile, 'EPSG:3857')) as new_crs
FROM rasters;
+----+-----------------------------------------------------------+---------+
|path|webmercator_tile |new_crs |
+----+-----------------------------------------------------------+---------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|EPSG:3857|
+----+-----------------------------------------------------------+---------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(
rx.rst_crs(rx.rst_transformcrs("tile", f.lit("EPSG:3857"))).alias("crs")
).first()
+----------+
|crs |
+----------+
|EPSG:3857 |
+----------+
(CRS string of the reprojected tile; accepts authority codes, WKT, or PROJ4)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(
rx.rst_crs(rx.rst_transformcrs("tile", f.lit("EPSG:3857"))).alias("crs")
).first()
+----------+
|crs |
+----------+
|EPSG:3857 |
+----------+
(CRS string of the reprojected tile; accepts authority codes, WKT, or PROJ4)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_crs(rx.rst_transformcrs(col("tile"), lit("EPSG:3857"))).alias("crs"))
result.show()
+----------+
|crs |
+----------+
|EPSG:3857 |
+----------+
(CRS string of the reprojected tile; accepts authority codes, WKT, or PROJ4)
rst_tryopen
LightweightHeavyweightPowered by rasterio.
Signature: rst_tryopen(tile: Column): Column — Validate raster can be opened.
Examples use the multiband fixture (rgb_nir_small.tif, committed to the repo, always openable).
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_tryopen(tile) AS try_open FROM multiband_rasters;
+--------+
|try_open|
+--------+
|true |
+--------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_tryopen("tile").alias("try_open")).first()
+--------+
|try_open|
+--------+
|true |
+--------+
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_tryopen("tile").alias("try_open")).first()
+--------+
|try_open|
+--------+
|true |
+--------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif — committed, always openable).
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_tryopen(col("tile")).alias("try_open"))
result.show()
+--------+
|try_open|
+--------+
|true |
+--------+
rst_updatetype
LightweightHeavyweightPowered by rasterio. Output is re-encoded as GeoTIFF; a NoData value that is not representable in the target type is dropped.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_updatetype(tile: Column, newType: Column): Column — Convert raster data type.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_updatetype(tile, 'Float32') as float_tile FROM rasters;
+-----------------------------------------------------------+
|float_tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(
rx.rst_updatetype("tile", f.lit("Float32")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(type-converted tile; use rst_type to confirm the new data type; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(
rx.rst_updatetype("tile", f.lit("Float32")).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(type-converted tile; use rst_type to confirm the new data type)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_updatetype(col("tile"), lit("Float32")).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(type-converted tile; use rst_type to confirm the new data type)
rst_worldtorastercoord
LightweightHeavyweightPowered by rasterio.
Signature: rst_worldtorastercoord(tile: Column, worldX: Column, worldY: Column): Column — World to pixel coordinates as a struct with .x and .y fields.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Find pixel coordinates for a specific location (center of raster)
SELECT
gbx_rst_worldtorastercoord(tile, 2122955.0, -10791275.0) as pixel_coord,
gbx_rst_worldtorastercoord(tile, 2122955.0, -10791275.0).x as pixel_col,
gbx_rst_worldtorastercoord(tile, 2122955.0, -10791275.0).y as pixel_row
FROM rasters;
+-----------+---------+---------+
|pixel_coord|pixel_col|pixel_row|
+-----------+---------+---------+
|{5490, ...}|5490 |5490 |
+-----------+---------+---------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
df = single_band_tile_df(spark)
# World (2122955, -10791275) in the raster CRS → pixel (100, 80) — the exact
# inverse of the rst_rastertoworldcoord example above.
result = df.select(
rx.rst_worldtorastercoord("tile", f.lit(2122955.0), f.lit(-10791275.0)).alias(
"pixel_coord"
)
).first()
+-----------+
|pixel_coord|
+-----------+
|{100, 80} |
+-----------+
(struct with x: INT, y: INT)
from pyspark.sql import functions as f
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("rasters")
# World (2122955, -10791275) in the raster CRS → pixel (100, 80)
result = df.select(
rx.rst_worldtorastercoord("tile", f.lit(2122955.0), f.lit(-10791275.0)).alias(
"pixel_coord"
)
).first()
+-----------+
|pixel_coord|
+-----------+
|{5490, ...}|
+-----------+
(struct with x: INT, y: INT)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_worldtorastercoord(col("tile"), lit(554880), lit(4545120)).alias("pixel_coord"))
result.show(truncate = false)
+-----------+
|pixel_coord|
+-----------+
|{5490, ...}|
+-----------+
(struct with x: INT, y: INT)
rst_worldtorastercoordx / rst_worldtorastercoordy
LightweightHeavyweightPowered by rasterio.
Signature: rst_worldtorastercoordx(tile: Column, worldX: Column, worldY: Column): Column, rst_worldtorastercoordy(tile: Column, worldX: Column, worldY: Column): Column — Pixel column / row for a world coordinate.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT
gbx_rst_worldtorastercoordx(tile, 2122955.0, -10791275.0) as pixel_col
FROM rasters;
+---------+
|pixel_col|
+---------+
|5490 |
+---------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
df = single_band_tile_df(spark)
# World (2122955, -10791275) → pixel column 100
result = df.select(
rx.rst_worldtorastercoordx("tile", f.lit(2122955.0), f.lit(-10791275.0)).alias(
"pixel_col"
)
).first()
+---------+
|pixel_col|
+---------+
|100 |
+---------+
from pyspark.sql import functions as f
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("rasters")
# World (2122955, -10791275) → pixel column 100
result = df.select(
rx.rst_worldtorastercoordx("tile", f.lit(2122955.0), f.lit(-10791275.0)).alias(
"pixel_col"
)
).first()
+---------+
|pixel_col|
+---------+
|5490 |
+---------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_worldtorastercoordx(col("tile"), lit(554880), lit(4545120)).alias("pixel_col"))
result.show(truncate = false)
+---------+
|pixel_col|
+---------+
|5490 |
+---------+
Web-Mercator Tile Output
Reproject rasters to EPSG:3857 (Web Mercator) and emit slippy-map XYZ tiles. Pair with gbx_pmtiles_agg or the PMTiles writer to publish a raster pyramid as a single .pmtiles archive. See the Helios notebooks for a worked example: NAIP aerial scenes are reprojected and pyramided into a PMTiles archive in NB02.
rst_tilexyz
LightweightHeavyweightPowered by rio-tiler + morecantile. Out-of-extent tiles return a transparent PNG (never null); available output formats depend on rasterio's bundled GDAL build.
Signature: rst_tilexyz(tile: Column, z: Column, x: Column, y: Column, format: Column, tileSize: Column, resampling: Column): Column — Render a single web-mercator XYZ tile from a raster as encoded image bytes (e.g. PNG, JPEG, WebP) at the given tile coordinates and pixel size.
The output is a display web-map tile, not the source's raw bands. PNG and WebP are RGBA (4-band); JPEG is RGB (3-band, no alpha). The alpha channel is a binary transparency mask derived from the source's valid-data footprint, so a pixel that is NoData — whether outside the raster or an internal hole — renders transparent, and both the lightweight and heavyweight tiers agree on which pixels are transparent. Band mapping matches the lightweight tier: a single-band source becomes greyscale RGB (R=G=B), a two-band source uses band 1 as greyscale RGB and band 2 as the alpha channel, three bands map to R, G, B, an existing fourth band is treated as alpha, and five-or-more-band sources use the first three bands as RGB. Non-8-bit sources are contrast-rescaled to 8-bit per the rescale argument (default "auto"; see the rescale note above). WebP alpha requires GDAL WebP-alpha support in the runtime; where absent, WebP falls back to RGB.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Render tile (z=10, x=512, y=512) as 256x256 PNG bytes.
-- rescale='auto' (default) rescales non-8-bit imagery by whole-dataset min/max
-- for display contrast; 'none' keeps the raw full-dtype-range mapping; a
-- 'min,max' string sets explicit bounds.
SELECT
path,
gbx_rst_tilexyz(tile, 10, 512, 512, 'PNG', 256, 'bilinear', 'auto') as tile_png
FROM rasters;
+----+--------+
|path|tile_png|
+----+--------+
|... |[BINARY]|
+----+--------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
df = single_band_tile_df(spark)
# Render z=12, x=1234, y=1523 as a 256x256 PNG. rescale="none" keeps the raw
# dtype mapping (a slippy-map tile off the raster's footprint renders
# transparent — rst_tilexyz never returns null).
result = df.select(
rx.rst_tilexyz(
"tile",
f.lit(12),
f.lit(1234),
f.lit(1523),
f.lit("PNG"),
f.lit(256),
f.lit("bilinear"),
f.lit("none"),
).alias("png_bytes")
).first()
+----------+
|png_bytes |
+----------+
|[BINARY] |
+----------+
(PNG image bytes, 256×256 pixels)
from pyspark.sql import functions as f
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("rasters")
# Render z=12, x=1234, y=1523 as a 256x256 PNG (rescale='none' keeps the raw
# dtype mapping; an off-footprint slippy tile renders transparent, never null).
result = df.select(
rx.rst_tilexyz(
"tile",
f.lit(12),
f.lit(1234),
f.lit(1523),
f.lit("PNG"),
f.lit(256),
f.lit("bilinear"),
f.lit("none"),
).alias("png_bytes")
).first()
+----------+
|png_bytes |
+----------+
|[BINARY] |
+----------+
(PNG image bytes, 256×256 pixels)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_tilexyz(col("tile"), lit(12), lit(1234), lit(1523)).alias("png_bytes"))
result.show(truncate = false)
+----------+
|png_bytes |
+----------+
|[BINARY] |
+----------+
(PNG image bytes, 256×256 pixels)
rst_to_webmercator
LightweightHeavyweightPowered by rasterio (rasterio.warp). Uses rasterio's bundled GDAL build, whose projection and driver coverage may be narrower than the heavyweight tier.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_to_webmercator(tile: Column): Column — Reproject a raster to EPSG:3857 (Web Mercator) using bilinear resampling by default. The returned tile carries srid = 3857.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Reproject to web mercator before slippy-map tiling (default bilinear resampling).
SELECT
path,
gbx_rst_to_webmercator(tile) as web_tile,
gbx_rst_srid(gbx_rst_to_webmercator(tile)) as new_srid
FROM rasters;
+----+-----------------------------------------------------------+--------+
|path|web_tile |new_srid|
+----+-----------------------------------------------------------+--------+
|... |{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|3857 |
+----+-----------------------------------------------------------+--------+
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = single_band_tile_df(spark)
# Reproject to Web Mercator (default bilinear resampling)
result = df.select(rx.rst_to_webmercator("tile").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(reprojected to Web Mercator, EPSG:3857)
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("rasters")
# Reproject to Web Mercator (default bilinear resampling)
result = df.select(rx.rst_to_webmercator("tile").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(reprojected to Web Mercator, EPSG:3857)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_to_webmercator(col("tile")).alias("tile"))
result.show(truncate = false)
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(reprojected to Web Mercator, EPSG:3857)
rst_xyzpyramid
LightweightHeavyweight Streaming UDTFPowered by rio-tiler + morecantile. Streams one XYZ tile row per intersecting tile via streaming UDTF. Bounded by max_z <= 20 and at most 1,000,000 candidate tiles.
Signature: rst_xyzpyramid(tile: Column, minZoom: Column, maxZoom: Column): Column — Generator: explode a raster into one row per intersecting (z, x, y) tile across a zoom range, producing PNG bytes per tile. Use LATERAL VIEW to materialize the rows; the output struct exposes z, x, y, and bytes. Each tile is rendered by rst_tilexyz, so the per-tile PNG is display RGBA with the same band mapping and binary NoData alpha described above.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Explode a raster into per-tile rows across zoom levels 4..6 (PNG, 256px).
-- Optional trailing rescale arg (default 'auto') controls 8-bit display contrast:
-- gbx_rst_xyzpyramid(tile, 4, 6, 'PNG', 256, 'bilinear', 'auto')
SELECT
path,
t.tile.z as z,
t.tile.x as x,
t.tile.y as y,
t.tile.bytes as png_bytes
FROM rasters
LATERAL VIEW gbx_rst_xyzpyramid(tile, 4, 6) AS t;
+----+-+-+-+---------+
|path|z|x|y|png_bytes|
+----+-+-+-+---------+
|... |4|5|6|[BINARY] |
+----+-+-+-+---------+
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
single_band_tile_df(spark).createOrReplaceTempView("rasters")
# One row per (z, x, y) tile across zoom 0..1.
return spark.sql(
"SELECT t.* FROM rasters, "
"LATERAL gbx_rst_xyzpyramid(tile, 0, 1, 'PNG', 256, 'bilinear', 'none') t"
).take(3)
+---+---+---+--------+
|z |x |y |bytes |
+---+---+---+--------+
|0 |0 |0 |[BINARY]|
+---+---+---+--------+
(one row per XYZ tile: z, x, y, and the PNG image bytes)
from pyspark.sql import functions as f
if rx is None:
raise ImportError("rasterx not installed")
df = spark.table("rasters")
# Explode raster into PNG tiles for z=10..12 (returns array via LATERAL VIEW)
result = df.select(
rx.rst_xyzpyramid("tile", f.lit(10), f.lit(12)).alias("tile_array")
).first()
+----------+
|tile_array|
+----------+
|[tile, ...|
+----------+
(array of tile structs: [{z, x, y, bytes}, ...])
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_xyzpyramid(col("tile"), lit(10), lit(12)).alias("tile_array"))
result.show(truncate = false)
+----------+
|tile_array|
+----------+
|[tile, ...|
+----------+
(array of tile structs: [{z, x, y, bytes}, ...])
Vector↔raster bridge
Move data between the raster (tile) and vector (geom) worlds.
rst_polygonize
LightweightHeavyweight Streaming UDTFPowered by rasterio (rasterio.features).
Heavyweight registers gbx_rst_polygonize as a scalar (ARRAY-returning) function — call
it directly: SELECT gbx_rst_polygonize(tile, band, connectedness) AS features FROM <table>,
where each array element is a struct(geom_wkb, value). The lightweight (pyrx) tier
registers it as a streaming Python table function, so lightweight SQL must use LATERAL
— which also streams polygon rows without buffering (avoids OOM on rasters with unbounded
polygon fan-out): SELECT t.geom_wkb, t.value FROM <table>, LATERAL gbx_rst_polygonize(tile, band, connectedness) t.
Both forms appear in the SQL tab below.
Signature (heavyweight): rst_polygonize(tile: Column, band: Column, connectedness: Column): Column — Trace contiguous-value regions of a tile into an array of features. Each feature carries the source pixel value as the value field.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Heavyweight: scalar ARRAY return — call directly; each feature carries the
-- source pixel value as the `value` field. Round-trip: rasterize a polygon then
-- immediately polygonize it.
SELECT gbx_rst_polygonize(
gbx_rst_rasterize(
unhex('010300000001000000050000000000000000000000000000000000000000000000000024400000000000000000000000000000244000000000000024400000000000000000000000000000244000000000000000000000000000000000'),
42.0, 0.0, 0.0, 10.0, 10.0, 100, 100, 4326
)
) AS features;
-- Lightweight (pyrx): streaming table function — must use LATERAL; one row per
-- contiguous-value region as (geom_wkb, value).
SELECT t.geom_wkb, t.value FROM rasters, LATERAL gbx_rst_polygonize(tile, 1, 4) t;
# Heavyweight SQL — one ARRAY of features per row:
+------------------+
|features |
+------------------+
|[{[BINARY], 42.0}]|
+------------------+
# Lightweight SQL — LATERAL streams one row per region (geom_wkb, value):
+--------+-----+
|geom_wkb|value|
+--------+-----+
|... |365.0|
+--------+-----+
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
single_band_tile_df(spark).createOrReplaceTempView("rasters")
# Polygonize band 1 with 4-connectivity.
return spark.sql(
"SELECT t.* FROM rasters, LATERAL gbx_rst_polygonize(tile, 1, 4) t"
).take(3)
+---------+-----+
|geom_wkb |value|
+---------+-----+
|... |365.0|
+---------+-----+
(one row per contiguous region: geom_wkb is WKB binary, value is the region value)
if rx is None:
raise ImportError("rasterx not installed")
from pyspark.sql import functions as f
df = spark.table("rasters")
return df.select(
rx.rst_polygonize("tile", f.lit(1), f.lit(4)).alias("features")
).first()["features"]
+----------------------------------------+
|features |
+----------------------------------------+
|[{[BINARY], 365.0}, {[BINARY], 366.0}] |
+----------------------------------------+
(ARRAY<struct(geom_wkb BINARY (WKB), value DOUBLE)> — one element per region;
the example returns this array via .first()["features"])
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val raster = spark.table("rasters")
val result = raster.select(rx.rst_polygonize(col("tile")).alias("features"))
result.show(truncate = false)
+--------+
|features|
+--------+
|[{...}] |
+--------+
(array of {geom_wkb: binary (WKB), value: double})
rst_rasterize
LightweightHeavyweightPowered by rasterio (rasterio.features).
out_crs parameterThe lightweight Python binding accepts an optional trailing out_crs (string CRS) argument that the SQL and heavyweight/Scala tiers do not — the heavyweight RST_Rasterize builder is strictly 9-argument (out_srid only) and rejects a 10th argument. In lightweight Python, out_crs (string) wins over the int out_srid; both → error; neither → the geometry's carried source CRS. This is a lightweight superset, not a heavyweight regression.
Signature: rst_rasterize(geom: Column, burnValue: Column, xMin: Column, yMin: Column, xMax: Column, yMax: Column, width: Column, height: Column, [out_srid: Column = null]): Column — Burn a polygon (WKB) into a fresh GeoTIFF tile at the given extent and pixel dimensions. Pixels inside the polygon carry burnValue; pixels outside are NoData. The output CRS is out_srid (integer EPSG code); the geometry is reprojected from its source CRS into the output CRS before burning. When out_srid is omitted the geometry's carried source CRS is used. See Coordinate Reference Systems.
SQL:
- SQL
- Python (light)
- Python (heavy)
- Scala
-- WKB hex below is POLYGON((0 0, 10 0, 10 10, 0 10, 0 0)). The output `tile`
-- is a GTiff-backed raster at the given extent and resolution; pixels inside
-- the polygon carry the burn value (42.0), pixels outside are NoData.
SELECT gbx_rst_rasterize(
unhex('010300000001000000050000000000000000000000000000000000000000000000000024400000000000000000000000000000244000000000000024400000000000000000000000000000244000000000000000000000000000000000'),
42.0, 0.0, 0.0, 10.0, 10.0, 100, 100, 4326
) AS tile;
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(rasterized tile: pixels inside the polygon carry the burn value; outside = NoData)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import Row, functions as f
rx.register(spark)
# Create a small synthetic geometry DataFrame with WKT polygon
df = spark.createDataFrame([Row(geom="POLYGON((2 2, 8 2, 8 8, 2 8, 2 2))")])
# Rasterize a square polygon (value=1.0, extent=(0,0,10,10), 10x10 pixels, EPSG:4326)
result = df.select(
rx.rst_rasterize(
f.col("geom"),
f.lit(1.0),
f.lit(0.0),
f.lit(0.0),
f.lit(10.0),
f.lit(10.0),
f.lit(10),
f.lit(10),
f.lit(4326),
).alias("tile")
).collect()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(rasterized tile: pixels inside the polygon carry the burn value; outside = NoData)
from pyspark.sql import Row, functions as f
if rx is None:
raise ImportError("rasterx not installed")
rx.register(spark)
# Create a synthetic geometry DataFrame with WKT polygon
df = spark.createDataFrame([Row(geom="POLYGON((2 2, 8 2, 8 8, 2 8, 2 2))")])
# Rasterize a square polygon (value=1.0, extent=(0,0,10,10), 10x10 pixels, EPSG:4326)
result = df.select(
rx.rst_rasterize(
f.col("geom"),
f.lit(1.0),
f.lit(0.0),
f.lit(0.0),
f.lit(10.0),
f.lit(10.0),
f.lit(10),
f.lit(10),
f.lit(4326),
).alias("tile")
).collect()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(rasterized tile: pixels inside the polygon carry the burn value; outside = NoData)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
rx.register(spark)
val wkt = "POLYGON((2 2, 8 2, 8 8, 2 8, 2 2))"
val result = spark.createDataFrame(Seq(
(wkt,)
)).toDF("geom")
.select(rx.rst_rasterize(
col("geom"),
lit(1.0),
lit(0.0), lit(0.0), lit(10.0), lit(10.0),
lit(10), lit(10),
lit(4326)
).alias("tile"))
result.show(truncate = false)
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(rasterized tile: pixels inside the polygon carry the burn value; outside = NoData)
Terrain Analysis
Thin wrappers around gdal.DEMProcessing for digital elevation model (DEM) derivatives. Each function takes a single-band DEM tile and returns a derived tile of the same footprint. See the Helios notebooks for a worked example: gbx_rst_slope, gbx_rst_aspect, and gbx_rst_hillshade are applied to 3DEP DEMs in NB03 to produce terrain layers and a per-H3-cell solar score.
rst_aspect
LightweightHeavyweightPowered by NumPy — a reimplementation of GDAL gdaldem; results are close but not bit-identical to the heavyweight tier (edge pixels are filled and NoData is not excluded from the 3×3 window).
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_aspect(tile: Column, trigonometric: Column, zeroForFlat: Column): Column — Compass direction of steepest descent in degrees (0=N, 90=E, 180=S, 270=W). Flat areas return -9999 unless zeroForFlat = true. Set trigonometric = true for mathematical convention (0=E, counter-clockwise).
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Aspect in compass degrees (0=N, 90=E, 180=S, 270=W). Flat areas get -9999
-- unless zero_for_flat=true.
SELECT gbx_rst_aspect(tile, false, false) AS aspect FROM dem_rasters;
+-----------------------------------------------------------+
|aspect |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(aspect in compass degrees: 0=N, 90=E, 180=S, 270=W)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
df = dem_tile_df(spark)
result = df.select(
rx.rst_aspect("tile", f.lit(False), f.lit(False)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(aspect in compass degrees: 0=N, 90=E, 180=S, 270=W)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("dem_rasters")
result = df.select(
rx.rst_aspect("tile", f.lit(False), f.lit(False)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(aspect in compass degrees: 0=N, 90=E, 180=S, 270=W)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val dem = spark.table("dem_rasters")
val result = dem.select(rx.rst_aspect(col("tile"), lit(false), lit(false)).alias("aspect"))
result.show(truncate = false)
+-----------------------------------------------------------+
|aspect |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(aspect in compass degrees: 0=N, 90=E, 180=S, 270=W)
rst_color_relief
LightweightHeavyweightThe two tiers use different color interpolation engines, so their pixel values are close but not identical and are not byte-comparable: the heavyweight tier calls GDAL gdal.DEMProcessing color-relief (GDAL's native C interpolation), while the lightweight (pyrx) tier is a NumPy reimplementation using per-channel np.interp. Two known differences on the lightweight tier: the gdaldem default color keyword is not supported (it is skipped; out-of-range elevations are clamped to the nearest color stop by np.interp instead), and boundary/edge interpolation may differ slightly. Both tiers emit an RGB or RGBA Byte tile (RGBA when any color-table entry carries an alpha column). Because the outputs diverge, this function is not cross-tier fingerprint-compared in benchmarks (measured timing-only); pick the tier by your pipeline (heavyweight for gdaldem-exact output, lightweight for the pure-Python/Serverless path).
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_color_relief(tile: Column, colorTablePath: Column): Column — Apply a gdaldem color table (elevation R G B [A] per line) to produce an RGB(A) visualization tile. Special values nv (NoData color), 0%, and 100% (percentages of the band value range) are honored on both tiers; the default keyword is honored only on the heavyweight tier (see the tier note above).
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Map elevation values to RGBA colors via a gdaldem color table.
SELECT gbx_rst_color_relief(tile, '{COLOR_TABLE_PATH}') AS rgba
FROM dem_rasters;
+-----------------------------------------------------------+
|rgba |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(4-band RGBA tile mapped via gdaldem color table)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
from _fixtures import color_table_path
rx.register(spark)
df = dem_tile_df(spark)
clr_path = str(color_table_path())
result = df.select(
rx.rst_color_relief("tile", f.lit(clr_path)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(4-band RGBA tile mapped via gdaldem color table)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
from ._fixtures import color_table_path
df = spark.table("dem_rasters")
clr_path = str(color_table_path())
result = df.select(
rx.rst_color_relief("tile", f.lit(clr_path)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(4-band RGBA tile mapped via gdaldem color table)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val dem = spark.table("dem_rasters")
val result = dem.select(rx.rst_color_relief(col("tile"), lit("src/test/resources/binary/elevation/elevation.clr")).alias("rgba"))
result.show(truncate = false)
+-----------------------------------------------------------+
|rgba |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(4-band RGBA tile mapped via gdaldem color table)
rst_hillshade
LightweightHeavyweightPowered by NumPy — a reimplementation of GDAL gdaldem; results are close but not bit-identical to the heavyweight tier (edge pixels are filled and NoData is not excluded from the 3×3 window).
xscale / yscale parametersThe lightweight Python binding accepts two optional horizontal-scale overrides — xscale and yscale — that the SQL and heavyweight/Scala tiers do not (the heavyweight RST_Hillshade builder is strictly 4-argument: tile, azimuth, altitude, z_factor). By default the horizontal scale is auto-derived from the CRS; pass both xscale and yscale to override it. This is a lightweight superset, not a heavyweight regression.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_hillshade(tile: Column, azimuth: Column, altitude: Column, zFactor: Column): Column — 8-bit (0..255) shaded relief image. Common values: NW sun azimuth 315.0, altitude 45.0, zFactor = 1.0.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- 8-bit (0..255) hillshade: NW sun, 45-deg altitude, default z-factor.
SELECT gbx_rst_hillshade(tile, 315.0, 45.0, 1.0) AS hillshade FROM dem_rasters;
+-----------------------------------------------------------+
|hillshade |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(8-bit hillshade: 0..255, NW azimuth 45-degree altitude)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
df = dem_tile_df(spark)
result = df.select(
rx.rst_hillshade("tile", f.lit(315.0), f.lit(45.0), f.lit(1.0)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(8-bit hillshade: 0..255, NW azimuth 45-degree altitude)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("dem_rasters")
result = df.select(
rx.rst_hillshade("tile", f.lit(315.0), f.lit(45.0), f.lit(1.0)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(8-bit hillshade: 0..255, NW azimuth 45-degree altitude)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val dem = spark.table("dem_rasters")
val result = dem.select(rx.rst_hillshade(col("tile"), lit(315.0), lit(45.0), lit(1.0)).alias("shade"))
result.show(truncate = false)
+-----------------------------------------------------------+
|shade |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(8-bit hillshade: NW azimuth, 45-degree altitude)
rst_roughness
LightweightHeavyweightPowered by NumPy — a reimplementation of GDAL gdaldem; results are close but not bit-identical to the heavyweight tier (edge pixels are filled and NoData is not excluded from the 3×3 window).
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_roughness(tile: Column): Column — Largest absolute difference between a pixel and any of its 8 neighbours in a 3×3 window.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Roughness: max absolute neighbour difference in a 3x3 window.
SELECT gbx_rst_roughness(tile) AS roughness FROM dem_rasters;
+-----------------------------------------------------------+
|roughness |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(roughness: max absolute difference in 3x3 window)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = dem_tile_df(spark)
result = df.select(rx.rst_roughness("tile").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(roughness: max absolute difference in 3x3 window)
from databricks.labs.gbx.rasterx import functions as rx
df = spark.table("dem_rasters")
result = df.select(rx.rst_roughness("tile").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(roughness: max absolute difference in 3x3 window)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val dem = spark.table("dem_rasters")
val result = dem.select(rx.rst_roughness(col("tile")).alias("roughness"))
result.show(truncate = false)
+-----------------------------------------------------------+
|roughness |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(roughness: max absolute difference in 3x3 window)
rst_slope
LightweightHeavyweightPowered by NumPy — a reimplementation of GDAL gdaldem. Results are close but not bit-identical: edge pixels are filled by replicating the border (gdaldem leaves them NoData) and NoData cells are not excluded from the 3×3 window.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_slope(tile: Column, unit: Column, xscale: Column, yscale: Column): Column — Compute slope per pixel. unit is 'degrees' or 'percent'; xscale and yscale are the elevation/horizontal unit ratio per axis (supply both or neither; when omitted, GDAL 3.11+ auto-derives from the raster CRS). For isotropic scaling pass the same value to both (e.g. 1.0, 1.0 for a projected CRS in metres).
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Slope in degrees per pixel (auto-scale from CRS). Use unit='percent' for rise/run.
-- Pass xscale and yscale together to override the horizontal scale per axis.
SELECT gbx_rst_slope(tile, 'degrees', 1.0, 1.0) AS slope FROM dem_rasters;
+-----------------------------------------------------------+
|slope |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(slope in degrees; auto-scaled from raster CRS units)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
df = dem_tile_df(spark)
result = df.select(
rx.rst_slope("tile", f.lit("degrees"), f.lit(1.0), f.lit(1.0)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(slope in degrees; auto-scaled from raster CRS units)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("dem_rasters")
result = df.select(
rx.rst_slope("tile", f.lit("degrees"), f.lit(1.0), f.lit(1.0)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(slope in degrees; auto-scaled from raster CRS units)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val dem = spark.table("dem_rasters")
val result = dem.select(rx.rst_slope(col("tile"), lit("degrees"), lit(1.0), lit(1.0)).alias("slope"))
result.show(truncate = false)
+-----------------------------------------------------------+
|slope |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(slope in degrees; auto-scaled from raster CRS units)
rst_tpi
LightweightHeavyweightPowered by NumPy — a reimplementation of GDAL gdaldem; results are close but not bit-identical to the heavyweight tier (edge pixels are filled and NoData is not excluded from the 3×3 window).
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_tpi(tile: Column): Column — Topographic Position Index — pixel value minus the mean of its 8 neighbours. Positive values are ridges, negative values are valleys.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- TPI: difference from neighbour-mean; +ve = ridge, -ve = valley.
SELECT gbx_rst_tpi(tile) AS tpi FROM dem_rasters;
+-----------------------------------------------------------+
|tpi |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TPI: positive=ridge, negative=valley)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = dem_tile_df(spark)
result = df.select(rx.rst_tpi("tile").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TPI: positive=ridge, negative=valley)
from databricks.labs.gbx.rasterx import functions as rx
df = spark.table("dem_rasters")
result = df.select(rx.rst_tpi("tile").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TPI: positive=ridge, negative=valley)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val dem = spark.table("dem_rasters")
val result = dem.select(rx.rst_tpi(col("tile")).alias("tpi"))
result.show(truncate = false)
+-----------------------------------------------------------+
|tpi |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TPI: positive=ridge, negative=valley)
rst_tri
LightweightHeavyweightPowered by NumPy — a reimplementation of GDAL gdaldem; results are close but not bit-identical to the heavyweight tier (edge pixels are filled and NoData is not excluded from the 3×3 window).
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_tri(tile: Column): Column — Terrain Ruggedness Index — mean absolute difference between a pixel and its 8 neighbours. Useful for landscape-ecology habitat scoring.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- TRI: mean absolute neighbour difference; useful for landscape ecology.
SELECT gbx_rst_tri(tile) AS tri FROM dem_rasters;
+-----------------------------------------------------------+
|tri |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TRI: mean absolute neighbour difference)
from databricks.labs.gbx.pyrx import functions as rx
rx.register(spark)
df = dem_tile_df(spark)
result = df.select(rx.rst_tri("tile").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TRI: mean absolute neighbour difference)
from databricks.labs.gbx.rasterx import functions as rx
df = spark.table("dem_rasters")
result = df.select(rx.rst_tri("tile").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TRI: mean absolute neighbour difference)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val dem = spark.table("dem_rasters")
val result = dem.select(rx.rst_tri(col("tile")).alias("tri"))
result.show(truncate = false)
+-----------------------------------------------------------+
|tri |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(TRI: mean absolute neighbour difference)
Spectral Indices
Multi-band satellite math built on gbx_rst_mapalgebra. Band arguments are 1-based GDAL band indices; the output is always a single-band Float32 GeoTIFF tile. gbx_rst_ndvi is documented under Operations.
rst_evi
LightweightHeavyweightPowered by rasterio + NumPy. Valid pixels match the heavyweight tier; pixels with a zero denominator are set to NoData (-9999) rather than left as non-finite values.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_evi(tile: Column, redBand: Column, nirBand: Column, blueBand: Column): Column — Enhanced Vegetation Index. Formula: G * (NIR - Red) / (NIR + C1*Red - C2*Blue + L) with MODIS canonical coefficients G=2.5, L=1.0, C1=6.0, C2=7.5.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_evi(tile, 1, 2, 3) AS evi FROM multiband_rasters;
+-----------------------------------------------------------+
|evi |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band EVI raster: G*(NIR-Red)/(NIR+C1*Red-C2*Blue+L))
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
# EVI requires red (band 1), NIR (band 2), blue (band 3 as proxy)
result = df.select(
rx.rst_evi("tile", f.lit(1), f.lit(2), f.lit(3)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band EVI raster: G*(NIR-Red)/(NIR+C1*Red-C2*Blue+L))
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
result = df.select(
rx.rst_evi("tile", f.lit(1), f.lit(2), f.lit(3)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band EVI raster: G*(NIR-Red)/(NIR+C1*Red-C2*Blue+L))
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture with red, NIR, and green (as blue)
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_evi(col("tile"), lit(1), lit(2), lit(3)).alias("evi"))
result.show(truncate = false)
+-----------------------------------------------------------+
|evi |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band EVI raster: G*(NIR-Red)/(NIR+C1*Red-C2*Blue+L))
rst_index
LightweightHeavyweightPowered by rasterio + NumExpr. Generic named-index dispatcher over a band_map; single-band Float32. Zero-denominator pixels are set to NoData (−9999).
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_index(tile: Column, indexName: Column, bandMap: Column): Column — Generic dispatcher that picks a built-in formula by name and wires bands via a MAP<STRING, INT> (e.g. map('red', 1, 'nir', 2)). Built-in names: ndvi, gndvi, msavi, ndvi_re, ndmi, ndsi.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_index(tile, 'ndvi', map('red', 1, 'nir', 2)) AS ndvi FROM multiband_rasters;
+-----------------------------------------------------------+
|ndvi |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band index raster computed from named formula)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
# Compute NDVI via the generic index dispatcher with a red/NIR band map.
band_map = f.create_map(f.lit("red"), f.lit(1), f.lit("nir"), f.lit(2))
result = df.select(
rx.rst_index("tile", f.lit("ndvi"), band_map).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band index raster computed from named formula)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
band_map = f.create_map(f.lit("red"), f.lit(1), f.lit("nir"), f.lit(2))
result = df.select(
rx.rst_index("tile", f.lit("ndvi"), band_map).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band index raster computed from named formula)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Generic index dispatcher: computes NDVI via named formula with band map
val rasters = spark.table("multiband_rasters")
val bandMap = create_map(lit("red"), lit(1), lit("nir"), lit(2))
val result = rasters.select(rx.rst_index(col("tile"), lit("ndvi"), bandMap).alias("index"))
result.show(truncate = false)
+-----------------------------------------------------------+
|index |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band index raster from named formula)
rst_nbr
LightweightHeavyweightPowered by rasterio + NumPy. Valid pixels match the heavyweight tier; pixels with a zero denominator are set to NoData (-9999) rather than left as non-finite values.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_nbr(tile: Column, nirBand: Column, swirBand: Column): Column — Normalized Burn Ratio. Formula: (NIR - SWIR) / (NIR + SWIR). The pre-/post-fire difference (dNBR) is the canonical burn-severity index.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_nbr(tile, 2, 3) AS nbr FROM multiband_rasters;
+-----------------------------------------------------------+
|nbr |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band NBR raster: (NIR-SWIR)/(NIR+SWIR))
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
# Band 2 (NIR), band 3 (SWIR proxy)
result = df.select(rx.rst_nbr("tile", f.lit(2), f.lit(3)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band NBR raster: (NIR-SWIR)/(NIR+SWIR))
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
result = df.select(rx.rst_nbr("tile", f.lit(2), f.lit(3)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band NBR raster: (NIR-SWIR)/(NIR+SWIR))
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Normalized Burn Ratio: NIR (band 2) and green (band 3) as SWIR substitute
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_nbr(col("tile"), lit(2), lit(3)).alias("nbr"))
result.show(truncate = false)
+-----------------------------------------------------------+
|nbr |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band NBR raster: (NIR-SWIR)/(NIR+SWIR))
rst_ndwi
LightweightHeavyweightPowered by rasterio + NumPy. Valid pixels match the heavyweight tier; pixels with a zero denominator are set to NoData (-9999) rather than left as non-finite values.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_ndwi(tile: Column, greenBand: Column, nirBand: Column): Column — Normalized Difference Water Index (McFeeters 1996). Formula: (Green - NIR) / (Green + NIR). Positive values typically indicate open water.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_ndwi(tile, 3, 2) AS ndwi FROM multiband_rasters;
+-----------------------------------------------------------+
|ndwi |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band NDWI raster: (Green-NIR)/(Green+NIR))
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
# Use green (band 3) and NIR (band 2)
result = df.select(rx.rst_ndwi("tile", f.lit(3), f.lit(2)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band NDWI raster: (Green-NIR)/(Green+NIR))
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
result = df.select(rx.rst_ndwi("tile", f.lit(3), f.lit(2)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band NDWI raster: (Green-NIR)/(Green+NIR))
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Normalized Difference Water Index: green (band 3) and NIR (band 2)
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_ndwi(col("tile"), lit(3), lit(2)).alias("ndwi"))
result.show(truncate = false)
+-----------------------------------------------------------+
|ndwi |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band NDWI raster: (Green-NIR)/(Green+NIR))
rst_savi
LightweightHeavyweightPowered by rasterio + NumPy. Valid pixels match the heavyweight tier; pixels with a zero denominator are set to NoData (-9999) rather than left as non-finite values.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_savi(tile: Column, redBand: Column, nirBand: Column, l: Column): Column — Soil-Adjusted Vegetation Index. Formula: (NIR - Red) / (NIR + Red + L) * (1 + L). L = 0.5 (the canonical default) is a balanced soil/vegetation tradeoff; L = 0 reduces SAVI to NDVI.
- SQL
- Python (light)
- Python (heavy)
- Scala
SELECT gbx_rst_savi(tile, 1, 2, 0.5) AS savi FROM multiband_rasters;
+-----------------------------------------------------------+
|savi |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band SAVI raster: (NIR-Red)/(NIR+Red+L)*(1+L))
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
result = df.select(rx.rst_savi("tile", f.lit(1), f.lit(2)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band SAVI raster: (NIR-Red)/(NIR+Red+L)*(1+L))
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
result = df.select(rx.rst_savi("tile", f.lit(1), f.lit(2)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band SAVI raster: (NIR-Red)/(NIR+Red+L)*(1+L))
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Soil-Adjusted Vegetation Index: uses red (band 1) and NIR (band 2)
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_savi(col("tile"), lit(1), lit(2)).alias("savi"))
result.show(truncate = false)
+-----------------------------------------------------------+
|savi |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single-band SAVI raster: (NIR-Red)/(NIR+Red+L)*(1+L))
Pixel ops + extraction
Per-pixel transformations and band-level extraction.
rst_band
LightweightHeavyweightPowered by rasterio.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_band(tile: Column, bandIndex: Column): Column — Extract a single band from a multi-band raster as a new single-band tile (gdal.Translate -b N). 1-based band index.
Example uses the multiband fixture (rgb_nir_small.tif, 3 bands) to demonstrate extraction; result has num_bands = 1.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Pull band 1 (1-based) as a fresh single-band tile.
SELECT gbx_rst_band(tile, 1) AS b1 FROM rasters;
+-----------------------------------------------------------+
|b1 |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("multiband_rasters")
result = df.select(rx.rst_band("tile", f.lit(1)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single band extracted from the 3-band multiband fixture; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("multiband_rasters")
result = tile_df.select(rx.rst_band("tile", f.lit(1)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single band extracted from the 3-band multiband fixture)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands) to demonstrate band extraction.
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_band(col("tile"), lit(1)).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(single band extracted from the 3-band multiband fixture)
rst_buildoverviews
LightweightHeavyweightPowered by rasterio. Builds internal GeoTIFF overviews.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_buildoverviews(tile: Column, levels: Column, [resampling: Column = lit("average")]): Column — Add pyramid overview levels to a tile via ds.BuildOverviews. levels is an ARRAY<INT> (e.g. array(2, 4, 8, 16)); resampling is one of nearest, average, gauss, cubic, cubicspline, lanczos, bilinear, mode.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Add 2x / 4x overviews to the tile via the 'average' resampling.
SELECT gbx_rst_buildoverviews(tile, array(2, 4), 'average') AS withovr
FROM rasters;
+-----------------------------------------------------------+
|withovr |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(
rx.rst_buildoverviews("tile", f.array(f.lit(2), f.lit(4))).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(tile with internal overviews at levels [2, 4] embedded; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(
rx.rst_buildoverviews("tile", f.array(f.lit(2), f.lit(4))).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(tile with internal overviews at levels [2, 4] embedded)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_buildoverviews(col("tile"), array(lit(2), lit(4))).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(tile with internal overviews at levels [2, 4] embedded)
rst_fillnodata
LightweightHeavyweightPowered by rasterio (rasterio.fill).
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_fillnodata(tile: Column, [maxSearchDist: Column = lit(100), smoothingIter: Column = lit(0)]): Column — Fill NoData pixels via gdal.FillNodata using inverse-distance interpolation from neighbors within maxSearchDist pixels. smoothingIter applies an optional post-fill 3×3 smoothing pass.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Fill NoData holes searching up to 100 pixels in each direction.
SELECT gbx_rst_fillnodata(tile, 100.0, 0) AS filled FROM rasters;
+-----------------------------------------------------------+
|filled |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(
rx.rst_fillnodata("tile", f.lit(100.0), f.lit(0)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(filled tile; NoData holes searched within 100 pixels; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(
rx.rst_fillnodata("tile", f.lit(100.0), f.lit(0)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(filled tile; NoData holes searched within 100 pixels)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_fillnodata(col("tile"), lit(100.0), lit(0)).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(filled tile; NoData holes searched within 100 pixels)
rst_histogram
LightweightHeavyweightPowered by rasterio + NumPy. Per-band bucket counts via numpy.histogram; map keys are band_<i> (1-based).
Signature: rst_histogram(tile: Column, [bands: Column = null, nBuckets: Column = lit(256), min: Column = null, max: Column = null, includeNodata: Column = lit(false)]): Column — Compute per-band histograms via band.GetHistogram. Returns MAP<STRING, ARRAY<LONG>> keyed by "band_<n>" with bucket counts. If bands is null, all bands are processed; if min / max are null, GDAL auto-detects the range.
Examples use the multiband fixture (rgb_nir_small.tif, 3 bands) so the histogram has entries for each band.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- multiband_rasters view is from rgb_nir_small.tif (3 bands: red, NIR, green)
SELECT gbx_rst_histogram(tile) AS histogram FROM multiband_rasters;
+-------------------------------------------------+
|histogram |
+-------------------------------------------------+
|{band_1 -> [1, 0, 0, ...], band_2 -> [1, 0, 1,...|
+-------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("multiband_rasters")
result = df.select(rx.rst_histogram("tile").alias("histogram")).first()
+--------------------------------------------------+
|histogram |
+--------------------------------------------------+
|{band_1 -> [1, 0, 0, ...], band_2 -> [1, 0, 1,... |
+--------------------------------------------------+
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("multiband_rasters")
result = tile_df.select(
rx.rst_histogram("tile", f.lit(256), f.lit(0.0), f.lit(255.0)).alias(
"histogram"
)
).first()
+--------------------------------------------------+
|histogram |
+--------------------------------------------------+
|{band_1 -> [1, 0, 0, ...], band_2 -> [1, 0, 1,... |
+--------------------------------------------------+
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
// Uses multiband fixture (rgb_nir_small.tif, 3 bands) so histogram has entries per band.
val rasters = spark.table("multiband_rasters")
val result = rasters.select(rx.rst_histogram(col("tile")).alias("histogram"))
result.show(truncate = false)
+--------------------------------------------------+
|histogram |
+--------------------------------------------------+
|{band_1 -> [1, 0, 0, ...], band_2 -> [1, 0, 1,... |
+--------------------------------------------------+
rst_sample
LightweightHeavyweightPowered by rasterio. Point geometries only; the point is assumed to be in the raster's CRS.
Signature: rst_sample(tile: Column, geom: Column, [crs: Column = null]): Column — Sample the raster at the geometry's location(s). For a POINT, returns ARRAY<DOUBLE> of one value per band at the nearest pixel. The point is reprojected from its source CRS to the raster CRS: an EWKB/EWKT embedded SRID wins, else the optional crs (string, source role), else assumed already aligned. See Coordinate Reference Systems.
rst_sample is GeoBrix's equivalent of the ArcGIS Sample tool's RS_VALUE — reading a raster's cell value at a point. It returns ARRAY<DOUBLE> (one value per band, in band order), or null where the point falls outside the raster (the NoData/out-of-extent case). Sampling is nearest-pixel; for bilinear or cubic, run rst_resample first, then sample.
- Multiple rasters (Esri's
RS_VALUE1,RS_VALUE2, …): callrst_sampleonce per raster and alias each result column. - A points table against a raster tile set (the Sample-tool workflow at scale): join the points to the tiles that contain them — with
st_intersects, or a raster grid tessellation such asrst_h3_tessellatefor an index join — then callrst_sample(tile, point)per row.
See Raster Sampling for the full workflow guide.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Sample the DEM at a point in its native CRS (EPSG:32618). Tagging the point
-- with an SRID (EWKT `SRID=32618;...`) lets gbx_rst_sample land it correctly.
SELECT gbx_rst_sample(tile, 'SRID=32618;POINT(500320 4500320)') AS values FROM dem_rasters;
+-------+
|values |
+-------+
|[302.0]|
+-------+
(array of sampled values, one per band)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
df = dem_tile_df(spark)
result = df.select(
rx.rst_sample("tile", f.lit("SRID=32618;POINT(500320 4500320)")).alias("values")
).first()
+-------+
|values |
+-------+
|[302.0]|
+-------+
(array of sampled values, one per band)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("dem_rasters")
result = df.select(
rx.rst_sample("tile", f.lit("SRID=32618;POINT(500320 4500320)")).alias("values")
).first()
+-------+
|values |
+-------+
|[302.0]|
+-------+
(array of sampled values, one per band)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val dem = spark.table("dem_rasters")
val result = dem.select(rx.rst_sample(col("tile"), lit("SRID=4326;POINT(-73.97 40.75)")).alias("sampled"))
result.show(truncate = false)
+--------+
|sampled |
+--------+
|[1234.5]|
+--------+
(array of sampled values, one per band)
rst_setcrs
LightweightHeavyweightPowered by rasterio + pyproj. Stamps the CRS without reprojecting.
Signature: rst_setcrs(tile: Column, crs: Column): Column — Stamp a CRS onto a raster that lacks (or has a wrong) spatial reference, from a CRS string (EPSG:x / ESRI:x / WKT / PROJ4; an int-castable string is treated as a SRID). Does NOT reproject — only rewrites the CRS metadata. Use rst_transformcrs when you need an actual reprojection. See Coordinate Reference Systems.
Distinct from rst_setsrid (integer EPSG only): rst_setcrs accepts any CRS string — an authority code (EPSG:3857, ESRI:54008), WKT, or PROJ4 — so a non-EPSG CRS can be applied. An int-castable string ('4326') behaves like rst_setsrid(tile, 4326). See the CRS: SRID int vs CRS string note above.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Relabel the tile's CRS to Web Mercator without warping pixels.
-- Accepts authority strings (EPSG:/ESRI:), WKT, or PROJ4; an int-castable
-- string ('4326') behaves like rst_setsrid(tile, 4326).
-- Use rst_transformcrs if you actually need a reprojection.
SELECT gbx_rst_setcrs(tile, 'EPSG:3857') AS tagged FROM rasters;
+-----------------------------------------------------------+
|tagged |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(
rx.rst_crs(rx.rst_setcrs("tile", f.lit("EPSG:32618"))).alias("crs")
).first()
+----------+
|crs |
+----------+
|EPSG:32618|
+----------+
(CRS string after stamping; does NOT reproject — use rst_transformcrs to reproject)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(
rx.rst_crs(rx.rst_setcrs("tile", f.lit("EPSG:32618"))).alias("crs")
).first()
+----------+
|crs |
+----------+
|EPSG:32618|
+----------+
(CRS string after stamping; does NOT reproject — use rst_transformcrs to reproject)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_crs(rx.rst_setcrs(col("tile"), lit("EPSG:32618"))).alias("crs"))
result.show()
+----------+
|crs |
+----------+
|EPSG:32618|
+----------+
(CRS string after stamping; does NOT reproject — use rst_transformcrs to reproject)
rst_setsrid
LightweightHeavyweightPowered by rasterio. Stamps the CRS without reprojecting.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_setsrid(tile: Column, srid: Column): Column — Stamp a SRID (>= 0; an EPSG or ESRI code) onto a raster that lacks (or has a wrong) spatial reference; 0 clears the CRS. Does NOT reproject — only rewrites the CRS metadata. A negative SRID is rejected, and a positive code that is neither EPSG nor ESRI raises when stamped. Use rst_transform when you need an actual reprojection, or rst_setcrs to stamp from a CRS string. See Coordinate Reference Systems.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Tag the tile as EPSG:4326 without warping pixels.
-- Use rst_transform if you actually need a reprojection.
SELECT gbx_rst_setsrid(tile, 4326) AS tagged FROM rasters;
+-----------------------------------------------------------+
|tagged |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(rx.rst_setsrid("tile", f.lit(32618)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(tile with SRID stamped to 32618; does NOT reproject — use rst_transform to reproject; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_setsrid("tile", f.lit(32618)).alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(tile with SRID stamped to 32618; does NOT reproject — use rst_transform to reproject)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_setsrid(col("tile"), lit(32618)).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(tile with SRID stamped to 32618; does NOT reproject — use rst_transform to reproject)
rst_threshold
LightweightHeavyweightPowered by NumPy. This tier keeps each passing pixel's original value and sets failing pixels to NoData; the heavyweight tier instead returns a 0/1 binary mask (Float32). Choose the tier that matches the output you need.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_threshold(tile: Column, op: Column, value: Column): Column — Binarize the raster: pixels matching op value get 1, others get 0. op is one of >, >=, <, <=, ==, !=. Output is a Byte raster (0/1) sized to the input extent. Implemented as a gbx_rst_mapalgebra template.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Mark all pixels above 100 m as 1, others as 0.
SELECT gbx_rst_threshold(tile, '>', 100.0) AS mask FROM rasters;
+-----------------------------------------------------------+
|mask |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
df = spark.table("rasters")
result = df.select(
rx.rst_threshold("tile", f.lit(">"), f.lit(0.0)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(binary mask tile; pixels > 0.0 → 1, others → 0; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
tile_df = spark.table("rasters")
result = tile_df.select(
rx.rst_threshold("tile", f.lit(">"), f.lit(0.0)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(binary mask tile; pixels > 0.0 → 1, others → 0)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_threshold(col("tile"), lit(">"), lit(0.0)).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(binary mask tile; pixels > 0.0 → 1, others → 0)
Analysis
Higher-level analytical transforms wrapping single GDAL primitives — COG layout publishing, proximity surfaces, contour extraction, and viewshed analysis. See the Helios notebooks for a worked example of gbx_rst_cog_convert converting 3DEP DEMs to COGs and cataloging them in a STAC Delta table (NB03).
rst_cog_convert
LightweightHeavyweightRe-encodes the tile as a Cloud Optimized GeoTIFF (validated with cog_validate). compression defaults to "auto" — a size-adaptive ZSTD level with a dtype-matched predictor (the GeoBrix materialize baseline; see Materialized Compression) — or an explicit codec name (zstd, deflate, lzw, lerc, jpeg, webp, none). The tile's metadata.driver is GTiff (a COG is a valid GeoTIFF).
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_cog_convert(tile: Column, [compression: Column, blocksize: Column = lit(512), overviewResampling: Column = lit("AVERAGE")]): Column — Re-layout a raster tile as a Cloud Optimized GeoTIFF via gdal.Translate -of COG. Both tiers use a ZSTD + dtype-predictor baseline. compression accepts an explicit codec on either tier — ZSTD, DEFLATE, LZW, NONE, LERC, JPEG, WEBP — and its default is ZSTD on the heavyweight tier. The lightweight tier additionally accepts AUTO (its default): a size-adaptive ZSTD level with the dtype predictor. blocksize is the internal tile size in pixels (square). overviewResampling is the algorithm for the auto-generated overview pyramid. Output is a GTiff-on-disk variant suitable for HTTP range serving. See Materialized Compression for the codec/level details and when to override. See Materialized Compression for the codec/level details and when to override.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Convert to COG with ZSTD compression, 512-pixel blocks, AVERAGE overviews.
-- The lightweight tier also accepts 'AUTO' (size-adaptive ZSTD + dtype
-- predictor, the default); pass 'DEFLATE' for a portable hand-off file.
-- See the Materialized Compression page.
SELECT gbx_rst_cog_convert(tile, 'ZSTD', 512, 'AVERAGE') AS cog
FROM rasters;
+-----------------------------------------------------------+
|cog |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
from databricks.labs.gbx.pyrx import functions as rx
df = spark.table("rasters")
result = df.select(rx.rst_cog_convert("tile").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(COG tile; a COG is a valid GeoTIFF with tiled internal layout; light tier returns a materialized v2 Tile)
from databricks.labs.gbx.rasterx import functions as rx
tile_df = spark.table("rasters")
result = tile_df.select(rx.rst_cog_convert("tile").alias("tile")).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(COG tile; a COG is a valid GeoTIFF with tiled internal layout)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val rasters = spark.table("rasters")
val result = rasters.select(rx.rst_cog_convert(col("tile")).alias("tile"))
result.show()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(COG tile; a COG is a valid GeoTIFF with tiled internal layout)
rst_contour
LightweightHeavyweightPowered by scikit-image (measure.find_contours). Returns ARRAY<struct(geom_wkb, value)> of contour LineStrings (in the raster CRS) at each fixed level, or at base + k*interval across the data range when levels is empty; NoData is masked before tracing. The marching-squares line geometry differs slightly from the heavyweight GDAL contours.
Signature: rst_contour(tile: Column, levels: Column, [interval: Column = lit(0.0), base: Column = lit(0.0), attrField: Column = lit("elev")]): Column — Generate contour LineString features via gdal.ContourGenerateEx. Pass a non-empty levels ARRAY<DOUBLE> for fixed contour values, or pass array() and set interval (>0) for equal-step contours at base + n*interval. Returns ARRAY<struct(geom_wkb BINARY, value DOUBLE)> — one entry per contour line in the source raster's CRS.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Equal-interval contours every 50 m. Pass array() of fixed levels to override.
SELECT gbx_rst_contour(tile, array(), 50.0, 0.0, 'elev') AS contours
FROM dem_rasters;
+-----------------------------------------------------------------------------------------------------------------+
|contours |
+-----------------------------------------------------------------------------------------------------------------+
|[{[BINARY], 50.0}, {[BINARY], 100.0}, {[BINARY], 150.0}, {[BINARY], 200.0}, {[BINARY], 250.0}, {[BINARY], 300.0}]|
+-----------------------------------------------------------------------------------------------------------------+
(array of contour features: LineString geometry + elevation)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
df = dem_tile_df(spark)
result = df.select(
rx.rst_contour("tile", f.lit([]), f.lit(50.0), f.lit(0.0), f.lit("elev")).alias(
"contours"
)
).first()
+-----------------------------------------------------------------------------------------------------------------+
|contours |
+-----------------------------------------------------------------------------------------------------------------+
|[{[BINARY], 50.0}, {[BINARY], 100.0}, {[BINARY], 150.0}, {[BINARY], 200.0}, {[BINARY], 250.0}, {[BINARY], 300.0}]|
+-----------------------------------------------------------------------------------------------------------------+
(array of contour features: LineString geometry + elevation)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("dem_rasters")
result = df.select(
rx.rst_contour("tile", f.lit([]), f.lit(50.0), f.lit(0.0), f.lit("elev")).alias(
"contours"
)
).first()
+-----------------------------------------------------------------------------------------------------------------+
|contours |
+-----------------------------------------------------------------------------------------------------------------+
|[{[BINARY], 50.0}, {[BINARY], 100.0}, {[BINARY], 150.0}, {[BINARY], 200.0}, {[BINARY], 250.0}, {[BINARY], 300.0}]|
+-----------------------------------------------------------------------------------------------------------------+
(array of contour features: LineString geometry + elevation)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val dem = spark.table("dem_rasters")
val result = dem.select(rx.rst_contour(col("tile"), array(), lit(50.0), lit(0.0), lit("elev")).alias("contours"))
result.show(truncate = false)
+--------------------------------------+
|contours |
+--------------------------------------+
|[{[BINARY], 100.0}, {[BINARY], 200.0}]|
+--------------------------------------+
(array of contour features: LineString + elevation value)
rst_proximity
LightweightHeavyweightPowered by SciPy (scipy.ndimage.distance_transform_edt). Distance to the nearest source pixel (target_values, or any non-zero pixel by default), in GEO (CRS units) or PIXEL units; pixels beyond max_distance → NoData −1.0. Single-band Float32.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_proximity(tile: Column, [targetValues: Column = null, distUnits: Column = lit("GEO"), maxDistance: Column = null]): Column — Compute a Float32 raster where each pixel holds the distance to the nearest source pixel via gdal.ComputeProximity. targetValues is a comma-separated list of source-pixel values (e.g. "1,2,3"); null means any non-NoData pixel is a target. distUnits is "GEO" (CRS ground units, default) or "PIXEL". maxDistance caps the output; pixels beyond it get the NoData sentinel -1.0.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Distance in pixels to any non-NoData pixel; cap distances at 100 pixels.
SELECT gbx_rst_proximity(tile, '', 'PIXEL', cast(100.0 as double)) AS dist
FROM rasters;
+-----------------------------------------------------------+
|dist |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(distance in pixels to nearest non-NoData pixel, capped at 100)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
df = dem_tile_df(spark)
result = df.select(
rx.rst_proximity("tile", f.lit(""), f.lit("PIXEL"), f.lit(100.0)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(distance in pixels to nearest non-NoData pixel, capped at 100)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("dem_rasters")
result = df.select(
rx.rst_proximity("tile", f.lit(""), f.lit("PIXEL"), f.lit(100.0)).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(distance in pixels to nearest non-NoData pixel, capped at 100)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val dem = spark.table("dem_rasters")
val result = dem.select(rx.rst_proximity(col("tile"), lit(""), lit("PIXEL"), lit(100.0)).alias("distance"))
result.show(truncate = false)
+-----------------------------------------------------------+
|distance |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(distance in pixels to nearest non-NoData, capped at 100)
rst_viewshed
LightweightHeavyweightPowered by xarray-spatial (xrspatial.viewshed). Returns a Byte tile (255 visible / 0 invisible) from a DEM tile and a POINT observer. The CPU line-of-sight scan differs from the heavyweight GDAL sweep (no earth-curvature correction), but the binary visible/invisible classification matches. Pulls numba — the heaviest pyrx dependency.
The lightweight Python binding accepts three optional keyword arguments the SQL and heavyweight tiers do not — virtualize_dir, virtualize_prefix, and materialize — controlling whether the produced tile carries raster bytes or a bytes-free virtual reference. See Virtual-tile force-output params.
Signature: rst_viewshed(tile: Column, observerGeom: Column, observerHeight: Column, [targetHeight: Column = lit(1.6), maxDistance: Column = null]): Column — Compute a binary viewshed Byte raster (255 = visible, 0 = invisible / out-of-range) from a DEM tile and an observer POINT via gdal.ViewshedGenerate. observerGeom is WKB / WKT POINT; it is reprojected from its source CRS to the raster CRS (an EWKB/EWKT embedded SRID wins, else the optional crs argument, else assumed already aligned). Non-POINT geometries raise an error at execution time. Heights are above the DEM at each pixel. maxDistance clips the search radius; null = unlimited. See Coordinate Reference Systems.
- SQL
- Python (light)
- Python (heavy)
- Scala
-- Visibility from observer at (-73.5 40.5), eye 100 m, target 1.6 m, cap 5000 m.
SELECT gbx_rst_viewshed(tile, 'POINT(-73.5 40.5)', 100.0, 1.6, 5000.0) AS vs
FROM dem_rasters;
+-----------------------------------------------------------+
|vs |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(binary viewshed: 1=visible, 0=not visible)
from databricks.labs.gbx.pyrx import functions as rx
from pyspark.sql import functions as f
rx.register(spark)
df = dem_tile_df(spark)
result = df.select(
rx.rst_viewshed(
"tile",
f.lit("POINT(500320 4500320)"),
f.lit(100.0),
f.lit(1.6),
f.lit(500.0),
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(binary viewshed: 1=visible, 0=not visible)
from databricks.labs.gbx.rasterx import functions as rx
from pyspark.sql import functions as f
df = spark.table("dem_rasters")
result = df.select(
rx.rst_viewshed(
"tile",
f.lit("POINT(500320 4500320)"),
f.lit(100.0),
f.lit(1.6),
f.lit(500.0),
).alias("tile")
).first()
+-----------------------------------------------------------+
|tile |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(binary viewshed: 1=visible, 0=not visible)
import com.databricks.labs.gbx.rasterx.{functions => rx}
import org.apache.spark.sql.functions._
val dem = spark.table("dem_rasters")
val result = dem.select(
rx.rst_viewshed(col("tile"), lit("POINT(-73.5 40.5)"), lit(100.0), lit(1.6), lit(5000.0)).alias("viewshed")
)
result.show(truncate = false)
+-----------------------------------------------------------+
|viewshed |
+-----------------------------------------------------------+
|{0, <raster bytes>, <virtual path>, {driver -> GTiff, ...}}|
+-----------------------------------------------------------+
(binary visibility mask: 1=visible, 0=not visible)
Heavyweight-tier specifics
Every RasterX function runs in both execution tiers, so the per-function reference above applies regardless of tier. This section documents one heavyweight-tier execution detail: the GDAL VRT Python pixel-function configuration used by gbx_rst_combineavg / gbx_rst_derivedband (and their _agg forms).
VRT Python pixel functions
gbx_rst_combineavg, gbx_rst_combineavg_agg, gbx_rst_derivedband, and gbx_rst_derivedband_agg evaluate a Python expression on each pixel via GDAL's VRT Python pixel-function API. That API is gated behind the GDAL config option GDAL_VRT_ENABLE_PYTHON, which GeoBrix sets to NO at executor startup (see Security - Restrict GDAL drivers). When you call one of the four functions above, GeoBrix flips the option to YES for the duration of that call only — via the internal GDALManager.withVrtPython bracket — and restores NO immediately on return. You don't need to set anything on the cluster or in your notebook to use the built-in functions.
When you need to enable it yourself
If you're invoking the GDAL Python bindings (from osgeo import gdal) directly — outside the built-in RasterX functions — and you read a VRT that declares a <PixelFunctionLanguage>Python</...> band, you'll get an empty/null read unless you enable the option in the same process. Pick one of:
Python — programmatic, scoped to your read. Recommended in all cases. Mirrors what GeoBrix does internally, works for both driver-side pyspark.sql calls and inside mapPartitions / mapInPandas UDFs that load VRT-with-pyfunc via osgeo.gdal, and survives interleaving with GeoBrix built-in calls (each GeoBrix call resets the option to NO on exit, so re-set it on every read):
from osgeo import gdal
gdal.SetConfigOption("GDAL_VRT_ENABLE_PYTHON", "YES")
try:
ds = gdal.Open("/path/to/your/vrt-with-pixel-function.vrt")
arr = ds.GetRasterBand(1).ReadAsArray()
ds = None
finally:
gdal.SetConfigOption("GDAL_VRT_ENABLE_PYTHON", "NO")
Cluster env var — for Python-worker processes only. Setting spark.executorEnv.GDAL_VRT_ENABLE_PYTHON YES on the cluster works for Python UDF workers (a separate process from the JVM, where GDAL initializes from env vars). It does not help JVM-side reads — GeoBrix calls gdal.SetConfigOption("GDAL_VRT_ENABLE_PYTHON", "NO") at executor JVM startup, and SetConfigOption takes precedence over the env var. Prefer the programmatic form above unless you have a strong reason to globally enable.
Scala / JVM code. If you're writing custom Spark expressions that consume Python-pixel VRTs, wrap the read/translate in the same helper GeoBrix uses internally — it refcounts the option so concurrent tasks on the same executor JVM compose safely:
import com.databricks.labs.gbx.rasterx.gdal.GDALManager
val result = GDALManager.withVrtPython {
val ds = org.gdal.gdal.gdal.Open(vrtPath)
// ... GDAL reads / translates here see the Python pixel function ...
ds
}
Trusted-modules variant
GDAL also accepts GDAL_VRT_ENABLE_PYTHON=TRUSTED_MODULES plus a GDAL_VRT_PYTHON_TRUSTED_MODULES allowlist if you want pixel-function code restricted to specific Python module prefixes. GeoBrix uses the plain YES form because the pixel-function source is constructed in-process from trusted (geobrix-generated) strings, never from user-supplied VRT XML on disk. If your custom code path reads VRTs whose <PixelFunctionCode> originates from less-trusted sources, switch to the TRUSTED_MODULES form and allowlist only what you intend to load.
Escape hatches
When a raster operation isn't in the rst_* surface, drop down to rasterio/NumPy per tile (lightweight tier). These are Python-only helpers on databricks.labs.gbx.pyrx.functions — not SQL functions, and not registered as gbx_rst_*.
tile_to_numpy
from databricks.labs.gbx.pyrx.functions import tile_to_numpy
arr = tile_to_numpy(tile_or_bytes) # -> np.ndarray, shape (bands, rows, cols)
Read a tile's raster into a NumPy array (all bands). Accepts either a tile struct (a Row/dict with a raster field) or raw bytes. Useful when you've collected a tile to the driver, or inside your own UDF.
rst_apply
from pyspark.sql.types import DoubleType
from databricks.labs.gbx.pyrx.functions import rst_apply
df.select(
rst_apply("tile", lambda ds: float(ds.read(1).mean()), returnType=DoubleType()).alias("band1_mean")
)
Apply your own function to each tile's open rasterio dataset, returning one scalar per row. fn receives a rasterio DatasetReader; the return value must match returnType (default DoubleType(); any Spark DataType). A null/empty tile yields null. This is the "GeoBrix doesn't have function X — run my own rasterio per tile" path; it returns a scalar (raster→raster transforms are the domain of rst_mapalgebra / rst_derivedband).
For rendering tiles and building maps from results, see the Visualization (gbx.vizx) page.
Next Steps
- GridX Function Reference
- VectorX Function Reference
- PMTiles Function Reference — Aggregator (
gbx_pmtiles_agg) for publishing tile pyramids - PMTiles Writer — DataSource for streaming large pyramids to a single
.pmtilesfile - RasterX Readers
- Helios notebooks — worked end-to-end example: Web Mercator reprojection, XYZ pyramid generation, COG conversion, and terrain analytics packaged into PMTiles archives over San Francisco.