Skip to main content

COG Reader

cog_gbx reads Cloud-Optimized GeoTIFFs (COGs) into the shared (source, tile) schema. Its defining feature is area-of-interest clipping: pass one or more clip polygons and the reader issues HTTP range-requests (or FUSE range-reads) that fetch only the bytes that intersect each AOI, skipping the rest of the file entirely.

This makes cog_gbx the natural read path after COG preparation with the cog_gbx writer — the prepare-then-read pipeline:

  1. List source files with file_gbx.
  2. Prepare — convert each to a master COG with the cog_gbx writer (internal tiling + overview levels baked in).
  3. Read — clip to any AOI with cog_gbx + clipPolygons, pulling only the required tiles and overview levels.

cog_gbx also reads a VRT mosaic: point it at a mosaic.vrt index (written by the cog_gbx writer's mosaic mode) and the reader expands the index into one virtual tile per member mini-COG, so rst_* functions process the whole mosaic per-tile. See Reading a VRT mosaic below and the VRT & Mosaics reference.

Lightweight only

cog_gbx is a pure-Python lightweight DataSource (no JAR). Register it with register(spark) before use.

Options​

OptionDefaultDescription
virtualTiles"true"Default. Emit bytes-free virtual tiles — each row carries the source path + pixel window instead of raster bytes; pixels are read lazily when an operation needs them (the ingest-OOM-dissolving default for the light tier). Set "false" to materialize raster bytes into each row. See Virtual Tiles.
clipPolygonsnoneArea(s) of interest: one WKT/EWKT string, or a JSON-array string for a list, e.g. '["POLYGON((...))","POLYGON((...))"]'. One tile per polygon whose envelope intersects the COG; only the intersecting blocks/overviews are fetched. Materialized tiles are pre-clipped (NoData outside the polygon); virtual tiles defer the clip. Mutually exclusive with windows.
windowsnonePixel window(s): JSON 4-int array "[col,row,w,h]", or a JSON array of them. Mutually exclusive with clipPolygons/tileSize.
clipCrsnoneCRS for clipPolygons lacking an embedded SRID (e.g. "EPSG:27700"). Precedence: embedded EWKB/EWKT SRID → clipCrs → the COG's CRS. The reader reprojects the polygon internally.
splitStrategy"none"Split large COGs: none (default — one tile per file), serverless, classic, or auto. Splitting respects the COG's internal tile grid.
sizeInMB"-1"Power-user override: set a positive value to pin the per-tile budget in MiB. -1 defers to splitStrategy.
tileSizenoneRegular fixed-size grid over the COG: "w,h" or a single "n" (square). One tile per cell; mutually exclusive with clipPolygons/windows. Materialized cells guarded to ~2 GB; virtual unguarded. See Raster Options.
overlapPercent0Overlap % between tileSize cells (tileSize-only). See Raster Options.
filterRegex".*"When loading a directory, keep files whose full path matches this regex.
Virtual tiles are the default

Light raster readers now emit virtual tiles by default (virtualTiles=true) — bytes-free (path, window) references that read pixels lazily. Previously the reader materialized raster bytes into every row. To restore materialized reads, pass .option("virtualTiles", "false"). A virtual tile passed to a heavyweight function must be materialized first — see Virtual Tiles.

Default is no-split

Both raster_gbx and cog_gbx default to splitStrategy=none — one tile per file. Splitting is opt-in. COG files prepared by the cog_gbx writer already carry internal tiling and overviews, so the reader can serve any AOI without splitting the source into partitions.

Register​

from databricks.labs.gbx.ds.register import register
register(spark)

Read a COG directory​

from databricks.labs.gbx.ds.register import register
register(spark)

df = spark.read.format("cog_gbx").load("/Volumes/main/.../cog-prepared/nyc-sentinel2")
df.show()

Read with AOI clip​

# Step 3 — windowed read: clip to an area of interest.
# clipPolygons takes a WKT/EWKT string (or, for a list, a JSON array string).
# clipCrs gives the CRS of polygons that don't carry an embedded SRID
# (precedence: embedded EWKB/EWKT SRID -> clipCrs -> raster CRS).
# One tile is emitted per polygon whose envelope intersects the raster.
aoi_wkt = "POLYGON((-74.05 40.65,-73.90 40.65,-73.90 40.80,-74.05 40.80,-74.05 40.65))"
cog_df = (
    spark.read.format("cog_gbx")
         .option("clipPolygons", aoi_wkt)   # NYC area (WGS84)
         .option("clipCrs", "EPSG:4326")
         .load(OUT)
)
cog_df.show()
# source | tile   (tile.raster is pre-clipped to the polygon; masked pixels are NoData)
# The reader issues range-reads that fetch only the intersecting blocks.

Output schema​

cog_gbx emits the standard raster schema:

root
 |-- source: string   — path to the source COG file
 |-- tile: struct
 |    |-- cellid: bigint (nullable)
 |    |-- raster: binary (nullable) — clipped tile bytes (GeoTIFF); null for virtual tiles
 |    |-- path: string (nullable) — source path (provenance / virtual read target)
 |    |-- window: struct<col_off,row_off,width,height> (nullable) — pixel window read
 |    |-- clip_polygon: binary (nullable) — the AOI applied (materialized) or to apply (virtual)
 |    |-- clip_crs: string (nullable) — CRS of clip_polygon
 |    |-- crs: string (nullable) — working/target CRS
 |    |-- metadata: map<string,string> — driver, CRS, extent, …

For a materialized tile the raster bytes carry the CRS and geotransform of the clipped window (not the full source file), and window/clip_polygon/clip_crs are provenance of what was applied. A virtual tile (from .option("virtualTiles", "true")) leaves raster null and carries those fields as a deferred instruction. Downstream rst_* functions consume either identically.

Full prepare-then-read pipeline​

# Step 1 — list source files.
from databricks.labs.gbx.ds.register import register
register(spark)

refs = spark.read.format("file_gbx").load(
    "/Volumes/main/geobrix_samples/geobrix-examples/nyc/sentinel2"
)

# Step 2 — convert each source file to a master COG.
import tempfile, os
OUT = "/Volumes/main/geobrix_samples/cog-prepared/nyc-sentinel2"

(
    refs.write.format("cog_gbx")
        .option("cogBlockSize", "512")
        .option("cogOverviewResampling", "AVERAGE")
        .option("cogCompression", "DEFLATE")
        .mode("overwrite")
        .save(OUT)
)
print("COGs written to", OUT)
# Step 3 — windowed read: clip to an area of interest.
# clipPolygons takes a WKT/EWKT string (or, for a list, a JSON array string).
# clipCrs gives the CRS of polygons that don't carry an embedded SRID
# (precedence: embedded EWKB/EWKT SRID -> clipCrs -> raster CRS).
# One tile is emitted per polygon whose envelope intersects the raster.
aoi_wkt = "POLYGON((-74.05 40.65,-73.90 40.65,-73.90 40.80,-74.05 40.80,-74.05 40.65))"
cog_df = (
    spark.read.format("cog_gbx")
         .option("clipPolygons", aoi_wkt)   # NYC area (WGS84)
         .option("clipCrs", "EPSG:4326")
         .load(OUT)
)
cog_df.show()
# source | tile   (tile.raster is pre-clipped to the polygon; masked pixels are NoData)
# The reader issues range-reads that fetch only the intersecting blocks.

Reading a VRT mosaic​

When the load path points directly at a mosaic.vrt (can be written by the cog_gbx writer's mosaic mode), the reader parses the VRT XML, enumerates its member paths, and emits one virtual tile row per member mini-COG — each row a whole-file reference (raster=NULL, path set to the member, window=NULL). All downstream rst_* functions then run per-tile, exactly as for a directory of flat COGs:

from databricks.labs.gbx.ds.register import register
register(spark)

# One virtual tile per member mini-COG
df = spark.read.format("cog_gbx").load("/Volumes/.../mosaic/mosaic.vrt")

Options that apply to a VRT load:

OptionEffect on a VRT load
.vrt path recognitionPointing the load at a mosaic.vrt triggers member expansion (one row per tile). Pointing at the containing directory instead walks the directory for raster files and skips .vrt indexes, so the tiles are read directly. Both yield one row per tile; the VRT path is the canonical way to load a mosaic.
clipPolygons + clipCrsRestrict the expansion to only the members whose extent intersects the area of interest — non-intersecting mini-COGs are skipped without being opened. clipCrs supplies the CRS for a polygon lacking an embedded SRID.
virtualTilesApplies as elsewhere (default "true"). Members are emitted as bytes-free virtual tiles; set "false" to materialize member bytes into each row.

Full detail — including mint_vrt for on-demand transient VRTs — is on the VRT & Mosaics page.

Common functions: used vs excluded​

See GBX Common Functions for the full catalog of shared file-access primitives. The table below shows which are active in this reader and which are not, and why.

Common capabilityUsed here?How / why
list_local_files (session-free enumeration)UsedAll directory reads — recursive, include_hidden, extensions, path_glob_filter options are routed through this shared predicate.
enumerate_files (FILE-tier enumeration)Not in the DataSourceThe DataSource is session-less on Connect and does not call enumerate_files; FILE-tier enumeration is only available through gbx_file_read at the function layer.
gbx_file_read / gbx_file_write (FILE tier)Not in the DataSourceThe DataSource is FUSE-only (session-less on Connect); FILE reads go through gbx_file_read → rst_fromfile at the function layer.
Shared file-access layer

Lightweight readers use the shared file_gbx file-access base for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there.

Next steps​