Skip to main content

Choosing a Format

Since: v0.6.0 (the format selection guide that used to sit on the Raster reader page, expanded)

Geospatial data arrives in a handful of shapes:

  • pictures of the ground (raster);
  • labelled multi-dimensional cubes (array);
  • features with geometry (vector);
  • pre-rendered map tiles (tile).

This page helps you pick the format to store, exchange and process each of them. For every format it shows the reader, the writer and the functions that work on it.

Working with Zarr? These pages cover it end to end:

One set of *_gbx readers and writers runs everywhere. They are pure Python, need no JAR and run on Serverless; see Surfaces & Levels.

The short version​

  • Start with GeoTIFF, and use COG when the files sit in cloud storage and are read in pieces.
  • Use Zarr for cubes that many jobs read slices of, or write results back to: weather, climate and other gridded data with a time dimension. GeoBrix makes Zarr stores transactional, versioned and safe for concurrent writers. See Zarr stores.
  • NetCDF, HDF5 and GRIB2 are what data providers publish. Read them where they are; move the data to Zarr or Delta when you need to write back or fan out at scale.
  • Keep vector data in Delta tables with GEOMETRY or GEOGRAPHY columns, and use the Databricks built-in ST_* functions. Use GeoPackage, Shapefile, GeoJSON and File Geodatabase to exchange data with other tools.
  • Use PMTiles to serve a tile pyramid from a single file in object storage.

Raster formats​

FormatGood forTiersReadWriteFunctions
GeoTIFF / COGImagery, elevation models, any single raster. COG adds internal tiling and overviews, so a reader fetches only the pieces it needs.Bothgtiff_gbx and cog_gbxgtiff_gbx and cog_gbxRasterX rst_*
JPEG 2000High-compression imagery.Boththe generic raster reader: raster_gbxthe generic raster writerRasterX rst_*

Array formats​

Array formats hold gridded data with more than two dimensions: time, level, band, latitude and longitude together.

FormatGood forTiersReadWriteFunctions
ZarrLarge cubes that many jobs read slices of in parallel and write back to, with versions, tags and concurrent writers.Python (no JAR)gbx_zarr_read, gbx_zarr_sample, and the zarr_gbx DataSource for plain storesgbx_zarr_writerst_* on Zarr tiles; the Zarr functions: gbx_zarr_register, gbx_zarr_sync, gbx_zarr_optimize, gbx_zarr_vacuum, gbx_zarr_tag, gbx_zarr_untag, gbx_zarr_history, gbx_zarr_restore, gbx_zarr_export
NetCDF / HDF5Self-describing scientific files from data providers.Bothnetcdf_gbx; HDF5 through the generic raster readernetcdf_gbxRasterX rst_*
GRIB2Weather model output, such as NOAA HRRR or ECMWF.Boththe generic raster readerRasterX rst_*

Vector formats​

FormatGood forTiersReadWriteFunctions
Delta or Parquet with GEOMETRY / GEOGRAPHYAnalytics at scale. The natural home for vector data in Databricks.Built inSpark, spark.read.tableSparkDatabricks ST_* and H3 functions; GeoBrix adds gbx_st_*
GeoPackageExchange of one or more layers in a single file.Bothgpkg_gbxgpkg_gbxgbx_st_*
ShapefileExchange with older tools.Bothshapefile_gbxshapefile_gbxgbx_st_*
File GeodatabaseEsri data.Bothfile_gdb_gbxfile_gdb_gbxgbx_st_*
GeoJSON / GeoJSONLWeb exchange. GeoJSONL (one feature per line) shards and streams.Bothgeojson_gbx and geojsonl_gbxgeojson_gbx; geojsonl_gbxgbx_st_*

Tile formats​

FormatGood forTiersReadWriteFunctions
PMTilesA raster or vector tile pyramid in one file that a web map reads straight from object storage.Bothpmtiles_gbxpmtiles_gbxgbx_pmtiles_agg, and gbx_st_asmvt for vector tiles

Point clouds and images with positions have their own readers. LiDAR reads LAS and LAZ, which feed the DSM and CHM functions. EXIF reads drone and camera images.

How the raster and array formats compare​

FormatSizeCompressionMulti-bandTime seriesCloud-friendly
GeoTIFFGoodGoodYesNoYes (COG)
NetCDFExcellentGoodYesYesModerate
HDF5ExcellentGoodYesYesPoor
GRIB2ExcellentExcellentYesYesPoor
JPEG 2000ExcellentExcellentYesNoModerate
ZarrExcellentGoodYesYesExcellent

"Cloud-friendly" means a reader can fetch just the part it needs from object storage. For how Zarr compares in more detail, including writes and versions, see Why Zarr.

Delta-indexed collections​

Several GeoBrix tools keep a Delta table as the index of a collection of files. Two examples:

  • a table with a FILE column that points at rasters or vectors on a Volume;
  • the manifest table of a Zarr store, which points at its chunks and shards.

Both follow one pattern. The bytes stay in the files. The table says which of them make up the collection, at which version. You can query and govern the collection with SQL, and read the index as it was at an earlier version. See Delta-indexed collections.