Choosing a Format
Since: v0.6.0 (the format selection guide that used to sit on the Raster reader page, expanded)
Geospatial data arrives in a handful of shapes:
- pictures of the ground (raster);
- labelled multi-dimensional cubes (array);
- features with geometry (vector);
- pre-rendered map tiles (tile).
This page helps you pick the format to store, exchange and process each of them. For every format it shows the reader, the writer and the functions that work on it.
Working with Zarr? These pages cover it end to end:
One set of *_gbx readers and writers runs everywhere. They are pure Python, need no JAR and run
on Serverless; see Surfaces & Levels.
The short version
- Start with GeoTIFF, and use COG when the files sit in cloud storage and are read in pieces.
- Use Zarr for cubes that many jobs read slices of, or write results back to: weather, climate and other gridded data with a time dimension. GeoBrix makes Zarr stores transactional, versioned and safe for concurrent writers. See Zarr stores.
- NetCDF, HDF5 and GRIB2 are what data providers publish. Read them where they are; move the data to Zarr or Delta when you need to write back or fan out at scale.
- Keep vector data in Delta tables with
GEOMETRYorGEOGRAPHYcolumns, and use the Databricks built-inST_*functions. Use GeoPackage, Shapefile, GeoJSON and File Geodatabase to exchange data with other tools. - Use PMTiles to serve a tile pyramid from a single file in object storage.
Raster formats
| Format | Good for | Tiers | Read | Write | Functions |
|---|---|---|---|---|---|
| GeoTIFF / COG | Imagery, elevation models, any single raster. COG adds internal tiling and overviews, so a reader fetches only the pieces it needs. | Both | gtiff_gbx and cog_gbx | gtiff_gbx and cog_gbx | RasterX rst_* |
| JPEG 2000 | High-compression imagery. | Both | the generic raster reader: raster_gbx | the generic raster writer | RasterX rst_* |
Array formats
Array formats hold gridded data with more than two dimensions: time, level, band, latitude and longitude together.
| Format | Good for | Tiers | Read | Write | Functions |
|---|---|---|---|---|---|
| Zarr | Large cubes that many jobs read slices of in parallel and write back to, with versions, tags and concurrent writers. | Python (no JAR) | gbx_zarr_read, gbx_zarr_sample, and the zarr_gbx DataSource for plain stores | gbx_zarr_write | rst_* on Zarr tiles; the Zarr functions: gbx_zarr_register, gbx_zarr_sync, gbx_zarr_optimize, gbx_zarr_vacuum, gbx_zarr_tag, gbx_zarr_untag, gbx_zarr_history, gbx_zarr_restore, gbx_zarr_export |
| NetCDF / HDF5 | Self-describing scientific files from data providers. | Both | netcdf_gbx; HDF5 through the generic raster reader | netcdf_gbx | RasterX rst_* |
| GRIB2 | Weather model output, such as NOAA HRRR or ECMWF. | Both | the generic raster reader | RasterX rst_* |
Vector formats
| Format | Good for | Tiers | Read | Write | Functions |
|---|---|---|---|---|---|
Delta or Parquet with GEOMETRY / GEOGRAPHY | Analytics at scale. The natural home for vector data in Databricks. | Built in | Spark, spark.read.table | Spark | Databricks ST_* and H3 functions; GeoBrix adds gbx_st_* |
| GeoPackage | Exchange of one or more layers in a single file. | Both | gpkg_gbx | gpkg_gbx | gbx_st_* |
| Shapefile | Exchange with older tools. | Both | shapefile_gbx | shapefile_gbx | gbx_st_* |
| File Geodatabase | Esri data. | Both | file_gdb_gbx | file_gdb_gbx | gbx_st_* |
| GeoJSON / GeoJSONL | Web exchange. GeoJSONL (one feature per line) shards and streams. | Both | geojson_gbx and geojsonl_gbx | geojson_gbx; geojsonl_gbx | gbx_st_* |
Tile formats
| Format | Good for | Tiers | Read | Write | Functions |
|---|---|---|---|---|---|
| PMTiles | A raster or vector tile pyramid in one file that a web map reads straight from object storage. | Both | pmtiles_gbx | pmtiles_gbx | gbx_pmtiles_agg, and gbx_st_asmvt for vector tiles |
Point clouds and images with positions have their own readers. LiDAR reads LAS and LAZ, which feed the DSM and CHM functions. EXIF reads drone and camera images.
How the raster and array formats compare
| Format | Size | Compression | Multi-band | Time series | Cloud-friendly |
|---|---|---|---|---|---|
| GeoTIFF | Good | Good | Yes | No | Yes (COG) |
| NetCDF | Excellent | Good | Yes | Yes | Moderate |
| HDF5 | Excellent | Good | Yes | Yes | Poor |
| GRIB2 | Excellent | Excellent | Yes | Yes | Poor |
| JPEG 2000 | Excellent | Excellent | Yes | No | Moderate |
| Zarr | Excellent | Good | Yes | Yes | Excellent |
"Cloud-friendly" means a reader can fetch just the part it needs from object storage. For how Zarr compares in more detail, including writes and versions, see Why Zarr.
Delta-indexed collections
Several GeoBrix tools keep a Delta table as the index of a collection of files. Two examples:
- a table with a
FILEcolumn that points at rasters or vectors on a Volume; - the manifest table of a Zarr store, which points at its chunks and shards.
Both follow one pattern. The bytes stay in the files. The table says which of them make up the collection, at which version. You can query and govern the collection with SQL, and read the index as it was at an earlier version. See Delta-indexed collections.