Mosaic is an extension to the Apache Spark framework for fast + easy processing of very large geospatial datasets. It provides:

[1] The choice of a Scala, SQL and Python language bindings (written in Scala).

[2] Raster and Vector APIs.

[3] Easy conversion between common spatial data encodings (WKT, WKB and GeoJSON).

[4] Constructors to easily generate new geometries from Spark native data types.

[5] Many of the OGC SQL standard ST_ functions implemented as Spark Expressions for transforming,

aggregating and joining spatial datasets.

[6] High performance through implementation of Spark code generation within the core Mosaic functions.

[7] Performing point-in-polygon joins using an approach we co-developed with Ordnance Survey (blog post).

Note

We recommend using Databricks Runtime with Photon enabled to leverage the Databricks H3 expressions.

Version 0.4.x Series

Warning

For Mosaic <= 0.4.1 %pip install databricks-mosaic will no longer install “as-is” in DBRs due to the fact that Mosaic left geopandas unpinned in those versions. With geopandas 0.14.4, numpy dependency conflicts with the limits of scikit-learn in DBRs. The workaround is %pip install geopandas==0.14.3 databricks-mosaic. Mosaic 0.4.2+ limits the geopandas version.

Mosaic 0.4.x series only supports DBR 13.x DBRs. If running on a different DBR it will throw an exception:

DEPRECATION ERROR: Mosaic v0.4.x series only supports Databricks Runtime 13. You can specify %pip install 'databricks-mosaic<0.4,>=0.3' for DBR < 13.

Mosaic 0.4.x series issues an ERROR on standard, non-Photon clusters ADB | AWS | GCP:

DEPRECATION ERROR: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic 0.4.x series restricts executing this cluster.

As of Mosaic 0.4.0 / DBR 13.3 LTS (subject to change in follow-on releases):

• Assigned Clusters

  • Mosaic Python, SQL, R, and Scala APIs.

• Shared Access Clusters

  • Mosaic Scala API (JVM) with Admin allowlisting.

  • Mosaic Python bindings (to Mosaic Scala APIs) are blocked by Py4J Security on Shared Access Clusters.

  • Mosaic SQL expressions cannot yet be registered due to Unity Catalog. API changes, more here.

Note

Mosaic is a custom JVM library that extends spark, which has the following implications in DBR 13.3 LTS:

• Unity Catalog enforces process isolation which is difficult to accomplish with custom JVM libraries; as such only built-in (aka platform provided) JVM APIs can be invoked from other supported languages in Shared Access Clusters.

• Clusters can read Volumes via relevant built-in (aka platform provided) readers and writers or via custom python calls which do not involve any custom JVM code.

Version 0.3.x Series

We recommend using Databricks Runtime versions 12.2 LTS with Photon enabled. For Mosaic versions < 0.4.0 please use the 0.3.x docs.

As of the 0.3.11 release, Mosaic issues the following WARNING when initialized on a cluster that is neither Photon Runtime nor Databricks Runtime ML ADB | AWS | GCP:

DEPRECATION WARNING: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic will stop working on this cluster after v0.3.x.

If you are receiving this warning in v0.3.11+, you will want to begin to plan for a supported runtime. The reason we are making this change is that we are streamlining Mosaic internals to be more aligned with future product APIs which are powered by Photon. Along this direction of change, Mosaic has standardized to JTS as its default and supported Vector Geometry Provider.

Note

For Mosaic versions < 0.4 please use the 0.3 docs.

Documentation

Contents:

• API Documentation
  • Vector Format Readers
  • Raster Format Readers
  • Geometry constructors
  • Geometry accessors
  • Spatial functions
  • Spatial grid indexing
  • Spatial predicates
  • Spatial aggregation functions
  • Raster functions
  • Rasterio + GDAL UDFs
• Usage
  • Installation guide
  • GDAL Installation guide
  • Using grid index systems in Mosaic
  • BNG - British National Grid
  • Quickstart notebook
  • Kepler visualizations
  • Automatic SQL registration
  • Checkpointing for raster operations
• Models
  • Spatial K Nearest Neighbours
• Mosaic Videos and Talks

Indices and tables

• Index

• Search Page

Project Support

Please note that all projects in the databrickslabs github space are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects.

Any issues discovered through the use of this project should be filed as GitHub Issues on the Repo. They will be reviewed as time permits, but there are no formal SLAs for support.