Skip to main content

Vector Writer

vector_gbx — pyogrio-backed, pure-Python DataSource V2 writer. It is the generic catch-all vector writer: pass any OGR-supported driver via driverName and it round-trips with the matching vector_gbx reader. For common formats, use the named writers instead:

Compute & scale
  • Large datasets: this writer assembles output on a single node (sequential driver merge); for very large outputs, use a classic cluster (high-memory Serverless does not help here) or switch to the scalable geojsonl_gbx / gpkg_gbx writers. See Choosing a writer for large datasets.
  • Tier & compute: the lightweight (*_gbx) writers need no JAR or init script and are the only option on Serverless, standard (shared), and ARM clusters. The heavyweight raster/PMTiles writers require a classic x86 cluster (JAR + GDAL init script). Your compute usually decides the tier — then data scale. See Benchmarking for timings and methodology.
Before you write
  • Register first — call register(spark) once before using any *_gbx format (see the Writers Overview).
  • Lightweight-only — this single-file writer isn't implemented in the heavyweight tier; for output in both tiers, use the sharded GeoJSONL writer (geojsonl_gbx lightweight / geojsonl heavyweight).
  • Input geometry — provide geometry as WKB (binary) or WKT (text), with the CRS in the companion *_srid / *_srid_proj columns. EWKB, EWKT, and GeoJSON-encoded geometry are not accepted. Writing from a Databricks GEOMETRY / GEOGRAPHY column? Export to WKB first with ST_AsBinary (or ST_AsText for WKT) — see the Schema section below — and avoid ST_GeomAsEWKB / ST_AsEWKT / ST_AsGeoJSON.

Options​

OptionDefaultBehavior
driverNamerequiredOGR driver name (e.g. "GeoJSON", "ESRI Shapefile"). Named writers preset this automatically.
modeoverwriteoverwrite only; append is rejected.
geometryTypeinferred from the dataOverride the OGR geometry type.
layerNamedriver defaultOutput layer name where supported.

Schema​

All the vector writers — the named shapefile_gbx / geojson_gbx / gpkg_gbx / file_gdb_gbx and this generic vector_gbx — share one column contract: a geometry column plus its CRS companions.

root
 |-- <geom>: binary (WKB)  or  string (WKT)   # the geometry
 |-- <geom>_srid: string        # REQUIRED — CRS authority code, e.g. "4326" ("0" if unknown)
 |-- <geom>_srid_proj: string   # optional — PROJ4 string, CRS fallback when srid is "0"
 |-- ...any other columns       # written as feature attributes

The geometry is the column that has a companion <geom>_srid. <geom>_srid is required — it supplies the CRS and identifies the geometry. The _srid / _srid_proj columns are CRS metadata and are not written as fields; every other column becomes a feature attribute.

There are three ways to satisfy this contract — pick whichever matches your data. These patterns apply to every vector writer (just swap the format name and any preset driverName).

(a) Point the writer at your columns — geomCol / sridCol / projCol​

If your frame already carries geometry + CRS under other names, name them with options — no renaming required:

(df  # e.g. columns: the_geom (WKB), epsg, crs_proj, name, population
 .write.format("vector_gbx").option("driverName", "GeoJSON")
 .option("geomCol", "the_geom")   # geometry column (WKB or WKT)
 .option("sridCol", "epsg")       # REQUIRED — CRS authority-code column
 .option("projCol", "crs_proj")   # optional — PROJ4 fallback column
 .mode("overwrite").save("/Volumes/cat/sch/vol/out"))

sridCol is required; projCol is optional. See Named Vector Formats.

(b) Or coerce to the convention column names​

Build the geom_0 / geom_0_srid / geom_0_srid_proj columns the *_gbx readers emit:

from pyspark.sql import functions as F

df.select(
    F.col("my_geom_wkb").alias("geom_0"),     # geometry as WKB (or WKT)
    F.lit("4326").alias("geom_0_srid"),       # CRS authority code
    F.lit("").alias("geom_0_srid_proj"),      # optional PROJ4 fallback
    "name", "population",                      # -> feature attributes
).write.format("vector_gbx").option("driverName", "GeoJSON").mode("overwrite").save("/Volumes/cat/sch/vol/out")

(c) From a Databricks GEOMETRY / GEOGRAPHY column​

Native spatial types aren't a writer input directly — export to WKB plus an SRID first with ST_AsWKB and ST_SRID. Avoid ST_GeomAsEWKB / ST_AsEWKT / ST_AsGeoJSON — extended and GeoJSON encodings are not accepted.

df.selectExpr(
    "ST_AsWKB(my_geom) AS geom_0",                      # GEOMETRY/GEOGRAPHY -> WKB
    "CAST(ST_SRID(my_geom) AS STRING) AS geom_0_srid",  # SRID -> string
    "'' AS geom_0_srid_proj",
    "name", "population",                               # -> feature attributes
).write.format("vector_gbx").option("driverName", "GeoJSON").mode("overwrite").save("/Volumes/cat/sch/vol/out")

See Databricks Spatial for the full ST function reference.

The written file round-trips with the matching *_gbx reader — reading it back yields (…attributes, geom_0, geom_0_srid, geom_0_srid_proj).

Output: the file is written with the OGR driver passed via driverName; attribute and geometry handling follow that driver's conventions.

How it scales​

Unlike a single-node pyogrio.write_* call that serializes one file on one machine, the vector_gbx writer runs as a Spark DataSource V2 two-phase write: each partition is written concurrently by its executor to a scratch fragment, then the driver merges the fragments into one output file. The merge is sequential and rename-free, so it is safe on FUSE-mounted cloud storage (Unity Catalog Volumes, DBFS). Repartition the input to control write parallelism.

Example​

# Generic lightweight vector writer (pyogrio; no JAR)
from databricks.labs.gbx.ds.register import register
register(spark)
src = f"{SAMPLE_DATA_BASE}/nyc/boroughs/nyc_boroughs.geojson"
df = spark.read.format("geojson_gbx").load(src)
out = "/tmp/nyc_boroughs.geojson"  # a single named output file
(df.write.format("vector_gbx")
   .option("driverName", "GeoJSON")
   .mode("overwrite").save(out))
back = spark.read.format("vector_gbx").option("driverName", "GeoJSON").load(out)
assert back.count() == df.count()

Typical pipeline: export a table to a single file​

Export a table of vector data to one file — no coalesce needed (the writer merges partitions into a single file on commit):

df = spark.table("main.geo.features")  # vector data in a (Delta) table
df.write.format("vector_gbx").option("driverName", "GeoJSON").mode("overwrite").save("/Volumes/main/geo/exports/features.geojson")

Each partition is written concurrently, then merged into one output file. See Benchmarking for light-vs-heavy export figures.