Skip to main content

Vector Writer

Since: lightweight vector_gbx v0.4.0 · (light-only — no heavyweight tier)

vector_gbx — pyogrio-backed, pure-Python DataSource V2 writer. It is the generic catch-all vector writer: pass any OGR-supported driver via driverName and it round-trips with the matching vector_gbx reader. For common formats, use the named writers instead:

Compute & scale
  • Large datasets: this writer assembles output on a single node (sequential driver merge); for very large outputs, use a classic cluster (high-memory Serverless does not help here) or switch to the scalable geojsonl_gbx / gpkg_gbx writers. See Choosing a writer for large datasets.
  • Tier & compute: the lightweight (*_gbx) writers need no JAR or init script and are the only option on Serverless, standard (shared), and ARM clusters. The heavyweight raster/PMTiles writers require a classic x86 cluster (JAR + GDAL init script). Your compute usually decides the tier — then data scale. See Benchmarking for timings and methodology.
Before you write
  • Register first — call register(spark) once before using any *_gbx format (see the Writers Overview).
  • Lightweight-only — this single-file writer isn't implemented in the heavyweight tier; for output in both tiers, use the sharded GeoJSONL writer (geojsonl_gbx lightweight / geojsonl heavyweight).
  • Input geometry — provide geometry as WKB (binary) or WKT (text), with the CRS in the companion *_srid / *_srid_proj columns. EWKB, EWKT, and GeoJSON-encoded geometry are not accepted. Writing from a Databricks GEOMETRY / GEOGRAPHY column? Export to WKB first with ST_AsBinary (or ST_AsText for WKT) — see the Schema section below — and avoid ST_GeomAsEWKB / ST_AsEWKT / ST_AsGeoJSON.

Options​

OptionDefaultBehavior
driverNamerequiredOGR driver name (e.g. "GeoJSON", "ESRI Shapefile"). Named writers preset this automatically.
modeoverwriteoverwrite only; append is rejected.
geometryTypeinferred from the dataOverride the OGR geometry type.
layerNamedriver defaultOutput layer name where supported.

Schema​

All the vector writers — the named shapefile_gbx / geojson_gbx / gpkg_gbx / file_gdb_gbx and this generic vector_gbx — share one column contract: a geometry column plus its CRS companions.

root
 |-- <geom>: binary (WKB)  or  string (WKT)   # the geometry
 |-- <geom>_srid: string        # REQUIRED — CRS authority code, e.g. "4326" ("0" if unknown)
 |-- <geom>_srid_proj: string   # optional — PROJ4 string, CRS fallback when srid is "0"
 |-- ...any other columns       # written as feature attributes

The geometry is the column that has a companion <geom>_srid. <geom>_srid is required — it supplies the CRS and identifies the geometry. The _srid / _srid_proj columns are CRS metadata and are not written as fields; every other column becomes a feature attribute.

There are three ways to satisfy this contract — pick whichever matches your data. These patterns apply to every vector writer (just swap the format name and any preset driverName).

(a) Point the writer at your columns — geomCol / sridCol / projCol​

If your frame already carries geometry + CRS under other names, name them with options — no renaming required:

(df  # e.g. columns: the_geom (WKB), epsg, crs_proj, name, population
 .write.format("vector_gbx").option("driverName", "GeoJSON")
 .option("geomCol", "the_geom")   # geometry column (WKB or WKT)
 .option("sridCol", "epsg")       # REQUIRED — CRS authority-code column
 .option("projCol", "crs_proj")   # optional — PROJ4 fallback column
 .mode("overwrite").save("/Volumes/cat/sch/vol/out"))

sridCol is required; projCol is optional. See Named Vector Formats.

(b) Or coerce to the convention column names​

Build the geom_0 / geom_0_srid / geom_0_srid_proj columns the *_gbx readers emit:

from pyspark.sql import functions as F

df.select(
    F.col("my_geom_wkb").alias("geom_0"),     # geometry as WKB (or WKT)
    F.lit("4326").alias("geom_0_srid"),       # CRS authority code
    F.lit("").alias("geom_0_srid_proj"),      # optional PROJ4 fallback
    "name", "population",                      # -> feature attributes
).write.format("vector_gbx").option("driverName", "GeoJSON").mode("overwrite").save("/Volumes/cat/sch/vol/out")

(c) From a Databricks GEOMETRY / GEOGRAPHY column​

Native spatial types aren't a writer input directly — export to WKB plus an SRID first with ST_AsWKB and ST_SRID. Avoid ST_GeomAsEWKB / ST_AsEWKT / ST_AsGeoJSON — extended and GeoJSON encodings are not accepted.

df.selectExpr(
    "ST_AsWKB(my_geom) AS geom_0",                      # GEOMETRY/GEOGRAPHY -> WKB
    "CAST(ST_SRID(my_geom) AS STRING) AS geom_0_srid",  # SRID -> string
    "'' AS geom_0_srid_proj",
    "name", "population",                               # -> feature attributes
).write.format("vector_gbx").option("driverName", "GeoJSON").mode("overwrite").save("/Volumes/cat/sch/vol/out")

See Databricks Spatial for the full ST function reference.

The written file round-trips with the matching *_gbx reader — reading it back yields (…attributes, geom_0, geom_0_srid, geom_0_srid_proj).

Output: the file is written with the OGR driver passed via driverName; attribute and geometry handling follow that driver's conventions.

How it scales​

Unlike a single-node pyogrio.write_* call that serializes one file on one machine, the vector_gbx writer runs as a Spark DataSource V2 two-phase write: each partition is written concurrently by its executor to a scratch fragment, then the driver merges the fragments into one output file. The merge is sequential and rename-free, so it is safe on FUSE-mounted cloud storage (Unity Catalog Volumes, DBFS). Repartition the input to control write parallelism.

Example​

# Generic lightweight vector writer (pyogrio; no JAR)
from databricks.labs.gbx.ds.register import register
register(spark)
src = f"{SAMPLE_DATA_BASE}/nyc/boroughs/nyc_boroughs.geojson"
df = spark.read.format("geojson_gbx").load(src)
out = "/tmp/nyc_boroughs.geojson"  # a single named output file
(df.write.format("vector_gbx")
   .option("driverName", "GeoJSON")
   .mode("overwrite").save(out))
back = spark.read.format("vector_gbx").option("driverName", "GeoJSON").load(out)
assert back.count() == df.count()

Typical pipeline: export a table to a single file​

Export a table of vector data to one file — no coalesce needed (the writer merges partitions into a single file on commit):

df = spark.table("main.geo.features")  # vector data in a (Delta) table
df.write.format("vector_gbx").option("driverName", "GeoJSON").mode("overwrite").save("/Volumes/main/geo/exports/features.geojson")

Each partition is written concurrently, then merged into one output file. See Benchmarking for light-vs-heavy export figures.

Common functions: used vs excluded​

See GBX Common Functions for the full catalog of shared file-access primitives. The table below shows which are active in this writer and which are not, and why.

Common capabilityUsed here?How / why
list_local_files (session-free enumeration)Not used by the writerWriters do not enumerate source files; they write a DataFrame's output partitions.
gbx_file_write / FILE-tier writeNot in the DataSourceThe DataSource writer (df.write.format("vector_gbx")) is session-less on Connect and commits via FUSE. FILE-tier writes use vector_file_write at the function layer instead.
gbx_file_read (FILE-tier read)Not in the DataSourceFILE-tier reads for vector data go through vector_file_read at the function layer.
Coordinate reference systems

The writer reads the CRS from the geom_0_srid column (or sridCol) and applies it to the OGR layer via srs.SetFromUserInput. The output srid and proj4 columns record what was written. An EPSG, ESRI, or OGC authority string is accepted; if srid is "0", the geom_0_srid_proj PROJ4 fallback is used. See Coordinate reference systems for the full CRS model.

Shared file-access layer

Lightweight writers commit via the shared file_gbx file-access base — file_mode, layout, and the no-gating rule are described there. See also file_gbx Writer for the write API.