Skip to main content

Vector Reader

Both tiers produce the same (attrs..., geom_0, geom_0_srid, geom_0_srid_proj) schema — see Choosing an Execution Tier.

Benchmark & tradeoff

The lightweight (*_gbx) and heavyweight readers emit the same schema, but your compute usually decides the tier: the lightweight tier needs no JAR or init script and is the only option on Serverless, standard (shared), and ARM clusters. The heavyweight tier requires a classic x86 cluster (JAR + GDAL init script); where it is available it uses native GDAL on the JVM and tends to pull ahead on large workloads. See the Benchmarking page for light-vs-heavy timings and methodology.

Available Formats​

Both tiers read any OGR/GDAL vector driver, including:

  • ESRI Shapefile (.shp), GeoJSON (.geojson, .json), GeoPackage (.gpkg), File Geodatabase (.gdb)
  • KML (.kml), GML (.gml), CSV with geometry (.csv), PostgreSQL/PostGIS, and 80+ more.
Format availability

Driver coverage varies by environment — some formats need extra GDAL drivers/packages installed.

Options​

Both tiers (lightweight vector_gbx, heavyweight ogr) take the same options; the named format readers preset driverName.

OptionDefaultDescription
driverNamerequired on vector_gbx; auto-detected from the extension on ogr; preset on named readersOGR driver name (e.g. GPKG, ESRI Shapefile, GeoJSON) — forces a specific driver regardless of the file extension.
asWKB"true"Output geometry as WKB (binary) vs WKT (text).
chunkSize"10000"Records per read batch (in-memory batching on the single per-file read — not partition splitting).
layerName""Layer name for multi-layer formats (overrides the layer index).
layerNumber / layerN"0"Layer index for multi-layer formats (0-based) — layerNumber (lightweight) / layerN (heavyweight).

Example — forcing the driver explicitly:

# Explicit driver (sample-data Volumes path)
df = spark.read.format("ogr") \
    .option("driverName", "GeoJSON") \
    .load("/Volumes/main/default/geobrix_samples/geobrix-examples/nyc/boroughs/nyc_boroughs.geojson")
df.show()
Example output
+--------------------+-----------+-----+
|geom_0              |geom_0_srid|...  |
+--------------------+-----------+-----+
|[BINARY]            |4326       |...  |
|...                 |...        |...  |
+--------------------+-----------+-----+

vector_gbx is the lightweight catch-all vector reader (pyogrio-backed, no JAR). It reads any OGR-supported format and emits the same schema as the heavyweight ogr reader.

# Lightweight generic vector reader (pyogrio; no JAR)
from databricks.labs.gbx.ds.register import register
register(spark)
df = spark.read.format("vector_gbx").load(SAMPLE)   # (attrs..., geom_0, geom_0_srid, geom_0_srid_proj)
df.show()

It is the lightweight counterpart of the heavyweight ogr reader, supporting Python and SQL bindings (not Scala).

Typical pipeline: ingest into a table​

The common pattern is to land vector files in a table for downstream analytics — on Databricks a managed table is Delta:

df = (spark.read.format("vector_gbx")
      .option("driverName", "GeoJSON")           # pass any OGR driver name
      .load("/Volumes/main/geo/raw/"))            # a folder of files
df.write.mode("overwrite").saveAsTable("main.geo.features")  # Delta table on Databricks

Reading a folder fans the files across the cluster (one partition per file), so ingest scales with the data — unlike a single-node pyogrio.read_* that parses one file on one machine. See Benchmarking for light-vs-heavy ingest figures.

Reading from a FILE-column table​

vector_file_read (light tier) accepts either a Volume path/directory or a FILE-column Delta table name. A FILE-column table stores one vector file per row — each FILE reference points to a complete vector file (e.g. a .gpkg). Reading decodes all features from each referenced file via pyogrio.

from databricks.labs.gbx.pyvx import vector_file_read

# Read all vector files referenced in a FILE-column table
features = vector_file_read(
    spark,
    "main.geo.vector_files",   # fully-qualified table name
    driver="GPKG",
    as_wkb=True,
)
features.show()

source_type="auto" (the default) distinguishes a table from a path: a string starting with /, matching a URI scheme, or having a known vector file extension is treated as a path; a dotted name, an extension-less string, or a non-existent path is treated as a table. Pass source_type="table" or source_type="path" to bypass the heuristic.

Auto-order default. vector_file_read auto-orders by the resolved source path before decoding. For a vector FILE table, one file maps to one row — every source opens exactly once regardless of row order. Auto-ordering produces a deterministic, cross-format-consistent default row sequence; it is not a throughput lever here (there are no multiple tiles per file to amortize). Pass skip_ordering=True when the table is already physically ordered or when you control ordering downstream:

features = vector_file_read(
    spark,
    "main.geo.vector_files",
    skip_ordering=True,    # preserve table/scan order; no additional sort step
)

Durable co-location: CLUSTER BY + OPTIMIZE​

Use layout="cluster" in vector_file_write (or gbx_file_write) to declare CLUSTER BY path in the table DDL. After a bulk insert, run OPTIMIZE to materialize the clustering — rows for the same source land in the same data files, which makes repeated reads and compaction-stable:

from databricks.labs.gbx.pyvx import vector_file_write

vector_file_write(
    spark,
    local_out="/tmp/roads.gpkg",
    target="main.geo.vector_files",
    layout="cluster",           # CLUSTER BY path in the DDL
    file_mode="auto",
)
-- After initial write or after bulk inserts, materialize clustering:
OPTIMIZE main.geo.vector_files;

Once the table has been optimized, pass skip_ordering=True when reading — the rows are already co-located, and skipping the sort avoids a redundant step.

layout="order" (the default) applies ORDER BY path at insert time and works well for tables built in a single write. Use layout="cluster" for tables updated incrementally and maintained with periodic OPTIMIZE.

Common functions: used vs excluded​

See GBX Common Functions for the full catalog of shared file-access primitives. The table below shows which are active in this reader and which are not, and why.

Common capabilityUsed here?How / why
list_local_files (session-free enumeration)UsedDirectory reads — recursive, include_hidden, extensions, and path_glob_filter are routed through this shared predicate for all format-specific readers.
gbx_file_read / FILE-tier readNot in the DataSourceThe DataSource is session-less on Connect; FILE-tier reads go through vector_file_read (function layer) which injects _file_ref on the driver before mapInPandas. vector_file_read also accepts a FILE-column Delta table name — see Reading from a FILE-column table.
gbx_file_write / FILE-tier writeNot in the DataSourceFILE-tier writes go through vector_file_write (function layer). The DataSource writer commits via FUSE.

Next Steps​

Shared file-access layer

Lightweight readers use the shared file_gbx file-access base for FILE / FUSE routing and enumeration — capability tiers, the no-gating rule, and layout options are described there.