read_paths and reference
Since: v0.6.0
read_paths is the building block of the Ingestion Pipelines. It takes a DataFrame
of file paths (a batch listing or an Auto Loader stream) and reads every file with the GeoBrix
reader you name. Use it directly in notebooks and jobs. declare_ingest and the
Lakeflow connector read files with the same code, so the
Reference at the end of this page applies to all three.
read_paths is pure Python: no JAR and no register(spark) (it calls the
reader classes directly). It runs on Serverless, standard (shared) and classic compute.

Read a list of files
Each row of the input DataFrame names one file. Project an Auto Loader or binaryFile listing down
to path and modificationTime (the file bytes are never loaded) and pass it to read_paths
together with the reader name:
from databricks.labs.gbx.ds.paths import read_paths
files = (spark.read.format("binaryFile")
.option("pathGlobFilter", "*.geojson")
.load("/Volumes/main/default/geobrix_samples/geobrix-examples/nyc/boroughs")
.select("path", "modificationTime")) # file bytes are never loaded
df = read_paths(files, "geojson_gbx")
df.select("boroname", "_source_path").show()
+-------------+--------------------+
| boroname| _source_path|
+-------------+--------------------+
|Staten Island|/Volumes/main/def...|
| Bronx|/Volumes/main/def...|
| ...| ...|
+-------------+--------------------+
read_paths(df, format, options=None, *, path_col="path", mtime_col="modificationTime",
schema_path=None, num_partitions=None, lineage=True, on_error="fail")
| Parameter | Default | Description |
|---|---|---|
df | required | DataFrame with one row per file. Batch or streaming. |
format | required | Reader name from the formats table. |
options | None | Dict of reader options, exactly as on that reader's page (for example layerName, chunkSize or tileSize). |
path_col | "path" | Column holding the file path, a Unity Catalog Volume path (see Paths). dbfs: and file: prefixes are accepted. |
mtime_col | "modificationTime" | Column copied to _source_mtime. Pass None to leave _source_mtime NULL. |
schema_path | None | The file whose columns define the output schema, or a folder: then only its first file (in path order) matching the format's default file pattern is opened. For a batch DataFrame the default is the first path in df. Required for a streaming DataFrame. |
num_partitions | None | Spread the work across this many tasks. See Parallelism. |
lineage | True | Set False to leave out _source_path and _source_mtime. |
on_error | "fail" | "fail": an error fails the read, naming the file. "skip": what cannot be read is skipped with a warning in the executor logs, and everything else is read. See Unreadable files. |
Stream new files
Pass an Auto Loader stream the same way. Because a stream has no first row to look at, give
schema_path a file or a folder. For a folder, the output schema comes from its first file whose
name matches the format's default file pattern. That pattern, not the
stream's pathGlobFilter, picks the file, so point schema_path at a file to choose it exactly.
from databricks.labs.gbx.ds.paths import read_paths
landing = "/Volumes/main/geo/landing/boroughs"
files = (spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "binaryFile")
.option("pathGlobFilter", "*.geojson")
.load(landing)
.select("path", "modificationTime"))
rows = read_paths(files, "geojson_gbx", schema_path=landing) # required for a stream
(rows.writeStream
.option("checkpointLocation", "/Volumes/main/geo/checkpoints/boroughs")
.trigger(availableNow=True)
.toTable("main.geo.boroughs"))
Auto Loader's checkpoint remembers which files were delivered, so each run reads only the files
that arrived since the last one. Auto Loader (cloudFiles) is a Databricks feature. In a Lakeflow
pipeline, declare_ingest declares this stream for you, together with what
happens when a file is overwritten.
Rasters
Raster files (raster_gbx, gtiff_gbx and cog_gbx) are planned into their tile windows first.
The windows are read as separate tasks only when num_partitions is set; without it, a large
raster is read by a single task. Set num_partitions (on Serverless, always) and use the raster
reader's tileSize option to choose the window size:
from databricks.labs.gbx.ds.paths import read_paths
files = (spark.read.format("binaryFile")
.option("pathGlobFilter", "*.tif")
.load("/Volumes/main/default/geobrix_samples/geobrix-examples/nyc/sentinel2")
.select("path", "modificationTime"))
tiles = read_paths(
files, "gtiff_gbx",
options={"tileSize": "512"}, # cut each raster into 512 x 512 pixel windows
num_partitions=8, # spread the windows across 8 tasks
)
tiles.select("tile.window", "_source_path").show(3, truncate=False)
+---------------------+----------------------------------+
|window |_source_path |
+---------------------+----------------------------------+
|{0, 0, 512, 512} |/Volumes/.../nyc_sentinel2_red.tif|
|{1536, 0, 512, 512} |/Volumes/.../nyc_sentinel2_red.tif|
|... |... |
+---------------------+----------------------------------+
By default each raster file is a single whole-file tile; tileSize cuts it into a grid of windows
(see the Raster Reader for its options). The raster manifest and
tilesTable options are rejected by read_paths: each input row already names one file, and a
manifest would be read again for every row. Read a manifest with spark.read.format("gtiff_gbx")
instead, and a tile index with read_tiles_table. A .vrt
source reports the .vrt file itself as its _source_path.
PMTiles inputs are planned the same way, in groups of tilesPerPartition tiles, for an archive
(source="archive") and for a GeoTIFF it renders (source="raster"). A NetCDF file is read whole
by one task in both modes. The reader reads all of a file's variables and slices in one
pass.
Reference
Everything below applies to every route: the Lakeflow connector,
Auto Loader pipelines and read_paths.
Supported formats
read_paths and declare_ingest take the name of a *_gbx reader. The _gbx suffix is
optional and the name is case-insensitive; the connector's format table option takes the same
names.
format | Reader | One input file becomes |
|---|---|---|
vector_gbx | Vector | Features (generic OGR reader) |
shapefile_gbx | Shapefile | Features |
geojson_gbx | GeoJSON | Features |
geojsonl_gbx | GeoJSONL | Features |
gpkg_gbx | GeoPackage | Features |
file_gdb_gbx | File Geodatabase | Features |
lidar_gbx | LiDAR | Points |
exif_gbx | EXIF/GPS | Image metadata |
raster_gbx, gtiff_gbx | Raster, GeoTIFF | Raster tiles |
cog_gbx | COG | Raster tiles, split into tile windows the same way as raster_gbx (see Rasters). The Lakeflow connector maps a .tif file to gtiff_gbx; set format: cog there to use the COG reader. |
netcdf_gbx | NetCDF | Set by the mode option: "raster" (default) gives one (source, tile) GeoTIFF tile per grid variable (and per slice with fanout); "vector" gives one point feature per observation or grid cell. |
pmtiles_gbx | PMTiles | Set by the source option, which is required here: "archive" gives one (z, x, y, bytes) row per tile stored in a .pmtiles file; "raster" renders (z, x, y, bytes) XYZ tiles from a GeoTIFF. Each file is rendered on its own, so a tile that covers two files appears once for each. |
file_gbx is not supported. It only lists files, and the binaryFile or Auto Loader listing that
feeds read_paths already gives each file's path, size and modification time.
Lineage columns
Each row carries the columns that reader normally returns, plus two lineage columns:
| Column | Type | Meaning |
|---|---|---|
_source_path | STRING | Bare /Volumes/... path of the file the row came from |
_source_mtime | TIMESTAMP | That file's modification time |
Default file patterns
Auto Loader delivers every file under a folder separately. The parts of a multi-file dataset (a
shapefile's .shx, .dbf and .prj sidecars, or the files inside a .gdb folder) would otherwise
be read as inputs of their own. When its glob is not set, declare_ingest passes the format's
pattern to Auto Loader. The same pattern picks the schema file from a folder schema_path, in
read_paths too. (The connector picks a table's files by extension instead; see its
discovery rules.)
format | Default glob |
|---|---|
vector_gbx | *.{gpkg,geojson,geojsonl,geojsons,shp,shz,shp.zip,gdb.zip,GPKG,GEOJSON,GEOJSONL,GEOJSONS,SHP,SHZ,SHP.ZIP,GDB.ZIP} |
shapefile_gbx | *.{shp,shz,shp.zip,SHP,SHZ,SHP.ZIP} |
geojson_gbx | *.{geojson,GEOJSON} |
geojsonl_gbx | *.{geojsonl,geojsons,GEOJSONL,GEOJSONS} |
gpkg_gbx | *.{gpkg,GPKG} |
file_gdb_gbx | *.{gdb.zip,GDB.ZIP} |
lidar_gbx | *.{las,laz,LAS,LAZ} |
exif_gbx | *.{jpg,jpeg,JPG,JPEG} |
raster_gbx, gtiff_gbx, cog_gbx | *.{tif,tiff,TIF,TIFF} |
netcdf_gbx | *.{nc,NC} |
pmtiles_gbx with source="archive" | *.{pmtiles,PMTILES} |
pmtiles_gbx with source="raster" | *.tif |
The patterns match the file name as written, so they cover lower- and upper-case extensions
(photo.jpg and PHOTO.JPG) but not mixed case such as .Jpg. Set glob for other names, for
example glob="*.jp2" for another raster format or glob="*.{nc,nc4}" for NetCDF-4 files named
.nc4. Deliver a File Geodatabase as a zipped .gdb.zip file. The pmtiles_gbx raster pattern is
*.tif only because that reader's filterRegex option keeps .tif files by default; to render
other files, set both filterRegex and glob.
GeoJSON is matched strictly by its .geojson extension, because many .json files are not
GeoJSON. For GeoJSON files named .json, pass glob="*.{geojson,json}".
A glob you set replaces the default pattern rather than narrowing it, so it must still leave out
the other files of a multi-file dataset. For shapefiles, end it in the data file's extension:
glob="zones_*.shp" delivers only the .shp files, while glob="zones_*" also delivers each
shapefile's .dbf, .shx and .prj sidecars as inputs of their own. Patterns use Spark
pathGlobFilter syntax: case-sensitive, with {a,b} groups, and a backslash escapes a pattern
character (\* matches a literal *).
Schema
The table schema comes from one file: schema_path, or for a folder its first file matching the
format's default file pattern (or glob). Files and folders whose names
start with _ or . are skipped, as Spark's file sources skip them. For file_gdb_gbx and
vector_gbx, an unzipped .gdb folder inside that folder counts as one file. No other file is
opened to resolve it, so a folder that already holds files with other columns still resolves.
Every file that is read must have the same columns with the same types. A file that differs by
column name or type fails with an error that names the file. A file whose columns are in a
different order is realigned by name. A file with no features adds no rows.
Paths
Paths are Unity Catalog Volume paths: /Volumes/..., or dbfs:/Volumes/... as Auto Loader and
binaryFile report them. The readers open each file through the Volume's file system. An
object-store URI (s3://, abfss://, gs://) or a GDAL virtual path (/vsis3/...,
/vsizip/..., /vsimem/...) therefore fails with an error that names it. It is not skipped as
missing. To ingest from cloud storage, create an external volume over the location and use its
/Volumes path.
Unreadable files
What happens to a file, or a part of one, that cannot be read (truncated, corrupt, or not the
format its name says) depends on on_error:
on_error="fail", the default ofread_paths. The Lakeflow connector always reads this way. A read error fails the read, with an error that names the file. A few readers handle some damage themselves, as the second column of the table below says.on_error="skip"(read_paths) andon_error="quarantine"(the default ofdeclare_ingest). Each file is read on its own, and so is each raster window or group of PMTiles tiles. What cannot be read is a bad record:read_pathsskips it with a warning in the executor logs, anddeclare_ingestwrites it to<t>_bad_records. Everything else is read, including the rows of a file read before an error in it. Errors of the environment are not bad data, so they still fail the read. These are out of memory, a missing module, an I/O error of the Volume (retried first), and an image too large forexif_gbxqcmode.
With skip or quarantine, a file that cannot be opened is one bad record for the whole file.
The same holds for a file whose columns differ from the table schema (see Schema). Smaller bad records depend on
the format:
format | With on_error="fail" | Also set aside with skip / quarantine |
|---|---|---|
vector_gbx, shapefile_gbx, file_gdb_gbx | Fails the read with the reader's error. | A feature (feature=<n>) whose geometry bytes are not readable WKB, or whose text is not valid UTF-8 (with asWKB="false", also a curve or surface, which cannot be returned as WKT: read those with the default asWKB). Features the file declares but the reader does not return (a dropped .dbf record, a truncated table) are one record for the file; records a .dbf marks deleted, in a plain or a zipped shapefile, are not data and are not counted. |
geojson_gbx | Fails the read with the reader's error. | A feature, as for vector_gbx. A GeoJSON document is parsed whole, so a syntax error or a truncated file sets the whole file aside. |
geojsonl_gbx | Fails the read with the reader's error. A line that is not JSON is dropped by the reader. | A feature, as for vector_gbx, and each line that is not JSON (line=<n>, the line in raw). |
gpkg_gbx | Fails the read with the reader's error. A corrupt page ends the read early, without an error. | A feature, as for vector_gbx. When a corrupt page ends the read early, it resumes after the page, and each feature on the page is a record (fid=<n>). |
lidar_gbx | An empty file, one whose header cannot be read, or a LAZ file whose chunk table or LASzip record is implausible or cannot be decoded, whose chunk table has more chunks than its header's point count needs, or whose compressed points (point formats with GPS time) hold a run of 0xFF bytes (it is set aside before the file is decoded), is skipped with a warning and adds no rows. In points mode a truncated LAS file, or a LAZ file that stops decoding, fails the read. In metadata mode only the header is read, so a truncated file still gives its row. | An empty file, one whose header cannot be read, or a LAZ file whose chunk table or LASzip record is implausible or cannot be decoded, whose chunk table has more chunks than its header's point count needs, or whose compressed points (point formats with GPS time) hold a run of 0xFF bytes (it is set aside before the file is decoded). In points mode: a truncated LAS file has its complete points read and the rest is one record; a chunk that fails to decode (LAZ) has the points that decode read, and a stretch that does not is skipped to where decoding works again and is one record (points=<first>-<last>); after max(3 × chunkSize, 250,000) points in a row that could not be read, the rest of the file is one record; points outside the bounds the header declares, widened on every side by their largest extent, are a record (points=…, or point=<n> with its x, y and z): this catches garbled point records and garbage that damaged LAZ data decodes to without an error, and keeps the points of a header whose bounds are a little stale (the reader option checkHeaderBounds="false" turns it off). Waveform data stored after the points (LAS 1.3) is not taken for points. metadata mode reads only the header and checks nothing more. |
exif_gbx | An empty file, or one whose EXIF block makes the parser fail, is skipped with a warning and adds no row. Any other file gives one row, with NULL in the metadata columns it cannot read (a file that is not an image gives a row of NULLs). qc mode fails the read on an image it cannot decode. | An empty file, or one whose EXIF block makes the parser fail. A .jpg / .jpeg file without the JPEG start marker, or whose header segments do not lead to a frame header (nothing is decoded, so a very large image is still read); a missing end marker (a truncated scan, or a Motion Photo's video after the image) is only a warning, and the file's metadata is read. In qc mode, an image it cannot decode. |
raster_gbx, gtiff_gbx, cog_gbx | Fails the read when the file is opened: to plan its windows (tileSize set) or to load its pixels (virtualTiles set to "false"). Otherwise the file is not opened while it is ingested: its row is a virtual tile that refers to the file, and the error comes when a function reads the pixels. | A tile window (window=<column>,<row>,<width>,<height>) whose pixels cannot be decoded, when the pixels are loaded (virtualTiles="false" with tileSize). Virtual tiles are not opened, so they are not checked. |
netcdf_gbx | Fails the read with the reader's error. | In raster mode a variable, or one slice of it (variable=temp,time=1), that cannot be read. A NetCDF-3 file (classic or 64-bit offset) shorter than its header declares, in both modes. |
pmtiles_gbx | Fails the read with the reader's error. A tile cut short by a truncated archive is returned as it is. | A tile shorter than its directory entry declares (tile=<z>/<x>/<y>). A group of tiles that fails to render (tiles=<first>..<last>, source="raster"). |
A NULL or empty geometry is data in every mode: a feature without a location, or POINT EMPTY,
is read like any other. GDAL also returns NULL for a geometry it cannot decode (an unknown geometry
type, unreadable coordinates, a damaged GeoPackage or shapefile record). Such a feature is therefore read
with a NULL geometry. To watch for them, query the geometry column with IS NULL and
ST_IsEmpty. Curves and surfaces (CircularString, CompoundCurve, CurvePolygon, TIN,
PolyhedralSurface, Triangle, their multi forms and collections holding them) are data too.
Some damage reads back as plausible values, and no reader can tell it from data. Examples are garbled
coordinates or attribute values within their ranges (a damaged GeoPackage cell can turn 'abc'
into 0). Other examples are the pixels of an uncompressed raster or NetCDF variable. So are garbled LiDAR points that land
within the header's widened bounds (or anywhere with checkHeaderBounds="false"). A PMTiles tile's bytes
are in this group too (the tile is returned as stored). So is EXIF metadata in a file whose JPEG markers are intact.
These rows are read as data.
Files deleted before they are read
A file that is deleted after it was listed but before it is read is skipped with a warning, not treated as an error. In a stream, Auto Loader counts such a file as processed.
Parallelism
- Vector, LiDAR, EXIF and NetCDF files are each read whole by a single task.
num_partitionsrepartitions the files across that many tasks. - Raster files are planned into tile windows, and PMTiles inputs into groups of tiles; these are
spread across tasks only when
num_partitionsis set. Without it a large raster or archive is read by a single task. On Serverless, setnum_partitions(see the Serverless notes).
See also
- Ingestion Pipelines: choosing between the connector, Auto Loader pipelines and
read_paths. - The readers
read_pathsdrives, and their options: Vector, LiDAR, EXIF/GPS, Raster, COG, NetCDF and PMTiles. - Serverless & Memory: memory limits and parallelism on Serverless.
- Virtual Tiles: how raster windows are read lazily.