Skip to main content

read_paths and reference

Since: v0.6.0

read_paths is the building block of the Ingestion Pipelines. It takes a DataFrame of file paths (a batch listing or an Auto Loader stream) and reads every file with the GeoBrix reader you name. Use it directly in notebooks and jobs. declare_ingest and the Lakeflow connector read files with the same code, so the Reference at the end of this page applies to all three.

read_paths is pure Python: no JAR and no register(spark) (it calls the reader classes directly). It runs on Serverless, standard (shared) and classic compute.

read_paths flow — a paths DataFrame has one row per file, projected to path and modificationTime. It comes from a batch binaryFile or file listing, or from an Auto Loader stream with schema_path. One *_gbx reader reads each file. Vector, LiDAR, EXIF and NetCDF files are each read whole by one task. num_partitions spreads the files across tasks. Raster and PMTiles inputs take two stages. The first stage plans each raster file into its tile windows (tileSize) and each PMTiles input into groups of tiles. The second stage reads each window on its own, spread across tasks with num_partitions. The output rows have the reader’s columns, one row per feature, point, image or tile. They add the _source_path (STRING, the bare /Volumes path of the file) and _source_mtime (TIMESTAMP, its modification time) lineage columns. lineage=False leaves those out. What cannot be read fails the read with on_error fail, the default, naming the file, or is skipped with a warning with on_error skip. Where the rows go is yours; declare_ingest and the connector read files with the same code

Read a list of files​

Each row of the input DataFrame names one file. Project an Auto Loader or binaryFile listing down to path and modificationTime (the file bytes are never loaded) and pass it to read_paths together with the reader name:

from databricks.labs.gbx.ds.paths import read_paths

files = (spark.read.format("binaryFile")
.option("pathGlobFilter", "*.geojson")
.load("/Volumes/main/default/geobrix_samples/geobrix-examples/nyc/boroughs")
.select("path", "modificationTime")) # file bytes are never loaded
df = read_paths(files, "geojson_gbx")
df.select("boroname", "_source_path").show()
Example output
+-------------+--------------------+
| boroname| _source_path|
+-------------+--------------------+
|Staten Island|/Volumes/main/def...|
| Bronx|/Volumes/main/def...|
| ...| ...|
+-------------+--------------------+
read_paths(df, format, options=None, *, path_col="path", mtime_col="modificationTime",
schema_path=None, num_partitions=None, lineage=True, on_error="fail")
ParameterDefaultDescription
dfrequiredDataFrame with one row per file. Batch or streaming.
formatrequiredReader name from the formats table.
optionsNoneDict of reader options, exactly as on that reader's page (for example layerName, chunkSize or tileSize).
path_col"path"Column holding the file path, a Unity Catalog Volume path (see Paths). dbfs: and file: prefixes are accepted.
mtime_col"modificationTime"Column copied to _source_mtime. Pass None to leave _source_mtime NULL.
schema_pathNoneThe file whose columns define the output schema, or a folder: then only its first file (in path order) matching the format's default file pattern is opened. For a batch DataFrame the default is the first path in df. Required for a streaming DataFrame.
num_partitionsNoneSpread the work across this many tasks. See Parallelism.
lineageTrueSet False to leave out _source_path and _source_mtime.
on_error"fail""fail": an error fails the read, naming the file. "skip": what cannot be read is skipped with a warning in the executor logs, and everything else is read. See Unreadable files.

Stream new files​

Pass an Auto Loader stream the same way. Because a stream has no first row to look at, give schema_path a file or a folder. For a folder, the output schema comes from its first file whose name matches the format's default file pattern. That pattern, not the stream's pathGlobFilter, picks the file, so point schema_path at a file to choose it exactly.

from databricks.labs.gbx.ds.paths import read_paths

landing = "/Volumes/main/geo/landing/boroughs"

files = (spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "binaryFile")
.option("pathGlobFilter", "*.geojson")
.load(landing)
.select("path", "modificationTime"))
rows = read_paths(files, "geojson_gbx", schema_path=landing) # required for a stream

(rows.writeStream
.option("checkpointLocation", "/Volumes/main/geo/checkpoints/boroughs")
.trigger(availableNow=True)
.toTable("main.geo.boroughs"))

Auto Loader's checkpoint remembers which files were delivered, so each run reads only the files that arrived since the last one. Auto Loader (cloudFiles) is a Databricks feature. In a Lakeflow pipeline, declare_ingest declares this stream for you, together with what happens when a file is overwritten.

Rasters​

Raster files (raster_gbx, gtiff_gbx and cog_gbx) are planned into their tile windows first. The windows are read as separate tasks only when num_partitions is set; without it, a large raster is read by a single task. Set num_partitions (on Serverless, always) and use the raster reader's tileSize option to choose the window size:

from databricks.labs.gbx.ds.paths import read_paths

files = (spark.read.format("binaryFile")
.option("pathGlobFilter", "*.tif")
.load("/Volumes/main/default/geobrix_samples/geobrix-examples/nyc/sentinel2")
.select("path", "modificationTime"))
tiles = read_paths(
files, "gtiff_gbx",
options={"tileSize": "512"}, # cut each raster into 512 x 512 pixel windows
num_partitions=8, # spread the windows across 8 tasks
)
tiles.select("tile.window", "_source_path").show(3, truncate=False)
Example output
+---------------------+----------------------------------+
|window |_source_path |
+---------------------+----------------------------------+
|{0, 0, 512, 512} |/Volumes/.../nyc_sentinel2_red.tif|
|{1536, 0, 512, 512} |/Volumes/.../nyc_sentinel2_red.tif|
|... |... |
+---------------------+----------------------------------+

By default each raster file is a single whole-file tile; tileSize cuts it into a grid of windows (see the Raster Reader for its options). The raster manifest and tilesTable options are rejected by read_paths: each input row already names one file, and a manifest would be read again for every row. Read a manifest with spark.read.format("gtiff_gbx") instead, and a tile index with read_tiles_table. A .vrt source reports the .vrt file itself as its _source_path.

PMTiles inputs are planned the same way, in groups of tilesPerPartition tiles, for an archive (source="archive") and for a GeoTIFF it renders (source="raster"). A NetCDF file is read whole by one task in both modes. The reader reads all of a file's variables and slices in one pass.

Reference​

Everything below applies to every route: the Lakeflow connector, Auto Loader pipelines and read_paths.

Supported formats​

read_paths and declare_ingest take the name of a *_gbx reader. The _gbx suffix is optional and the name is case-insensitive; the connector's format table option takes the same names.

formatReaderOne input file becomes
vector_gbxVectorFeatures (generic OGR reader)
shapefile_gbxShapefileFeatures
geojson_gbxGeoJSONFeatures
geojsonl_gbxGeoJSONLFeatures
gpkg_gbxGeoPackageFeatures
file_gdb_gbxFile GeodatabaseFeatures
lidar_gbxLiDARPoints
exif_gbxEXIF/GPSImage metadata
raster_gbx, gtiff_gbxRaster, GeoTIFFRaster tiles
cog_gbxCOGRaster tiles, split into tile windows the same way as raster_gbx (see Rasters). The Lakeflow connector maps a .tif file to gtiff_gbx; set format: cog there to use the COG reader.
netcdf_gbxNetCDFSet by the mode option: "raster" (default) gives one (source, tile) GeoTIFF tile per grid variable (and per slice with fanout); "vector" gives one point feature per observation or grid cell.
pmtiles_gbxPMTilesSet by the source option, which is required here: "archive" gives one (z, x, y, bytes) row per tile stored in a .pmtiles file; "raster" renders (z, x, y, bytes) XYZ tiles from a GeoTIFF. Each file is rendered on its own, so a tile that covers two files appears once for each.

file_gbx is not supported. It only lists files, and the binaryFile or Auto Loader listing that feeds read_paths already gives each file's path, size and modification time.

Lineage columns​

Each row carries the columns that reader normally returns, plus two lineage columns:

ColumnTypeMeaning
_source_pathSTRINGBare /Volumes/... path of the file the row came from
_source_mtimeTIMESTAMPThat file's modification time

Default file patterns​

Auto Loader delivers every file under a folder separately. The parts of a multi-file dataset (a shapefile's .shx, .dbf and .prj sidecars, or the files inside a .gdb folder) would otherwise be read as inputs of their own. When its glob is not set, declare_ingest passes the format's pattern to Auto Loader. The same pattern picks the schema file from a folder schema_path, in read_paths too. (The connector picks a table's files by extension instead; see its discovery rules.)

formatDefault glob
vector_gbx*.{gpkg,geojson,geojsonl,geojsons,shp,shz,shp.zip,gdb.zip,GPKG,GEOJSON,GEOJSONL,GEOJSONS,SHP,SHZ,SHP.ZIP,GDB.ZIP}
shapefile_gbx*.{shp,shz,shp.zip,SHP,SHZ,SHP.ZIP}
geojson_gbx*.{geojson,GEOJSON}
geojsonl_gbx*.{geojsonl,geojsons,GEOJSONL,GEOJSONS}
gpkg_gbx*.{gpkg,GPKG}
file_gdb_gbx*.{gdb.zip,GDB.ZIP}
lidar_gbx*.{las,laz,LAS,LAZ}
exif_gbx*.{jpg,jpeg,JPG,JPEG}
raster_gbx, gtiff_gbx, cog_gbx*.{tif,tiff,TIF,TIFF}
netcdf_gbx*.{nc,NC}
pmtiles_gbx with source="archive"*.{pmtiles,PMTILES}
pmtiles_gbx with source="raster"*.tif

The patterns match the file name as written, so they cover lower- and upper-case extensions (photo.jpg and PHOTO.JPG) but not mixed case such as .Jpg. Set glob for other names, for example glob="*.jp2" for another raster format or glob="*.{nc,nc4}" for NetCDF-4 files named .nc4. Deliver a File Geodatabase as a zipped .gdb.zip file. The pmtiles_gbx raster pattern is *.tif only because that reader's filterRegex option keeps .tif files by default; to render other files, set both filterRegex and glob.

GeoJSON is matched strictly by its .geojson extension, because many .json files are not GeoJSON. For GeoJSON files named .json, pass glob="*.{geojson,json}".

A glob you set replaces the default pattern rather than narrowing it, so it must still leave out the other files of a multi-file dataset. For shapefiles, end it in the data file's extension: glob="zones_*.shp" delivers only the .shp files, while glob="zones_*" also delivers each shapefile's .dbf, .shx and .prj sidecars as inputs of their own. Patterns use Spark pathGlobFilter syntax: case-sensitive, with {a,b} groups, and a backslash escapes a pattern character (\* matches a literal *).

Schema​

The table schema comes from one file: schema_path, or for a folder its first file matching the format's default file pattern (or glob). Files and folders whose names start with _ or . are skipped, as Spark's file sources skip them. For file_gdb_gbx and vector_gbx, an unzipped .gdb folder inside that folder counts as one file. No other file is opened to resolve it, so a folder that already holds files with other columns still resolves. Every file that is read must have the same columns with the same types. A file that differs by column name or type fails with an error that names the file. A file whose columns are in a different order is realigned by name. A file with no features adds no rows.

Paths​

Paths are Unity Catalog Volume paths: /Volumes/..., or dbfs:/Volumes/... as Auto Loader and binaryFile report them. The readers open each file through the Volume's file system. An object-store URI (s3://, abfss://, gs://) or a GDAL virtual path (/vsis3/..., /vsizip/..., /vsimem/...) therefore fails with an error that names it. It is not skipped as missing. To ingest from cloud storage, create an external volume over the location and use its /Volumes path.

Unreadable files​

What happens to a file, or a part of one, that cannot be read (truncated, corrupt, or not the format its name says) depends on on_error:

  • on_error="fail", the default of read_paths. The Lakeflow connector always reads this way. A read error fails the read, with an error that names the file. A few readers handle some damage themselves, as the second column of the table below says.
  • on_error="skip" (read_paths) and on_error="quarantine" (the default of declare_ingest). Each file is read on its own, and so is each raster window or group of PMTiles tiles. What cannot be read is a bad record: read_paths skips it with a warning in the executor logs, and declare_ingest writes it to <t>_bad_records. Everything else is read, including the rows of a file read before an error in it. Errors of the environment are not bad data, so they still fail the read. These are out of memory, a missing module, an I/O error of the Volume (retried first), and an image too large for exif_gbx qc mode.

With skip or quarantine, a file that cannot be opened is one bad record for the whole file. The same holds for a file whose columns differ from the table schema (see Schema). Smaller bad records depend on the format:

formatWith on_error="fail"Also set aside with skip / quarantine
vector_gbx, shapefile_gbx, file_gdb_gbxFails the read with the reader's error.A feature (feature=<n>) whose geometry bytes are not readable WKB, or whose text is not valid UTF-8 (with asWKB="false", also a curve or surface, which cannot be returned as WKT: read those with the default asWKB). Features the file declares but the reader does not return (a dropped .dbf record, a truncated table) are one record for the file; records a .dbf marks deleted, in a plain or a zipped shapefile, are not data and are not counted.
geojson_gbxFails the read with the reader's error.A feature, as for vector_gbx. A GeoJSON document is parsed whole, so a syntax error or a truncated file sets the whole file aside.
geojsonl_gbxFails the read with the reader's error. A line that is not JSON is dropped by the reader.A feature, as for vector_gbx, and each line that is not JSON (line=<n>, the line in raw).
gpkg_gbxFails the read with the reader's error. A corrupt page ends the read early, without an error.A feature, as for vector_gbx. When a corrupt page ends the read early, it resumes after the page, and each feature on the page is a record (fid=<n>).
lidar_gbxAn empty file, one whose header cannot be read, or a LAZ file whose chunk table or LASzip record is implausible or cannot be decoded, whose chunk table has more chunks than its header's point count needs, or whose compressed points (point formats with GPS time) hold a run of 0xFF bytes (it is set aside before the file is decoded), is skipped with a warning and adds no rows. In points mode a truncated LAS file, or a LAZ file that stops decoding, fails the read. In metadata mode only the header is read, so a truncated file still gives its row.An empty file, one whose header cannot be read, or a LAZ file whose chunk table or LASzip record is implausible or cannot be decoded, whose chunk table has more chunks than its header's point count needs, or whose compressed points (point formats with GPS time) hold a run of 0xFF bytes (it is set aside before the file is decoded). In points mode: a truncated LAS file has its complete points read and the rest is one record; a chunk that fails to decode (LAZ) has the points that decode read, and a stretch that does not is skipped to where decoding works again and is one record (points=<first>-<last>); after max(3 × chunkSize, 250,000) points in a row that could not be read, the rest of the file is one record; points outside the bounds the header declares, widened on every side by their largest extent, are a record (points=…, or point=<n> with its x, y and z): this catches garbled point records and garbage that damaged LAZ data decodes to without an error, and keeps the points of a header whose bounds are a little stale (the reader option checkHeaderBounds="false" turns it off). Waveform data stored after the points (LAS 1.3) is not taken for points. metadata mode reads only the header and checks nothing more.
exif_gbxAn empty file, or one whose EXIF block makes the parser fail, is skipped with a warning and adds no row. Any other file gives one row, with NULL in the metadata columns it cannot read (a file that is not an image gives a row of NULLs). qc mode fails the read on an image it cannot decode.An empty file, or one whose EXIF block makes the parser fail. A .jpg / .jpeg file without the JPEG start marker, or whose header segments do not lead to a frame header (nothing is decoded, so a very large image is still read); a missing end marker (a truncated scan, or a Motion Photo's video after the image) is only a warning, and the file's metadata is read. In qc mode, an image it cannot decode.
raster_gbx, gtiff_gbx, cog_gbxFails the read when the file is opened: to plan its windows (tileSize set) or to load its pixels (virtualTiles set to "false"). Otherwise the file is not opened while it is ingested: its row is a virtual tile that refers to the file, and the error comes when a function reads the pixels.A tile window (window=<column>,<row>,<width>,<height>) whose pixels cannot be decoded, when the pixels are loaded (virtualTiles="false" with tileSize). Virtual tiles are not opened, so they are not checked.
netcdf_gbxFails the read with the reader's error.In raster mode a variable, or one slice of it (variable=temp,time=1), that cannot be read. A NetCDF-3 file (classic or 64-bit offset) shorter than its header declares, in both modes.
pmtiles_gbxFails the read with the reader's error. A tile cut short by a truncated archive is returned as it is.A tile shorter than its directory entry declares (tile=<z>/<x>/<y>). A group of tiles that fails to render (tiles=<first>..<last>, source="raster").

A NULL or empty geometry is data in every mode: a feature without a location, or POINT EMPTY, is read like any other. GDAL also returns NULL for a geometry it cannot decode (an unknown geometry type, unreadable coordinates, a damaged GeoPackage or shapefile record). Such a feature is therefore read with a NULL geometry. To watch for them, query the geometry column with IS NULL and ST_IsEmpty. Curves and surfaces (CircularString, CompoundCurve, CurvePolygon, TIN, PolyhedralSurface, Triangle, their multi forms and collections holding them) are data too.

Some damage reads back as plausible values, and no reader can tell it from data. Examples are garbled coordinates or attribute values within their ranges (a damaged GeoPackage cell can turn 'abc' into 0). Other examples are the pixels of an uncompressed raster or NetCDF variable. So are garbled LiDAR points that land within the header's widened bounds (or anywhere with checkHeaderBounds="false"). A PMTiles tile's bytes are in this group too (the tile is returned as stored). So is EXIF metadata in a file whose JPEG markers are intact. These rows are read as data.

Files deleted before they are read​

A file that is deleted after it was listed but before it is read is skipped with a warning, not treated as an error. In a stream, Auto Loader counts such a file as processed.

Parallelism​

  • Vector, LiDAR, EXIF and NetCDF files are each read whole by a single task. num_partitions repartitions the files across that many tasks.
  • Raster files are planned into tile windows, and PMTiles inputs into groups of tiles; these are spread across tasks only when num_partitions is set. Without it a large raster or archive is read by a single task. On Serverless, set num_partitions (see the Serverless notes).

See also​