Skip to main content

File Access — file_gbx

file_gbx has two roles:

  1. File lister DataSource — emits one row per file (path, name, extension, size, modification time). Use it as the first step in a preparation pipeline: list the files you want to process, then pipe those references into a writer such as cog_gbx.
  2. Shared file-access base — the Python module that every lightweight reader and writer uses internally for FILE / FUSE routing, directory enumeration, and write-mode selection. See the Readers & Writers — Shared file-access base section for the capability tier diagram and the no-gating rule.

file_gbx is format-agnostic: it lists any file type and never decodes file content.

Lightweight only

file_gbx is a pure-Python lightweight DataSource. Register it with register(spark) before use. There is no heavyweight GDAL counterpart.

Output schema (DataSource)​

root
 |-- path:             string    — full absolute path to the file
 |-- name:             string    — filename including extension
 |-- extension:        string    — lowercase, no leading dot; NULL for files with no extension
 |-- size:             long      — file size in bytes
 |-- modificationTime: timestamp — last-modified time

No raster or vector content is loaded. The path values are the references you pass to a subsequent writer.

Register​

from databricks.labs.gbx.ds.register import register
register(spark)

DataSource options​

OptionDefaultDescription
filterRegex".*"Keep only files whose full path matches this regex.
Always recursive

The file_gbx DataSource always walks subdirectories. Use filterRegex to scope which files are included.

List files​

# List raster files in a directory — no content loaded.
from databricks.labs.gbx.ds.register import register
register(spark)

refs = spark.read.format("file_gbx").load(
    "/Volumes/main/geobrix_samples/geobrix-examples/nyc/sentinel2"
)
refs.show(truncate=False)
# path | name | extension | size | modificationTime

Filter by extension​

# Keep only .tif files via filterRegex:
refs = (
    spark.read.format("file_gbx")
         .option("filterRegex", r".*\\.tif$")
         .load("/Volumes/main/geobrix_samples/geobrix-examples/nyc/sentinel2")
)

Typical use: COG preparation​

The primary use of file_gbx is to feed the cog_gbx writer: list the source files, then write master COGs — one per source file — for efficient windowed reading later.

# Step 1 — list source files.
from databricks.labs.gbx.ds.register import register
register(spark)

refs = spark.read.format("file_gbx").load(
    "/Volumes/main/geobrix_samples/geobrix-examples/nyc/sentinel2"
)

# Step 2 — convert each source file to a master COG.
import tempfile, os
OUT = "/Volumes/main/geobrix_samples/cog-prepared/nyc-sentinel2"

(
    refs.write.format("cog_gbx")
        .option("cogBlockSize", "512")
        .option("cogOverviewResampling", "AVERAGE")
        .option("cogCompression", "DEFLATE")
        .mode("overwrite")
        .save(OUT)
)
print("COGs written to", OUT)

After preparation, read clipped windows with the cog_gbx reader. See COG Reader for the read path.

Python API​

The file_gbx module also exposes a Python API for direct use in UDFs, scripts, and pipeline code.

enumerate_files — directory listing​

from databricks.labs.gbx.ds.file_gbx import enumerate_files

# Basic: list all non-hidden files recursively (default)
files = enumerate_files("/Volumes/main/geo/rasters", spark=spark)

# Filter to GeoTIFFs only
tifs = enumerate_files(
    "/Volumes/main/geo/rasters",
    extensions=(".tif", ".tiff"),
    spark=spark,
)

# Include Hadoop metadata files (_SUCCESS, _committed_*, .crc, …)
all_files = enumerate_files(
    "/Volumes/main/geo/rasters",
    include_hidden=True,
    spark=spark,
)

On FILE-capable runtimes (DBR 13.3+), enumerate_files returns a Spark DataFrame with columns path, size, and file (a FILE reference). On FUSE-only runtimes it returns a Python list of {path, size, file} dicts where file is None.

Hidden-file filtering​

By default, files whose names start with _ or . are skipped — this matches Spark/Hadoop conventions where _SUCCESS, _committed_*, _delta_log, and .crc are metadata artefacts, not data files. Pass include_hidden=True to re-admit them.

Positive selection filters​

Use extensions or path_glob_filter to narrow which files are returned. The two parameters are mutually exclusive — providing both raises ValueError.

  • extensions: a tuple of case-insensitive suffixes, e.g. (".tif", ".nc"). Sugar for path_glob_filter — compiled internally to ["*.tif", "*.nc"].
  • path_glob_filter: an fnmatch-style glob applied to each file's basename, e.g. "*.tif" or "[!.]*".

The filter is ANDed with include_hidden: setting include_hidden=True + path_glob_filter="[!.]*" includes underscore-named files (_data.tif) but still excludes dot-named files (.crc, .DS_Store).

# Include _data.tif but exclude .crc / .DS_Store
filtered = enumerate_files(
    "/Volumes/main/geo/rasters",
    include_hidden=True,
    path_glob_filter="[!.]*",   # starts with any char that is NOT '.'
    spark=spark,
)

Capability tiers and the no-gating rule​

file_access_tier(spark) returns the best tier available at runtime:

from databricks.labs.gbx.ds.file_gbx import file_access_tier

tier = file_access_tier(spark)
# Returns: "read_files" (DBR 13.3+), "list_files" (DBR 18+), or "fuse" (always)

open_for_read(source, access="auto") is the read resolver every lightweight reader calls: it validates the access mode and enforces the no-gating rule (the size-adaptive routing — FILE byte-range stream, FUSE-of-FILE, or staging — runs in the reader layer on top of it). With access="auto" it accepts the best available tier and never errors. Passing access="managed" or access="external" on a FUSE-only runtime raises a clear error:

from databricks.labs.gbx.ds.file_gbx import open_for_read

# Auto — never errors regardless of runtime:
path = open_for_read("/Volumes/main/geo/scene.tif", spark=spark)

# Explicit FILE — raises ValueError on FUSE-only runtimes:
path = open_for_read(
    "/Volumes/main/geo/scene.tif",
    access="managed",
    spark=spark,
)

Ingest existing files into a managed FILE-column table​

ingest_files reads files from an external Volume path via read_files(format=>'file') and inserts them as FILE MANAGED references into a Delta table, without copying the bytes:

from databricks.labs.gbx.ds.file_gbx import ingest_files

ingest_files(
    spark,
    src="/Volumes/main/geo/archive/rasters",
    target="main.geo.raster_registry",
    filespace="/Volumes/main/geo/managed_store",
    file_col="tile_file",        # name of the FILE-typed column
    layout="order",              # ORDER BY path (default)
    recursive=True,
    overwrite=False,             # CREATE TABLE IF NOT EXISTS (idempotent)
)

ingest_files requires a FILE-capable runtime (DBR 13.3+). On FUSE-only runtimes it raises ValueError with an upgrade message — use open_for_write(file_mode="fuse") for a plain Delta write instead.

Next steps​