LiDAR Writer

Since: lightweight lidar_gbx v0.5.2 · (light-only — no heavyweight tier)

Write LAS/LAZ point-cloud files from a Spark DataFrame — the inverse of the lidar_gbx reader. df.write.format("lidar_gbx") writes a points DataFrame (x, y, z plus optional r, g, b) as sharded .laz files, one per Spark partition, with a deterministic *_<group>_<cluster> shard name when groupCol / clusterCol options are set.

lidar_gbx is light-only: there is no heavyweight JAR counterpart. The writer runs on Serverless, standard (shared), ARM, and classic compute — any environment where laspy and lazrs-python are available.

The lightweight Python DataSources are not auto-registered — call register(spark) once per session before using the lidar_gbx writer.

# Register the lightweight DataSources (once per session)
from databricks.labs.gbx.ds.register import register
register(spark)

Schema

The writer requires columns x, y, z (floating-point coordinates in the point cloud's CRS). RGB color is optional but all-or-nothing: include all of r, g, b (integers 0–255) or none — a partial RGB set raises an error.

Column	Required	Type	Description
`x`	Yes	`double`	Easting / longitude in the point cloud's CRS.
`y`	Yes	`double`	Northing / latitude.
`z`	Yes	`double`	Elevation or height.
`r`	Optional	`integer`	Red channel (0–255). Requires `g` and `b`.
`g`	Optional	`integer`	Green channel (0–255). Requires `r` and `b`.
`b`	Optional	`integer`	Blue channel (0–255). Requires `r` and `g`.

Naming columns (groupCol, clusterCol, nameCol) are read as options and are not written to the .laz output.

v1 column scope

The v1 writer supports x, y, z and optional r, g, b only. Extended per-point attributes such as intensity, classification, return_number, and gps_time are planned for v2; any such column in the DataFrame raises an error that directs you to drop it before writing.

Output

Parts mode (default): one .laz per Spark partition, named *_<group>_<cluster>.laz when groupCol and clusterCol are both set, or <partPrefix>-<uuid>.laz otherwise.

singleFile="true": all partitions are captured as intermediate feather fragments on executors; the driver merges them into a single .laz. This funnels through driver RAM — keep the default sharded parts for very large point clouds.

merge="true": post-hoc fold of existing .laz / .las files already in the output directory into one .laz, without re-reading the DataFrame.

Save modes

.mode("overwrite") clears the target directory's .laz / .las files before writing; .mode("append") writes new shards alongside existing ones. With singleFile, both modes write the resolved single file fresh from the current DataFrame — append does not add points to a pre-existing file. merge requires .mode("append") since it never deletes its inputs beforehand.

Parts mode (default)

Each Spark partition produces one .laz file. When groupCol and clusterCol are set, the shard name is derived from the first row's values in that partition (*_<group>_<cluster>), giving deterministic, collision-free names across repartitioned point clouds. Without naming columns, a UUID stem is used (<partPrefix>-<uuid>.laz).

# Parts mode (default): one .laz per Spark partition.
# With groupCol + clusterCol the shard name is deterministic: *_<group>_<cluster>.laz.
(df
    .repartitionByRange(4, "group", "cluster")
    .write.format("lidar_gbx")
    .option("groupCol", "group")
    .option("clusterCol", "cluster")
    .mode("overwrite")
    .save("/Volumes/main/geobrix_samples/lidar/output"))

# Read back via the lidar_gbx reader
cloud = (spark.read
    .format("lidar_gbx")
    .option("mode", "points")
    .load("/Volumes/main/geobrix_samples/lidar/output"))
cloud.select("x", "y", "z").show(5)

Single-file & directory merge

By default the writer shards output into one .laz per partition — safe at any scale. To consolidate into one .laz, opt in with singleFile, or merge already-written parts after the fact with merge.

# Consolidate into ONE .laz with singleFile="true".
# Executor partitions are captured as feather fragments; the driver merges them.
# Reach for this when the full point cloud fits in driver memory.
(df
    .repartition(4)
    .write.format("lidar_gbx")
    .option("singleFile", "true")
    .option("fileName", "merged")     # optional output stem
    .mode("overwrite")
    .save("/Volumes/main/geobrix_samples/lidar/single"))

# Post-hoc: merge .laz/.las files ALREADY in a directory into one,
# without re-running the source DataFrame (the DataFrame rows are ignored).
(spark.range(1).write               # any DataFrame; source rows are ignored
    .format("lidar_gbx")
    .mode("append")
    .option("merge", "true")
    .option("fileName", "all")
    .option("keepParts", "true")     # keep the source parts alongside the merged file
    .save("/Volumes/main/geobrix_samples/lidar/output"))

singleFile captures each partition's points as a feather fragment on the executor, then the driver concatenates all fragments into one .laz. The merge budget is sized to the node's available RAM (compute-aware): a GPU/AIR node with large host RAM merges large clouds; a minimal CPU node falls back to a conservative 512 MiB floor. Override with mergeMaxMB. Otherwise keep the default sharded parts for very large point clouds.
merge folds the .laz / .las files already in the output directory into one .laz without recomputing the source DataFrame (the rows are ignored). Merging an empty directory raises. merge implies single output and overrides singleFile if both are set. The merge budget is compute-aware (host-RAM-based): a GPU or high-RAM node handles large clouds; override with mergeMaxMB when needed.
keepParts (default false) controls the source parts after a successful merge. Parts are deleted only after the merged output is validated and durably written — a failed or partial merge always leaves every part intact. Set keepParts="true" to keep parts alongside the merged file.

The merge is sized by the driver_merge memory budget (the node's available RAM) — see Serverless & Memory. Override it with mergeMaxMB (MiB) or the GBX_LIDAR_MERGE_MAX_MB environment variable.

Options

Option	Default	Description
`groupCol`	—	Column supplying the group segment of the shard name (`*_<group>_<cluster>`). Both `groupCol` and `clusterCol` must be set for deterministic naming.
`clusterCol`	—	Column supplying the cluster segment of the shard name. Paired with `groupCol`.
`nameCol`	—	Column supplying the full output filename stem per partition (used when `groupCol` / `clusterCol` are not set).
`partPrefix`	`"part"`	Stem prefix for UUID-named shards (`<partPrefix>-<uuid>.laz`) when no naming columns are provided.
`fileName`	— (derived)	Output filename stem for `singleFile` / `merge` output. Absent → derived.
`crs`	—	EPSG code (integer or string, e.g. `"32611"`) to embed in the `.laz` CRS field. Absent → no CRS recorded.
`singleFile`	`"false"`	Write one `.laz` instead of sharded parts. See Single-file & directory merge.
`merge`	`"false"`	Post-hoc: merge the `.laz` / `.las` files already present in the output directory into one file, without re-running the source DataFrame. Requires `.mode("append")`.
`keepParts`	`"false"`	Meaningful only with `singleFile` / `merge`. When `false` (default), source part files are deleted after the merged output is validated and durably written; set to `"true"` to keep them. A failed or partial merge never deletes parts.
`mergeMaxMB`	—	Override the compute-aware merge budget in MiB (binary; 1 unit = 1,048,576 bytes). When set, the driver-side merge is allowed up to this many MiB of concatenated point bytes, regardless of the node's detected RAM. Use `GBX_LIDAR_MERGE_MAX_MB` as an environment-variable equivalent.
`overlap`	`"keep"`	Seam handling for overlapping flight strips, merge mode only: `"keep"` (default) retains every point; `"drop"` gives each seam point to exactly one owning cluster — see Seam unification.
`dedupeExact`	`"false"`	Merge mode only. Drop points that are exact duplicates on `x`, `y`, `z` (and RGB, when present) across the merged parts. See Seam unification.
`voxelSize`	—	Merge mode only. Cell size, in metres, for voxel-grid thinning — keeps one representative point per cell. See Seam unification.

Planned

The option below is parsed but rejected — it raises an error naming the planned feature — and is not yet implemented:

Option	Planned behaviour
`thinOverlap`	Overlap-strip thinning ahead of merge — spatial reduction of overlapping flight-strip points before the seam-unification stages run.

Seam unification

Point clouds assembled from overlapping flight strips or tiles often carry duplicate or near-duplicate points along the seams where two parts cover the same ground. Three merge-mode options unify those seams into one clean point set: overlap, dedupeExact, and voxelSize. All three require .option("merge", "true") — they operate on the .laz / .las parts already in the output directory, not on a fresh DataFrame write.

overlap="drop" resolves double-covered seam points by ownership rather than by discarding one side's file wholesale: each point is assigned to exactly one cluster — the nearest cluster center among every cluster whose bounding box covers that point — so the union comes out non-redundant without a hard tile edge. Ownership is a proxy: it's decided from each cluster's axis-aligned header bounding box and center, not from where that cluster actually has points. In a region where two clusters' bounding boxes overlap but only one of them has real points there, a point can be assigned to the cluster with no data at that location, thinning the seam. For compact clusters that both genuinely densify the shared seam — the typical case — this is benign. overlap="drop" also needs the parts to be cluster-identifiable: filenames must follow the *_<group>_<cluster>.laz pattern produced by writing with groupCol / clusterCol set — use those options, not nameCol, when merging with overlap="drop". A UUID-fallback stem (<partPrefix>-<uuid>.laz) or a stem with no underscore has no cluster id and raises; an arbitrary nameCol basename is not validated as a cluster id — if it happens to contain an underscore, its last _-token is taken as the cluster id as-is, which can silently mis-assign ownership rather than raising.
dedupeExact="true" drops points that are exact duplicates on x, y, z — and on r, g, b too when the parts carry RGB — after rounding to millimetre precision. Two points with the same coordinates but different colors are treated as distinct and both kept.
voxelSize (metres) buckets points into a regular 3D grid and keeps exactly one point per occupied cell: the real input point nearest that cell's geometric center. It never synthesizes an averaged point, so elevation and any RGB values in the output always come from an actual scan return.

The three compose in a fixed order when combined: dedupeExact removes exact duplicates first, then voxelSize thins what remains. overlap="drop" resolves ownership as each part streams in, ahead of both.

# Post-hoc: unify the .laz/.las parts already in the directory (source rows
# are ignored, same as a plain merge="true").
(spark.range(1).write
    .format("lidar_gbx")
    .mode("append")
    .option("merge", "true")
    .option("overlap", "drop")
    .option("dedupeExact", "true")
    .option("voxelSize", "0.5")
    .option("fileName", "unified")
    .save("/Volumes/main/geobrix_samples/lidar/output"))

The merge streams parts one at a time rather than concatenating every part into memory up front: each part is fully read, filtered, and released before the next one is read, while kept points accumulate in memory as they're emitted. Peak memory is therefore roughly the largest single part plus the growing output-sized accumulators — bounded, and far below the sum of every input part, though not literally just the size of the unified output. The same Serverless & Memory budget that governs a plain merge still applies, and mergeMaxMB / GBX_LIDAR_MERGE_MAX_MB still override it.

Column parity still deferred

overlap, dedupeExact, and voxelSize unify x, y, z, and RGB. Per-point attribute columns — intensity, return_number, number_of_returns, classification, and gps_time — remain out of scope for the writer (see v1 column scope) and are not carried through a seam-unification merge.

Round-trip

Reading a .laz file with the lidar_gbx reader (points mode) and writing it back with lidar_gbx reproduces x, y, z coordinates within the LAS scale-factor precision, and RGB channels when present. The writer re-computes the header bounds from the written points.

Coordinate precision

LAS files store coordinates as scaled integers — the scale values in the header control quantisation. A round-trip may introduce sub-scale precision differences (typically sub-millimetre for survey-grade data).

Coordinate reference systems

The crs option accepts an EPSG code (integer or string, e.g. "27700") to embed in the LAS/LAZ header. The LAS format can only encode EPSG codes in its header — ESRI and OGC CRS are not embeddable in the LAS CRS record. If your point cloud's CRS is non-EPSG (e.g. ESRI:54008), pass its EPSG equivalent if one exists, or leave crs unset and track the CRS externally. ESRI/OGC CRS information is preserved in the lidar_gbx reader crs column, but it cannot round-trip through the LAS header itself. See Coordinate reference systems for the full CRS model.

Next Steps

LiDAR Reader — the corresponding read paths (metadata mode, points mode, class/return filters).
LiDAR DSM functions — bin point clouds into DSM, CHM, and isoband rasters.
Writers Overview — all writer formats and the tier split.
NetCDF Writer — another pure-Python DataSource writer following the same pattern.

Schema​

Output​

Parts mode (default)​

Single-file & directory merge​

Options​

Planned​

Seam unification​

Round-trip​

Next Steps​