Skip to main content

Vapor-Eyes — Permian Methane Monitoring

Vapor-Eyes works a satellite methane signal over the Permian Basin — from a wide-area screen down to the operator whose well pad is leaking — using GeoBrix on Databricks. It comes in two flavors: a production-ready Lakeflow pipeline that can run on a schedule to stay current, and an interactive notebook series that is more of a quick start for understanding the GeoBrix functions involved in the detection cascade.

Requires GeoBrix 0.4.1+

Both flavors depend on capabilities introduced in the 0.4.1 release — the netcdf_gbx reader (Sentinel-5P swath → points) and the TropomiDownloader / EmitDownloader / WellsDownloader sample downloaders — so install geobrix[light,stac,vizx] 0.4.1 or newer.

A standalone, production-grade Permian Basin methane-monitoring pipeline: a Lakeflow Declarative Pipeline (SDP) plus an AI/BI dashboard, packaged as a Databricks Asset Bundle and running on the GeoBrix lightweight tier over Serverless compute.

This is the production counterpart to the notebook series (see the other tab) — same Permian AOI and satellite sources, but running on a schedule with incremental medallion tables instead of one-off notebook outputs, and a fifth data source that makes it a current (through 2026) monitoring view rather than a single historical case study.

View on GitHub

notebooks/examples/vapor-eyes/lakeflow — the bundle, pipeline transformations, dashboard, and full setup instructions live here. See its README for deploy/run steps, prerequisites, and good-citizen usage notes for Carbon Mapper.

Runs on the lightweight tier (Serverless)

The pipeline and its land downloader task both install geobrix[light,stac,vizx] from a staged wheel as a single pip dependency entry — pure Python/PySpark bindings (databricks.labs.gbx.pyrx, pyvx), no JAR, Serverless throughout (environment version 5). See Execution Tiers.

How it works: from a raw satellite signal to a leaderboard​

Vapor-Eyes Lakeflow pipeline — five raw sources land on a Volume, become bronze inventory, cascade through silver detection and attribution into gold ranked map-ready tables, and serve an AI/BI dashboard plus a shareable PMTiles map

Every scheduled run turns raw satellite files into one ranked, map-ready answer — who is leaking methane in the Permian, and where — through five stages:

  1. Land — a downloader task pulls the period's satellite scenes, plume catalogs, and well records onto a Unity Catalog Volume.
  2. Bronze — what landed — Auto Loader records one row per file, lifting dates/IDs out of each filename, so re-runs pick up only new data.
  3. Silver — the detective work — Sentinel-5P screens the basin for CH₄ hotspots, Sentinel-2 and EMIT sharpen and quantify a plume, Carbon Mapper adds current wind-corrected detections, and each plume is tied to the operator whose well was valid on the day it was seen.
  4. Gold — ranked and map-ready — findings roll up into the tables the dashboard reads: the leakiest-operators leaderboard, the monitoring panel, the regional CH₄ hotspot surface, and by-play / by-county rollups.
  5. Serve — gold feeds the four-page AI/BI dashboard; a parallel branch folds the same findings into a self-contained PMTiles map you can drop into an app.

Each stage reads only the one before it — declared as table-to-table dependencies, so Lakeflow builds the lineage graph, orders the stages, and recomputes only what changed.

Spatial functions used​

The pipeline composes GeoBrix functions with Databricks' built-in spatial SQL and H3 — RasterX for raster ops, VectorX + PMTiles for tiles, the light-tier readers for Serverless ingest, and native st_* / h3_* for geometry and gridding.

Spatial functions used across the Vapor-Eyes Lakeflow example — GeoBrix RasterX, VectorX + PMTiles, and light readers, plus Databricks built-in ST and H3 functions

Five data sources​

SourceLicenseRole
Sentinel-5P TROPOMICopernicus, openRegional CH4 screen (H3 hotspot surface)
Sentinel-2 SWIRCopernicus, openTargeted band-ratio detection at the strongest hotspot
EMITNASA LP DAAC, open (Earthdata token)Spectral validation — GeoBrix rst_clip/rst_summary cross-check against JPL's reported concentration
Carbon Mapper TanagerCarbon Mapper, public-good (free token)The authoritative current, quantified (emission_rate_kg_hr), wind-corrected plume layer
TX RRC WellSHLPublic (ArcGIS REST)Attribution — nearest-well / operator lookup

EMIT is a science mission on the ISS with sparse revisit and roughly a 10-month lag before plume products publish, so it validates historical detections rather than driving current status. Carbon Mapper is what keeps the pipeline current: it's the only source here with a wind-corrected emission rate, and it's what the headline leaderboard and monitoring-status panels are built on.

Two small context geometries ride alongside — EIA shale plays and Census TIGER counties — read straight from source with the GeoBrix light vector reader to give the detections a "where in the basin" home.

The leaderboard, done defensibly​

operator_emissions_leaderboard ranks operators by number of high-confidence Carbon Mapper plume detections, with mean/max per-detection emission rate (kg/hr) as the intensity columns — not a summed flow rate. A detection's emission_rate_kg_hr is an instantaneous rate at one overpass; summing it across a multi-year window would double-count repeat detections of the same source rather than describe a continuous flow, so the sum is retained only as a clearly-labeled secondary column, never used to rank.

cm_monitoring_status and cm_activity_monthly give the "is anything active right now" read (last detection date, plumes in the trailing 90 days, active operators) and the month-by-month activity timeline. hotspot_latest and hotspot_persistence carry the Sentinel-5P side: a current regional CH4 screen and a chronic-vs-transient emitter view.

Dashboard​

Four pages, all built on AI/BI's native-geometry map widgets — H3 hexagon choropleths for the regional CH4 screen, region choropleths for the play and county rollups, and a point map for Carbon Mapper detections colored by emission rate:

  1. Current Status & Leakiest Operators — monitoring-status KPIs, the operator leaderboard, and the Carbon Mapper point map with attribution.
  2. Activity Over Time — monthly Carbon Mapper activity and the regional CH4 trend, both scoped by a shared date-range filter.
  3. Regional Screen (Sentinel-5P) — the wide-area CH4 hotspot choropleth and the persistence choropleth.
  4. Regional Context — Carbon Mapper detections rolled up to the Permian's named shale plays (EIA) and its TX/NM counties (Census TIGER), read straight from source with the GeoBrix light vector reader and shown as two choropleths of where detections concentrate.

Good-citizen use of Carbon Mapper​

Carbon Mapper's plume catalog is public-good data, and this example is built to use it responsibly: every user brings their own free token (BYOT — the pipeline is a client, not a redistributor), the dashboard shows a "Data © Carbon Mapper" attribution wherever Carbon Mapper data appears, and no Carbon Mapper data is ever committed to the repository — it's fetched at runtime straight to your own Unity Catalog Volume. This example is scoped for public-good analytics and demonstration; for commercial or low-latency operational use of Carbon Mapper/Tanager data, contact Planet Labs. See the lakeflow README for the full terms-of-use notes and token setup.