Skip to main content

Vapor-Eyes — Permian Methane Monitoring

Vapor-Eyes works a satellite methane signal over the Permian Basin using GeoBrix on Databricks. It goes from a wide-area screen down to the operator whose well pad is leaking. It comes in two flavors. One is a Lakeflow pipeline that can run on a schedule to stay current. The other is an interactive notebook series. The series is more of a quick start for understanding the GeoBrix functions involved in the detection cascade.

Minimum GeoBrix version
  • Notebook series: GeoBrix 0.5.0 or newer (minimum) — the series depends on capabilities introduced in 0.4.1. These are the netcdf_gbx reader (Sentinel-5P swath → points) and the TropomiDownloader / EmitDownloader / WellsDownloader sample downloaders. It installs geobrix[stac,vizx] (the env6 wheel, with earthaccess as a separate step; the env5 wheel on Serverless environment 5).
  • Lakeflow SDP pipeline: GeoBrix 0.6.0 or newer — it reads its files with read_paths, new in 0.6.0. It installs the env wheel for its Lakeflow runtime channel (see the Lakeflow tab).

Requires GeoBrix 0.6.0 or newer.

A standalone Permian Basin methane-monitoring pipeline. It is a Lakeflow Declarative Pipeline (SDP) plus an AI/BI dashboard. It packages as a Databricks Asset Bundle and runs at the GeoBrix Python level over Serverless compute.

This is the production counterpart to the notebook series (see the other tab). It uses the same Permian AOI and satellite sources. It runs on a schedule with incremental medallion tables instead of one-off notebook outputs. A fifth data source makes it a current (through 2026) monitoring view rather than a single historical case study.

View on GitHub

examples/notebooks/vapor-eyes/lakeflow — the bundle, pipeline transformations, dashboard, and full setup instructions live here. See its README for deploy/run steps, prerequisites, and good-citizen usage notes for Carbon Mapper.

Runs on Serverless

As of 0.6.0, Lakeflow Spark Declarative Pipelines run DBR 19 on the PREVIEW channel; the examples use it with the env6 wheel. The pipeline installs geobrix[stac,vizx] from the env6 wheel. Its land downloader task (Serverless environment version 6) installs the same wheel. To fall back to the CURRENT channel (DBR 18), set pipeline_channel=CURRENT, point gbx_wheel at the env5 wheel, and set the job environment_version to 5. Each task takes the staged wheel as a single pip dependency entry. The pipeline uses the rasterx and vectorx Python API with no JAR, on Serverless throughout. See Surfaces & Levels.

How it works: from a raw satellite signal to a leaderboard​

The bundle deploys one job, vapor_eyes_lf_job, with two tasks: land downloads the period's raw files, then pipeline refreshes the Lakeflow pipeline over them:

The vapor_eyes_lf_job run graph in Databricks: the land Python task followed by the vapor_eyes_lf_pipeline pipeline task.

Opening the pipeline shows the lineage graph Lakeflow derived from the table-to-table dependencies, 34 tables in all. The streaming tables across the top (s5p_granules, wells_raw, s2_swir_assets, emit_scenes, cm_scenes) are the bronze inventory of what landed. The materialized views below them do the silver detection and attribution. They finish in the gold tables the dashboard reads, such as plume_leaderboard_latest, operator_emissions_leaderboard, and cm_monitoring_status. The PMTiles branch (portfolio_mvt_tiles → pmtiles_shards) runs alongside:

A completed vapor_eyes_lf_pipeline run: streaming bronze tables at the top feeding silver materialized views (s5p_hotspots, emit_plumes, plume_quant, cm_detections, cm_plume_attributed) and gold tables at the bottom, each node with its duration, output record count, expectations, and full-recompute or incremental refresh.

Every scheduled run turns raw satellite files into one ranked, map-ready answer — who is leaking methane in the Permian, and where — through five stages:

  1. Land — a downloader task pulls the period's satellite scenes, plume catalogs, and well records onto a Unity Catalog Volume.
  2. Bronze — what landed — Auto Loader records one row per file, lifting dates/IDs out of each filename, so re-runs pick up only new data.
  3. Silver — the detective work — Sentinel-5P screens the basin for CH₄ hotspots. Sentinel-2 and EMIT sharpen and quantify a plume. Carbon Mapper adds current wind-corrected detections. The pipeline ties each plume to the operator whose well was valid on the day it was seen.
  4. Gold — ranked and map-ready — findings roll up into the tables the dashboard reads. These tables are the leakiest-operators leaderboard, the monitoring panel, the regional CH₄ hotspot surface, and by-play / by-county rollups.
  5. Serve — gold feeds the four-page AI/BI dashboard; a parallel branch folds the same findings into a self-contained PMTiles map you can drop into an app.

Each stage reads only the one before it. The pipeline declares these as table-to-table dependencies. Lakeflow then builds the lineage graph, orders the stages, and recomputes only what changed.

Vapor-Eyes Lakeflow pipeline — five raw sources land on a Volume, become bronze inventory, cascade through silver detection and attribution into gold ranked map-ready tables, and serve an AI/BI dashboard plus a shareable PMTiles map

What it uses from GeoBrix​

  • Bronze: plain Auto Loader binaryFile streams list the staged files, one row per file with the date and IDs lifted out of its name (s5p_granules, s2_swir_assets, emit_scenes, wells_raw). They read no payload data.
  • Silver: read_paths reads exactly the files an inventory lists. It reads the Sentinel-5P granules with netcdf_gbx (one point per observation, spread over 256 tasks) for s5p_hotspots. It reads the Sentinel-2 band files with gtiff_gbx for s2_plume_cells. Each row's _source_path joins it back to its inventory row. As a result, observation_date, band and item_id come from bronze. With on_error="skip", the reader skips a file that cannot be read and writes a warning in the executor logs. The pipeline keeps going instead of failing the update. The pipeline pairs the Sentinel-2 B11 and B12 band files per scene (item_id plus observation_date), not just per date. Two scenes captured on the same day therefore each get their own index. The files never change once landed, so the example does not need declare_ingest. See Lakeflow SDP for the pipeline setup.
  • Detection and tiles: RasterX (rst_mapalgebra, rst_h3_tessellate, rst_clip, rst_summary), VectorX MVT and PMTiles, on Serverless.

Spatial functions used​

The pipeline composes GeoBrix functions with Databricks' built-in spatial SQL and H3. RasterX does the raster ops, VectorX + PMTiles the tiles, and native st_* / h3_* the geometry and gridding. The *_gbx readers, run over the bronze inventories by read_paths, ingest on Serverless.

Spatial functions used across the Vapor-Eyes Lakeflow example — GeoBrix RasterX, VectorX + PMTiles, and *_gbx readers with read_paths, plus Databricks built-in ST and H3 functions

Five data sources​

SourceLicenseRole
Sentinel-5P TROPOMICopernicus, openRegional CH4 screen (H3 hotspot surface)
Sentinel-2 SWIRCopernicus, openTargeted band-ratio detection at the strongest hotspot
EMITNASA LP DAAC, open (Earthdata token)Spectral validation — GeoBrix rst_clip/rst_summary cross-check against JPL's reported concentration
Carbon Mapper TanagerCarbon Mapper, public-good (free token)The authoritative current, quantified (emission_rate_kg_hr), wind-corrected plume layer
TX RRC WellSHLPublic (ArcGIS REST)Attribution — nearest-well / operator lookup

EMIT is a science mission on the ISS with sparse revisit and roughly a 10-month lag before plume products publish. It therefore validates historical detections rather than driving current status. Carbon Mapper is what keeps the pipeline current. It's the only source here with a wind-corrected emission rate. The headline leaderboard and monitoring-status panels are built on it.

Two small context geometries ride alongside: EIA shale plays and Census TIGER counties. The GeoBrix vector reader reads them straight from source. They give the detections a "where in the basin" home.

The leaderboard, done defensibly​

operator_emissions_leaderboard ranks operators by number of high-confidence Carbon Mapper plume detections, with mean/max per-detection emission rate (kg/hr) as the intensity columns — not a summed flow rate. A detection's emission_rate_kg_hr is an instantaneous rate at one overpass. Summing it across a multi-year window would double-count repeat detections of the same source rather than describe a continuous flow. The sum is therefore retained only as a clearly-labeled secondary column, never used to rank.

cm_monitoring_status and cm_activity_monthly give the "is anything active right now" read (last detection date, plumes in the trailing 90 days, active operators) and the month-by-month activity timeline. hotspot_latest and hotspot_persistence carry the Sentinel-5P side: a current regional CH4 screen and a chronic-vs-transient emitter view.

Dashboard​

Four pages, all built on AI/BI's native-geometry map widgets. They use H3 hexagon choropleths for the regional CH4 screen and region choropleths for the play and county rollups. A point map shows Carbon Mapper detections colored by emission rate.

Current Status & Leakiest Operators​

Monitoring-status KPIs, the operator leaderboard, and the Carbon Mapper point map with attribution.

Current Status & Leakiest Operators — monitoring-status KPI tiles, the operator emissions leaderboard, and the Carbon Mapper point map colored by emission rate

Activity Over Time​

Monthly Carbon Mapper activity and the regional CH4 trend, both scoped by a shared date-range filter.

Activity Over Time — monthly Carbon Mapper plume count / emission-rate combo chart and the regional CH4 trend line

Regional Screen (Sentinel-5P)​

The wide-area CH4 hotspot choropleth and the persistence choropleth.

Regional Screen (Sentinel-5P) — the wide-area CH4 hotspot H3 choropleth and the chronic-vs-transient persistence choropleth

Regional Context​

This page rolls up Carbon Mapper detections to the Permian's named shale plays (EIA) and its TX/NM counties (Census TIGER). The GeoBrix vector reader reads the plays and counties straight from source. Two choropleths show where detections concentrate.

Regional Context — Carbon Mapper detections rolled up to Permian shale plays and TX/NM counties as two choropleths

Good-citizen use of Carbon Mapper​

Carbon Mapper's plume catalog is public-good data. This example is built to use it responsibly. Every user brings their own free token (BYOT — the pipeline is a client, not a redistributor). The dashboard shows a "Data © Carbon Mapper" attribution wherever Carbon Mapper data appears. No Carbon Mapper data is ever committed to the repository. The pipeline fetches it at runtime straight to your own Unity Catalog Volume. This example is scoped for public-good analytics and demonstration; for commercial or low-latency operational use of Carbon Mapper/Tanager data, contact Planet Labs. See the lakeflow README for the full terms-of-use notes and token setup.