Skip to main content
Databricks Labs

SDP-META

Metadata-driven framework for automated Bronze and Silver pipelines on Lakeflow Spark Declarative Pipelines. Define your pipeline in JSON or YAML — SDP-META handles the rest.

Get Started →What is SDP-META?GitHub ↗
docspassingpypiv0.1.0downloads225k/molicenseDatabricks Labs

Describe your pipelines in YAML or JSON. SDP-META generates the entire Lakeflow graph — no pipeline code to write.

AutoloaderS3 / ADLS / GCSKafka / EventHubDeltaBronze → Silveronboarding.yaml / .jsonsource_format: cloudFilesbronze_table: orders_rawsilver_table: orderssilver_transformation_jsonbronze_data_quality_ expectations_jsonbronze_cdc_apply_changesOnboardingJobruns oncebronze_dataflowspecDelta table · Unity Catalogsilver_dataflowspecDelta table · Unity CatalogGenericPipeline· Bronze flows· Silver flows· DQ · CDC · SinksBronzetablesSilvertablesQuarantinetables
v0.1.0DLT-META is now SDP-META; existing onboarding files still work.Migration guide →

Zero to pipeline in minutes

The recommended path is a Declarative Automation Bundle — git-tracked, CI/CD-ready, promotes across dev/prod. One command scaffolds everything.

Full Quickstart Guide →
# Install
pip install 'databricks-labs-sdp-meta[mcp]'
databricks labs install sdp-meta

# Scaffold a bundle (zero prompts, dev-friendly defaults)
databricks labs sdp-meta bundle-init --quickstart

# Validate and deploy
databricks labs sdp-meta bundle-validate
databricks bundle deploy

Choose your path

SDP-META supports four deployment interfaces. Start with DABs for any real work.