Skip to main content

CLI Commands

All SDP-META operations are available through the Databricks Labs CLI extension.

Prerequisites:

  • Databricks CLI installed and authenticated
  • databricks labs install sdp-meta

Command summary​

CommandDescription
onboardInteractive onboarding wizard — prompts for config, pushes code, creates onboarding job
deployDeploy bronze/silver Declarative Pipeline interactively
bundle-initScaffold a new DAB bundle (--quickstart for zero-prompt fast path)
bundle-prepare-wheelBuild and upload the sdp-meta wheel to a UC Volume
bundle-add-flowAdd a new flow to an existing bundle from UC, Volumes, Kafka topics, or CSV inventory
bundle-add-pipelineAdd an independently configured pipeline topology and job wiring
bundle-validateValidate bundle configuration (enforces sdp_meta_dependency is set)
mcpStart the MCP server (stdio transport)

onboard​

Collects onboarding parameters, uploads configuration files to your workspace, and creates the onboarding Databricks Job.

databricks labs sdp-meta onboard

Prompts for: Databricks profile, onboarding file path, Bronze/silver dataflowspec table names, environment name, catalog, and schema. After completion, prints and opens the onboarding job URL.

note

If you have cloned the sdp-meta repository, pressing Enter at each prompt accepts the default demo values from the demo/ directory.

deploy​

Deploys a Lakeflow Spark Declarative Pipeline for a given layer and group.

databricks labs sdp-meta deploy

Prompts for: layer, pipeline group, dataflowspec table names, catalog/schema, and cluster configuration. After completion, prints and opens the pipeline URL.

bundle-init​

Scaffolds a new Databricks Asset Bundle (DAB) configured for SDP-META.

databricks labs sdp-meta bundle-init
databricks labs sdp-meta bundle-init --quickstart

The generated bundle includes databricks.yml, resources/variables.yml, resources/sdp_meta_pipelines.yml, resources/sdp_meta_onboarding_job.yml, and notebooks/init_sdp_meta_pipeline.py.

bundle-prepare-wheel​

Builds the databricks-labs-sdp-meta wheel from source and uploads it to a Unity Catalog Volume.

databricks labs sdp-meta bundle-prepare-wheel
FlagDescription
--volume-pathTarget Volume path, e.g. /Volumes/my_catalog/my_schema/my_volume/
--profileDatabricks CLI profile to use

After running, update sdp_meta_dependency in resources/variables.yml to the uploaded wheel path.

bundle-add-flow​

Adds a new data flow entry to an existing bundle's onboarding configuration from UC tables, UC Volumes, Kafka topics, or CSV inventory files.

databricks labs sdp-meta bundle-add-flow

bundle-add-pipeline​

Adds another bronze-only, silver-only, split bronze/silver, or combined bronze/silver topology to resources/sdp_meta_pipelines.yml. Each topology has its own data_flow_group and may override its bronze and silver target schemas. Existing single-topology bundles remain valid.

databricks labs sdp-meta bundle-add-pipeline

The command adds the pipeline resources and corresponding tasks to the pipelines job. In split mode, it also wires the silver task to depend on the matching bronze task. It widens the onboarding job when another layer is introduced; later bundle-add-flow calls for that group inherit the pipeline's layer and target schemas.

Each data_flow_group and layer may have exactly one pipeline owner. A bronze owner and silver owner may intentionally share a group as a split topology, but duplicate bronze, duplicate silver, and combined/split overlap are rejected before files are written. The preflight checks the complete merged topology—including every bundle target override—so a pre-existing ownership conflict must be repaired before another pipeline can be added. Dry-run performs the same preflight checks without writing files; it returns a non-zero result when the requested group has no onboarding rows or the merged topology would be invalid.

bundle-validate​

Validates every configured pipeline independently, checks its layer-specific dataflowspec table and group settings, and verifies that every pipeline is referenced exactly once by the pipelines job. Split silver pipelines must depend on a bronze pipeline for the same group. Target-specific variable overrides are resolved when --target is selected.

databricks labs sdp-meta bundle-validate

Returns a non-zero exit code when validation fails.

Interactive command

databricks labs sdp-meta bundle-validate currently prompts for its bundle directory and target. Until non-interactive flags are available, CI should use databricks bundle validate --target <target> and run the SDP-META-specific validator locally before merging topology changes. See CI/CD Automation.

mcp​

Starts the SDP-META MCP server using stdio transport.

databricks labs sdp-meta mcp

See the MCP Server guide for setup instructions.