Changelog
v0.1.1
New Features
- Multiple pipeline topologies per DAB bundle —
bundle-add-pipelineadds independently configured bronze, silver, split, or combined pipelines with their own data-flow groups and target schemas.bundle-validatevalidates each pipeline and its job wiring independently. (Issue #446) - Auto Loader schema-evolution demos — focused and at-scale demos exercise additive schema evolution, strict-mode failure behavior, and multi-table onboarding from Unity Catalog Volumes. (PR #490)
Fixes
- Databricks SDK compatibility floor — runtime and App dependencies now require
databricks-sdk>=0.138.0,<1, and CI tests serverless onboarding environment models against exactly that minimum. (Issue #457) - [Databricks App] Asynchronous demos and safe concurrent progress tracking — onboarding, pipeline deployment, and demo workflows now run asynchronously, including the featured 100-table Auto Loader demo. Multiple runs can be tracked from one browser session, with retained completion status and curated Databricks job or pipeline links. Bundled onboarding entries launch their supported demo workflow when available, while unsupported bundles continue with standard onboarding. Browser responses no longer expose raw backend exception or subprocess details. (Issue #491, Issue #508)
- Quarantine-only DQE flows — bronze and silver primary targets are now declared when a flow contains only
expect_or_quarantinerules, allowing primary and quarantine outputs to be created and validated correctly. (Issue #496) - Multi-source Auto CDC expectations —
expect,expect_or_drop, andexpect_or_failflows now create their streaming target correctly. Unsupportedexpect_or_quarantinetogether withcdc_apply_changes_flowsnow fails early with an actionable validation error rather than failing while the pipeline runs. (Issue #503) - Documentation dependency security — upgraded Docusaurus and remediated high and critical npm findings in the documentation build dependency graph.
- Data-quality expectations without quarantine targets — bronze and silver flows can apply DQ expectations when no quarantine table is configured, without referencing a missing quarantine target. (PR #487)
- Databricks App deployment packaging — deployment staging now includes the
examples/wheel-build inputs on both macOS/Linux and Windows, preventing remote App startup failures while building the SDP-META wheel. - DAB pipeline group ownership —
bundle-add-pipelineandbundle-validatereject duplicate ownership of the same data-flow group and layer, including conflicts introduced through target overrides. (Issue #458) - Customized DAB validation —
bundle-validatenow honors default and environment-selected targets, accepts transitive bronze-to-silver task dependencies, and reports malformed targets or dependency cycles cleanly. (Issue #459) - Mixed snapshot and non-snapshot pipelines — a layer-level snapshot callback no longer suppresses input views for CloudFiles and other non-snapshot specs. (Issue #443)
- Append-flow source metadata — CloudFiles append flows now serialize and select source metadata correctly. (Issue #444)
- Append-flow custom transformations — custom transformation functions now run for append-flow inputs as well as primary inputs. (Issue #445)
- Legacy dataflow-spec schema evolution — append onboarding adds current bronze and silver fields to v0.0.10 Delta spec tables before merging, preserving existing rows and audit fields. (Issue #460)
- Legacy Python wheel task entry point — the
dlt-metacompatibility package restores thegroup_1:runentry point used by v0.0.10 Databricks Python wheel tasks. (Issue #455) - Legacy workspace configuration keys — v0.0.10
dlt_meta_*workspace keys migrate to theirsdp_meta_*equivalents with deprecation warnings and deterministic precedence. (Issue #456) - Compat shim startup — importing the
dlt-metacompatibility package no longer prints startup tracebacks or fails immediately on environments without pyspark; unavailable runtime dependencies now produce an actionable error only when SDP-META functionality is accessed. - Onboarding PyPI dependency —
databricks labs sdp-meta onboardnow installs the publisheddatabricks-labs-sdp-metapackage instead of the nonexistentsdp-metapackage, preventing job-environment setup failures. - DAB wheel packaging and quickstart onboarding — installed wheels now include the full DAB template, generated bronze flows include required quarantine defaults when DQE is configured, and serverless runners force a clean dependency reinstall.
Compatibility warning
- Python 3.10 is now the minimum supported interpreter — the required
databricks-sdk>=0.138.0supports Python 3.10 and newer. Upgrade Python 3.8 and 3.9 environments before installing either v0.1.1 distribution. - Append-flow transforms now run on every input — v0.1.1 applies
bronze_custom_transform_funcandsilver_custom_transform_funcseparately to primary and append-flow DataFrames. This is the intended fix for Issue #445, but transforms that assume primary-only columns can fail or change append-flow output. Validate every append source and normalize all inputs to one target-compatible schema before upgrading. - Use the official persisted-spec readers during upgrades —
DataflowSpecUtils.get_bronze_dataflow_spec()andDataflowSpecUtils.get_silver_dataflow_spec()backfill the registered compatibility fields added since legacy schema versions. Direct dataclass construction from legacy Spark rows does not. See the v0.1.1 migration guidance and upgrade checklist.
v0.1.0
New Features
- Windows PowerShell deploy script —
scripts/deploy_app.ps1is a native port ofscripts/deploy_app.shusingrobocopy+ thedatabricksCLI. No Git Bash / WSL / Python required. Mirrors the bash flow (stage → sync → deploy) and adds a CRLF → LF normalization pass sostart.shreaches the Linux App container with LF endings. Uses-DatabricksProfileto avoid shadowing PowerShell's built-in$PROFILEvariable. See databricks_app/WINDOWS_DEPLOY.md. - Apps UI + Git folder deploy path — click-only alternative documented in databricks_app/UI_GIT_DEPLOY.md. Create a Databricks Git folder pointing at the repo, aim the App at
databricks_app/only, andstart.sh's Mode B clones the fullsdp-metarepo into/tmp/sdp-metaat container start. - Repo-wide line-ending policy —
.gitattributespins LF for shell/Python/YAML/JSON/template files, CRLF for.bat/.cmd, binary for images/archives. Prevents thebad interpreter: /bin/bash\rApp-container crash from recurring via any tooling path. - Automatic liquid clustering (
cluster_by_auto) for bronze and silver tables. When set totrue, Databricks automatically determines the optimal clustering columns. Works alongside explicitcluster_byto define initial keys followed by automatic optimization. Supported forbronze_cluster_by_auto,bronze_quarantine_table_cluster_by_auto, andsilver_cluster_by_auto. (Issue #238) - MCP Server support — opt-in
mcpCLI command (databricks labs sdp-meta mcp) exposes sdp-meta over stdio so MCP-capable clients (Claude Code, Cursor, Claude Desktop) can drive scaffolding and inspection. Install withpip install databricks-labs-sdp-meta[mcp]. - Declarative Automation Bundle (DAB) template with
bundle-init --quickstartzero-prompt fast path for instant bundle scaffolding. New CLI commands:bundle-init,bundle-add-flow,bundle-prepare-wheel,bundle-validate. Packaged template includes onboarding job, Lakeflow Spark Declarative Pipelines, runner notebook, and four flow-generation recipes. - Row filter support —
where_clausein silver transformations files for pipeline-time row filtering, with coverage for multi-source CDC flows. - Multi-source AUTO CDC — multiple CDC sources can now feed into a single target via
create_auto_cdc_flow. - End-to-end YAML support — onboarding, DQE rules, silver transformations, and packaged demos all accept YAML in addition to JSON.
build-and-upload-whlflag foronboardanddeploy— builds the local sdp-meta wheel, uploads it to a UC volume, and bakes the path into the runner notebook's%pip install(avoids needing PyPI access on the pipeline cluster).- Databricks App refactor — monolithic
app.pysplit intoroutes/(8 blueprints) +services/onboarding/helpers, input validation hardened, renamedlakehouse_app→databricks_app, PyPI install option added, UC preflight probe surfaces requiredGRANTSQL before demos. - Docs site migrated to Docusaurus 3 — 34 pages across Getting Started, Concepts, Reference, Guides, Operations, and Contributing sections.
- DAB conf staging for serverless onboarding jobs — new
stage_confwheel entry point stages bundleconf/files to a Unity Catalog Volume and rewrites${workspace.file_path}/confreferences before onboarding, so DAB-deployed onboarding jobs work on serverless compute. (PR #350) - Agent Skill for SDP-META — added
skills/sdp-meta/with workflow guidance and references for onboarding specs, CLI/DAB usage, MCP tools, and a zero-to-running walkthrough. (PR #356) - Databricks App documentation refresh — added app screenshots, a documentation map, clearer service-principal permission steps, post-deploy guidance, and matching Docusaurus App guide updates. (PR #352)
Breaking Changes
- PyPI package renamed:
dlt-meta→databricks-labs-sdp-meta - CLI commands renamed:
databricks labs dlt-meta→databricks labs sdp-meta - Python imports changed:
from dlt_meta import ...→from databricks.labs.sdp_meta import ... - Main class renamed:
DLTMeta→SDPMeta - Source layout changed: flat
src/→src/databricks/labs/sdp_meta/namespace package - Lakeflow Spark Declarative Pipelines API: DLT decorators/APIs migrated to
pyspark.pipelines. Update references that import fromdltto use the newpyspark.pipelinesmodule. (Issue #274) quarantine_tablefield: Renamedquarantine_table_nametoquarantine_tablein dataflow specs for naming consistency. (Issue #243)
Bug Fixes & Improvements
- Git portability for DAB template filenames — renamed three template files under
src/databricks/labs/sdp_meta/templates/dab/template/{{.bundle_name}}/conf/(onboarding.*.tmpl,silver_transformations.*.tmpl,dqe/example_table/bronze_expectations.*.tmpl) to use Gotext/templatebacktick string literals instead of double-quoted ones. The literal"in the previous filenames clashed with Git'score.protectNTFS=true(default on all platforms since Git 2.22, 2019) and causederror: invalid pathon checkout. Functionally identical todatabricks bundle init. - Security: Replaced unsafe
eval()onuc_enabledwidget with a strict parser. (Issue #260) - Performance: O(N+M) schema modification for wide tables in CDC flows (was previously O(N×M)). (Issue #284)
- Fixed cross-platform file URI handling in CLI; updated cloudFiles demo clustering metadata.
- Fixed SCD Type 2 processing; renamed demo tables to
sdp_meta. - Switched demo paths from DBFS to UC Volumes.
- Renamed references from "Lakeflow Declarative Pipelines" to "Lakeflow Spark Declarative Pipelines".
- DAB quickstart is more robust: validated overrides, clearer
output-dirsemantics, recovered bare--quickstartparsing, and scaffold version stamping. (PR #350) - Demo launchers and integration tests now tolerate both
schema=and legacytarget=forms forpipelines.create. (PR #354) - MCP server path/profile handling was hardened with root confinement, symlink escape checks, profile validation, and stricter
bundle_add_flowidentifiers. (PR #350) - Databricks App terminal output now escapes CLI stdout/stderr before ANSI colorization to prevent injected HTML. (PR #350)
- Onboarding fixes cover quarantine table comments, row-filter validation, snapshot source path validation, non-UC silver/multi-source CDC errors, typed
dataflowIdsfiltering, and SCD Type 2 explicit-schema fields. (PR #350)
Backward Compatibility
The dlt-meta compatibility wrapper package re-exports all symbols and forwards CLI commands. Existing code continues to work with deprecation warnings. src.* imports are supported via a sys.modules shim (removed in v0.2.0).
See the Migration guide for full details.
v0.0.10
Breaking Changes
- DPM mode removed: Pipelines using the Legacy Publishing Mode (DPM) flag must be migrated to the default publishing mode before upgrading. Follow Migrate to the default publishing mode. This migration is irreversible.
- Multi-level namespace qualifiers: Custom schema qualification in table names is no longer supported. Tables must be created without database qualifiers in the pipeline context.
invoke_dlt_pipelineargument names: Method arguments now require layer-specific prefixes (bronze_orsilver_). Replacecustom_transform_funcwithbronze_custom_transform_func, andnext_snapshot_and_versionwithbronze_next_snapshot_and_version.
Enhancements
apply_changes_from_snapshotsupport in the silver layer- Databricks App UI for onboarding and deploy commands
- Non-Delta sink support (Delta and Kafka sinks) via
bronze_sink/silver_sink - Quarantine support in the silver layer for data quality rules
- Table comments, column comments, and
cluster_bysupport - Catalog support for
sourceDetailsandtargetDetails - DBDemos integration
- YAML support for onboarding files
- Multiple column support for
create_auto_cdc_flow - Custom transformations support for Kafka and Delta sources
Migration
See the Migration guide.
v0.0.9
Enhancements
apply_changes_from_snapshotAPI support in the bronze layerappend_flowAPI support for the silver layer- File metadata column support for Autoloader
- Bring-your-own custom transformation support
- UC Volume and serverless support for CLI, integration tests, and demos
- Bronze/silver pipeline chaining into a single Declarative Pipeline
- Liquid clustering support
- Silver fanout demo
v0.0.8
Enhancements
append_flowAPI support for bronze and silver layers- File metadata columns for Autoloader
- Custom transformation support
- Silver fanout demo and unit tests
v0.0.7
Enhancements
- Fixed mismatched key:
read_dlt_delta()updated to usesource_databaseinstead ofdatabase
v0.0.6
Enhancements
- Migrated to
create_streaming_tableAPI from deprecatedcreate_streaming_live_table - Data quality support for the silver layer
- Unity Catalog integration test framework
v0.0.5
New Features
- Unity Catalog support
- Databricks Labs CLI support with
onboardanddeploycommands
v0.0.4
Bug Fixes
- New
eventhub.accessKeySecretNameoption for Event Hubs source
v0.0.3
Bug Fixes
- Infer datatypes from
sequence_byfor__START_AT/__END_ATin theapply_changesAPI
v0.0.2
New Features
- Table properties support for bronze, quarantine, and silver tables
track_history_columnsupport inapply_changes- Delta source support
- Bronze/silver onboarding validation
v0.0.1
Initial release.