Go to your Databricks landing page and select Create a notebook, or click New Icon New in the sidebar and select Notebook. The Create Notebook dialog appears.
In the Create Notebook dialogue, give your notebook a name e.g dlt_meta_pipeline and select Python from the Default Language dropdown menu. You can leave Cluster set to the default value. The Delta Live Tables runtime creates a cluster before it runs your pipeline.
Click Create.
You can add the example dlt pipeline code or import iPython notebook as is.
%pip install dlt-meta
layer = spark.conf.get("layer", None)
from src.dataflow_pipeline import DataflowPipeline
DataflowPipeline.invoke_dlt_pipeline(spark, layer)
Click Jobs Icon Workflows in the sidebar, click the Delta Live Tables tab, and click Create Pipeline.
Give the pipeline a name e.g. DLT_META_BRONZE and click File Picker Icon to select a notebook dlt_meta_pipeline created in step: Create a dlt launch notebook.
Optionally enter a storage location for output data from the pipeline. The system uses a default location if you leave Storage location empty.
Select Triggered for Pipeline Mode.
Enter Configuration parameters e.g.
"layer": "bronze",
"bronze.dataflowspecTable": "dataflowspec table name",
"bronze.group": "enter group name from metadata e.g. G1",
Enter target schema where you wants your bronze/silver tables to be created
Click Create.
Start pipeline: click the Start button on in top panel. The system returns a message confirming that your pipeline is starting