Silver Fanout Demo

Apply Changes From Snapshot Demo

• This demo will perform following steps
  • Showcase onboarding process for apply changes from snapshot pattern
  • Run onboarding for the bronze stores and products tables, which contains data snapshot data in csv files.
  • Run Bronze DLT to load initial snapshot (LOAD_1.csv)
  • Upload incremental snapshot LOAD_2.csv version=2 for stores and product
  • Run Bronze DLT to load incremental snapshot (LOAD_2.csv). Stores is scd_type=2 so updated records will expired and added new records with version_number. Products is scd_type=1 so in case records missing for scd_type=1 will be deleted.
  • Upload incremental snapshot LOAD_3.csv version=3 for stores and product
  • Run Bronze DLT to load incremental snapshot (LOAD_3.csv). Stores is scd_type=2 so updated records will expired and added new records with version_number. Products is scd_type=1 so in case records missing for scd_type=1 will be deleted.

Steps:

1. Launch Command Prompt

2. Install Databricks CLI

   • Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:

   databricks auth login --host WORKSPACE_HOST

3.  git clone https://github.com/databrickslabs/dlt-meta.git 

4.  cd dlt-meta

5. Set python environment variable into terminal

   dlt_meta_home=$(pwd)

   export PYTHONPATH=$dlt_meta_home

6.  python demo/launch_acfs_demo.py --uc_catalog_name=<<uc catalog name>>

   • uc_catalog_name : Unity catalog name
   • you can provide --profile=databricks_profile name in case you already have databricks cli otherwise command prompt will ask host and token.