Silver Fanout Demo

Silver Fanout Demo

• This demo will perform following steps
  • Showcase onboarding process for silver fanout pattern
  • Run onboarding for the bronze cars table, which contains data from various countries.
  • Run onboarding for the silver tables, which have a where_clause based on the country condition in silver_transformations_cars.json.
  • Run Bronze for cars tables
  • Run onboarding for the silver tables, fanning out from the bronze cars tables to country-specific tables such as cars_usa, cars_uk, cars_germany, and cars_japan.

Steps:

1. Launch Terminal/Command prompt

2. Install Databricks CLI

   • Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:

   databricks auth login --host WORKSPACE_HOST
   

3.  git clone https://github.com/databrickslabs/dlt-meta.git 
   

4.  cd dlt-meta
   

5. Set python environment variable into terminal

   dlt_meta_home=$(pwd)
   

   export PYTHONPATH=$dlt_meta_home
   
   

6. Run the command python demo/launch_silver_fanout_demo.py --source=cloudfiles --uc_catalog_name=<<uc catalog name>> --cloud_provider_name=aws --dbr_version=15.3.x-scala2.12 --dbfs_path=dbfs:/dais-dlt-meta-silver-fanout

   • cloud_provider_name : aws or azure

   • db_version : Databricks Runtime Version

   • dbfs_path : Path on your Databricks workspace where demo will be copied for launching DLT-META Pipelines

   • you can provide --profile=databricks_profile name in case you already have databricks cli otherwise command prompt will ask host and token.

   • • 6a. Databricks Workspace URL:
   • • Enter your workspace URL, with the format https://.cloud.databricks.com. To get your workspace URL, see Workspace instance names, URLs, and IDs.
   • • 6b. Token:

       • In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the drop down.

       • On the Access tokens tab, click Generate new token.

       • (Optional) Enter a comment that helps you to identify this token in the future, and change the token’s default lifetime of 90 days. To create a token with no lifetime (not recommended), leave the Lifetime (days) box empty (blank).

       • Click Generate.

       • Copy the displayed token

       • Paste to command prompt