DAIS DEMO

DAIS 2023 DEMO:

DAIS 2023 Session Recording

This demo showcases DLT-META’s capabilities of creating Bronze and Silver DLT pipelines with initial and incremental mode automatically.

  • Customer and Transactions feeds for initial load
  • Adds new feeds Product and Stores to existing Bronze and Silver DLT pipelines with metadata changes.
  • Runs Bronze and Silver DLT for incremental load for CDC events

Steps to launch DAIS demo in your Databricks workspace:

  1. Launch Terminal/Command promt

  2. Install Databricks CLI

    • Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:
    databricks auth login --host WORKSPACE_HOST

  3.  git clone https://github.com/databrickslabs/dlt-meta.git 

  4.  cd dlt-meta

  5. Set python environment variable into terminal

    dlt_meta_home=$(pwd)

    export PYTHONPATH=$dlt_meta_home

  6. Run the command python demo/launch_dais_demo.py --username=<<your databricks username>> --source=cloudfiles --uc_catalog_name=<<uc catalog name>> --cloud_provider_name=aws --dbr_version=13.3.x-scala2.12 --dbfs_path=dbfs:/dais-dlt-meta-demo-automated_new

    • cloud_provider_name : aws or azure or gcp

    • db_version : Databricks Runtime Version

    • dbfs_path : Path on your Databricks workspace where demo will be copied for launching DLT-META Pipelines

    • you can provide --profile=databricks_profile name in case you already have databricks cli otherwise command prompt will ask host and token.

      • 6a. Databricks Workspace URL:
      • Enter your workspace URL, with the format https://.cloud.databricks.com. To get your workspace URL, see Workspace instance names, URLs, and IDs.
      • 6b. Token:

        • In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the drop down.

        • On the Access tokens tab, click Generate new token.

        • (Optional) Enter a comment that helps you to identify this token in the future, and change the token’s default lifetime of 90 days. To create a token with no lifetime (not recommended), leave the Lifetime (days) box empty (blank).

        • Click Generate.

        • Copy the displayed token

        • Paste to command prompt