Append FLOW Eventhub Demo

Append FLOW Autoloader Demo:

  • Read from different eventhub topics and write to same target tables using dlt.append_flow API

Steps:

  1. Launch Terminal/Command prompt

  2. Install Databricks CLI

    • Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:
    databricks auth login --host WORKSPACE_HOST
    
  3.  git clone https://github.com/databrickslabs/dlt-meta.git 
    
  4.  cd dlt-meta
    
  5. Set python environment variable into terminal

    dlt_meta_home=$(pwd)
    
    export PYTHONPATH=$dlt_meta_home
    
  6. Eventhub

  • Needs eventhub instance running

  • Need two eventhub topics first for main feed (eventhub_name) and second for append flow feed (eventhub_name_append_flow)

  • Create databricks secrets scope for eventhub keys

    •       commandline databricks secrets create-scope eventhubs_dltmeta_creds
      
    •       databricks secrets put-secret --json '{
                "scope": "eventhubs_dltmeta_creds",
                "key": "RootManageSharedAccessKey",
                "string_value": "<<value>>"
                }' 
      
  • Create databricks secrets to store producer and consumer keys using the scope created in step 2

  • Following are the mandatory arguments for running EventHubs demo

    • cloud_provider_name: Cloud provider name e.g. aws or azure
    • dbr_version: Databricks Runtime Version e.g. 15.3.x-scala2.12
    • uc_catalog_name : unity catalog name e.g. ravi_dlt_meta_uc
    • dbfs_path: Path on your Databricks workspace where demo will be copied for launching DLT-META Pipelines e.g. dbfs:/tmp/DLT-META/demo/
    • eventhub_namespace: Eventhub namespace e.g. dltmeta
    • eventhub_name : Primary Eventhubname e.g. dltmeta_demo
    • eventhub_name_append_flow: Secondary eventhub name for appendflow feed e.g. dltmeta_demo_af
    • eventhub_producer_accesskey_name: Producer databricks access keyname e.g. RootManageSharedAccessKey
    • eventhub_consumer_accesskey_name: Consumer databricks access keyname e.g. RootManageSharedAccessKey
    • eventhub_secrets_scope_name: Databricks secret scope name e.g. eventhubs_dltmeta_creds
    • eventhub_port: Eventhub port
  1.  python3 demo/launch_af_eventhub_demo.py --cloud_provider_name=aws --dbr_version=15.3.x-scala2.12 --dbfs_path=dbfs:/tmp/DLT-META/demo/ --uc_catalog_name=ravi_dlt_meta_uc --eventhub_name=dltmeta_demo --eventhub_name_append_flow=dltmeta_demo_af --eventhub_secrets_scope_name=dltmeta_eventhub_creds --eventhub_namespace=dltmeta --eventhub_port=9093 --eventhub_producer_accesskey_name=RootManageSharedAccessKey --eventhub_consumer_accesskey_name=RootManageSharedAccessKey --eventhub_accesskey_secret_name=RootManageSharedAccessKey --uc_catalog_name=ravi_dlt_meta_uc
    

af_eh_demo.png