Launch Terminal/Command prompt
Install Databricks CLI
databricks auth login --host WORKSPACE_HOST
git clone https://github.com/databrickslabs/dlt-meta.git
cd dlt-meta
Set python environment variable into terminal
dlt_meta_home=$(pwd)
export PYTHONPATH=$dlt_meta_home
Eventhub
Needs eventhub instance running
Need two eventhub topics first for main feed (eventhub_name) and second for append flow feed (eventhub_name_append_flow)
Create databricks secrets scope for eventhub keys
commandline databricks secrets create-scope eventhubs_dltmeta_creds
databricks secrets put-secret --json '{
"scope": "eventhubs_dltmeta_creds",
"key": "RootManageSharedAccessKey",
"string_value": "<<value>>"
}'
Create databricks secrets to store producer and consumer keys using the scope created in step 2
Following are the mandatory arguments for running EventHubs demo
python3 demo/launch_af_eventhub_demo.py --cloud_provider_name=aws --dbr_version=15.3.x-scala2.12 --dbfs_path=dbfs:/tmp/DLT-META/demo/ --uc_catalog_name=ravi_dlt_meta_uc --eventhub_name=dltmeta_demo --eventhub_name_append_flow=dltmeta_demo_af --eventhub_secrets_scope_name=dltmeta_eventhub_creds --eventhub_namespace=dltmeta --eventhub_port=9093 --eventhub_producer_accesskey_name=RootManageSharedAccessKey --eventhub_consumer_accesskey_name=RootManageSharedAccessKey --eventhub_accesskey_secret_name=RootManageSharedAccessKey --uc_catalog_name=ravi_dlt_meta_uc