Integration Tests

Run Integration Tests

  1. Initial steps
  • Prerequisite: Datatbricks CLI installed as given here
    git clone https://github.com/databrickslabs/dlt-meta.git
    cd dlt-meta
    python -m venv .venv
    source .venv/bin/activate
    pip install databricks-sdk
    dlt_meta_home=$(pwd)
   export PYTHONPATH=$dlt_meta_home

  1. Run integration test against cloudfile or eventhub or kafka using below options: If databricks profile configured using CLI then pass --profile <profile-name> to below command otherwise provide workspace url and token in command line

    • 2a. Run the command for cloudfiles python integration_tests/run_integration_tests.py --source=cloudfiles uc_catalog_name=<<>>

    • 2b. Run the command for eventhub python integration-tests/run_integration_tests.py --cloud_provider_name=azure --dbr_version=15.3.x-scala2.12 --source=eventhub --dbfs_path=dbfs:/tmp/DLT-META/ --eventhub_name=iot --eventhub_secrets_scope_name=eventhubs_creds --eventhub_namespace=int_test-standard --eventhub_port=9093 --eventhub_producer_accesskey_name=producer --eventhub_consumer_accesskey_name=consumer

      • For eventhub integration tests, the following are the prerequisites:
        1. Needs eventhub instance running
        2. Use Databricks CLI, Create databricks secrets scope for eventhub keys (databricks secrets create-scope eventhubs_creds)
        3. Use Databricks CLI, Create databricks secrets to store producer and consumer keys using the scope created in step
      • Following are the mandatory arguments for running EventHubs integration test
        1. Provide your eventhub topic : –eventhub_name
        2. Provide eventhub namespace : –eventhub_namespace
        3. Provide eventhub port : –eventhub_port
        4. Provide databricks secret scope name : –eventhub_secrets_scope_name
        5. Provide eventhub producer access key name : –eventhub_producer_accesskey_name
        6. Provide eventhub access key name : –eventhub_consumer_accesskey_name

    • 2c. Run the command for kafka python3 integration_tests/run_integration_tests.py --cloud_provider_name=aws --dbr_version=15.3.x-scala2.12 --source=kafka --dbfs_path=dbfs:/tmp/DLT-META/ --kafka_topic_name=dlt-meta-integration-test --kafka_broker=host:9092

      • For kafka integration tests, the following are the prerequisites:
        1. Needs kafka instance running
      • Following are the mandatory arguments for running EventHubs integration test
        1. Provide your kafka topic name : –kafka_topic_name
        2. Provide kafka_broker : –kafka_broker

  2. Once finished integration output file will be copied locally to integration-test-output_<run_id>.txt

  3. Output of a successful run should have the following in the file

,0
0,Completed Bronze Lakeflow Declarative Pipeline.
1,Completed Silver Lakeflow Declarative Pipeline.
2,Validating Lakeflow Declarative Pipeline Bronze and Silver Table Counts...
3,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.transactions.
4,Expected: 10002 Actual: 10002. Passed!
5,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.transactions_quarantine.
6,Expected: 7 Actual: 7. Passed!
7,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.customers.
8,Expected: 98928 Actual: 98928. Passed!
9,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.customers_quarantine.
10,Expected: 1077 Actual: 1077. Passed!
11,Validating Counts for Table silver_7d1d3ccc9e144a85b07c23110ea50133.transactions.
12,Expected: 8759 Actual: 8759. Passed!
13,Validating Counts for Table silver_7d1d3ccc9e144a85b07c23110ea50133.customers.
14,Expected: 87256 Actual: 87256. Passed!