Skip to main content

Installation

DQX can be installed in three ways, depending on how you want to use it:

  • as a standalone library in a Databricks cluster or notebook, for code-level integration into pipelines (see DQX installation as a Library)
  • as a workspace tool via the Databricks CLI, which deploys the configuration-driven profiling, quality-checker, and end-to-end workflows (see DQX installation as a Tool)
  • as DQX Studio, a web UI deployed as a Databricks App for authoring, reviewing, running, and monitoring quality rules through a browser (see DQX Studio installation)
Recommended for a no-code experience

If you want to use DQX without writing code, DQX Studio is the recommended option — it provides a full browser-based UI on top of DQX's profiling and quality-checking capabilities, with no configuration files to maintain.

DQX installation as a Library​

Prerequisites​

Installing DQX in a cluster or notebook​

Install the library via pip in a cluster or notebook:

pip install databricks-labs-dqx

As a best practice, you should always pin the version so that your code is not affected by future changes in the library:

pip install databricks-labs-dqx==0.8.0

DQX also integrates seamlessly with Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). You can add DQX as a library dependency in your bundle configuration to install it on your cluster:

resources:
  jobs:
    my_job:
      # ...
      tasks:
        - task_key: my_task
          # ...
          libraries:
            - pypi:
                package: databricks-labs-dqx==0.8.0

Breaking changes between versions are listed in Github Release Notes under the BREAKING CHANGES! section.

Optional dependencies​

DQX provides optional extras for specific features:

AI-Assisted Quality Checks Generation​

To use AI-assisted rule generation with LLM support, install the llm extras:

pip install 'databricks-labs-dqx[llm]'

This installs additional dependencies including DSPy for LLM integration. See the AI-Assisted Generation Guide for more details.

PII detection capabilities​

To use built-in pii detection rules, install the pii extras:

pip install 'databricks-labs-dqx[pii]'

This installs additional dependencies including Microsoft Presidio library.

Data Contract Support​

To use data contract-based rule generation with ODCS (Open Data Contract Standard) support, install the datacontract extras:

pip install 'databricks-labs-dqx[datacontract]'

This installs the datacontract-cli library for parsing and validating ODCS contracts.

If your data contracts include text-based quality expectations (natural language rules), also install the llm extras:

pip install 'databricks-labs-dqx[datacontract,llm]'

See the Data Contract Quality Rules Generation Guide for more details.

Installing DQX with company-hosted PyPI mirror​

Some enterprises block the public PyPI index and host a company-controlled PyPI mirror. To install DQX while using a company-hosted PyPI mirror for finding its dependencies, add all DQX dependencies to the company-hosted PyPI mirror (see "dependencies" in pyproject.toml) and set the environment variable PIP_INDEX_URL to the company-hosted PyPI mirror URL while installing DQX:

PIP_INDEX_URL="https://url-to-company-hosted-pypi.internal" pip install databricks-labs-dqx

DQX installation as a Tool in a Databricks Workspace​

If you install DQX via PyPI and use it purely as a library, you don’t need to pre-install DQX in the workspace. However, installing DQX in the workspace offers additional benefits such as profiling and quality checker workflows allowing for no-code quality checking, a pre-configured dashboard, and convenient configuration management. For a fully no-code experience driven by a browser-based UI, DQX Studio is the recommended option.

See the Installation Configuration reference for a complete reference of the options available through the installation wizard.

Prerequisites​

Authenticate Databricks CLI​

Once you install Databricks CLI, authenticate your current machine to your Databricks Workspace:

databricks auth login --host <WORKSPACE_HOST>

To enable debug logs, simply add --debug flag to any command. More about authentication options here.

Install DQX using Databricks CLI​

Install DQX in your Databricks workspace via Databricks CLI:

databricks labs install dqx

As a best practice, you should always pin the version so that your code is not affected by future changes:

databricks labs install dqx@v0.8.0

You'll be prompted to provide several configuration options. You can use standard Databricks CLI options like --profile for authentication.

Breaking changes between versions are listed in Github Release Notes under the BREAKING CHANGES! section.

Execution Environment
  • Make sure to have Databricks CLI v0.241 or later installed locally to avoid encountering the error: ModuleNotFoundError: No module named 'pyspark'.
  • You must have Python 3.10 or later to install DQX using the Databricks Labs CLI. The Databricks Labs CLI relies on the user's Python installation to create a virtual environment and install the required DQX packages. The packages (e.g. pyspark) don't have to be installed locally before running the CLI.
  • Running the Databricks CLI from within a Databricks workspace is not supported. The CLI is designed for use from a local machine or a separate compute environment, not directly inside Databricks.
  • The CLI supports the private PyPI package index. If you encounter SSL-related errors, you may need to install OpenSSL on your system or reinstall Python.

Install a specific version of DQX in your Databricks workspace via Databricks CLI (e.g. version 0.8.0):

databricks labs install dqx@v0.8.0

You'll be prompted to select a configuration profile created by databricks auth login command, and other configuration options.

The cli command will install the following components in the workspace installation folder:

  • A Python wheel file with the library packaged.
  • DQX configuration file (config.yml).
  • Profiling workflow for generating quality rule candidates (not scheduled by default eliminating cost concerns)
  • Quality checking workflow to apply quality rules to the data (not scheduled by default eliminating cost concerns)
  • End-to-end (e2e) workflow to profile input data, generate quality checks and apply them to the data (not scheduled by default eliminating cost concerns)
  • Quality dashboard for monitoring to display information about the data quality issues (not scheduled by default eliminating cost concerns)

During the installation you will be prompted to select whether to use serverless clusters or not. It is recommended to use serverless clusters for the workflows, as it allows for automated cluster management and scaling. If serverless clusters are not used, a default cluster configuration will be used for the workflows. Alternatively, you can override the cluster configuration for each workflow in the config.yml file after the installation, to provide existing clusters to use.

Installation Options​

DQX offers flexible installation options. By default, DQX is installed in the user home directory under /Users/<user>/.dqx. You can also install DQX in a global folder or any custom workspace folder.

Environment Variable Override:

You can force global or user installation using the 'DQX_FORCE_INSTALL' environment variable:

  • DQX_FORCE_INSTALL=global databricks labs install dqx: forces installation to /Applications/dqx
  • DQX_FORCE_INSTALL=user databricks labs install dqx: forces installation to /Users/<user>/.dqx (default behaviour)

Custom Workspace Folder:

If you provide a custom path during installation (e.g., /Shared/dqx-team or /Users/shared-user/dqx-project), DQX will be installed there. You will be prompted to optionally enter a workspace path when installing DQX.

The custom folder installation is required when using Group Assigned cluster, as the concept of a user home directory does not exist in this setup.

Complete Profiling Guide

If a custom folder is provided during installation, the installation folder will take precedence over any environment variables (e.g. DQX_FORCE_INSTALL).

Configuration file​

DQX configuration file can contain multiple run configurations for different pipelines or projects, each defining specific input, output and quarantine locations, etc. The configuration file is created in the installation directory depending on installation options (see above):

  • User home (default): /Users/<user>/.dqx/config.yml
  • Global: /Applications/dqx/config.yml (when DQX_FORCE_INSTALL=global)
  • Custom folder: <your-installation-folder>/config.yml (when provided during installation)

A "default" run configuration is created during the installation. When DQX is upgraded, the configuration is preserved. The configuration can be updated / extended manually by the user after the installation. Each run config defines configuration for one specific input and output location.

Open the configuration file:

databricks labs dqx open-remote-config

You can add additional run configurations (under run_configs field) or update the default run configuration after the installation by editing the config.yml file. See the example config below with all configuration options:

log_level: INFO
version: 1

serverless_clusters: true           # <- whether to use serverless cluster for workflows or not (default is true)

# Using serverless clusters is recommended as it allows for automated cluster management and scaling.
# Below override clusters and spark conf is only applicable if `serverless_clusters` is set to `false`

profiler_override_clusters:         # <- optional dictionary mapping job cluster names to existing cluster IDs
  default: 0709-132523-cnhxf2p6     # <- existing cluster Id to use
profiler_spark_conf:                # <- optional spark configuration to use for the profiler workflow
  spark.sql.ansi.enabled: true      # <- example setting
profiler_max_parallelism: 4         # <- max parallelism for profiling multiple tables

quality_checker_override_clusters:  # <- optional dictionary mapping job cluster names to existing cluster IDs
  default: 0709-132523-cnhxf2p6     # <- existing cluster Id to use
quality_checker_spark_conf:         # <- optional spark configuration to use for the quality checker workflow
  spark.sql.ansi.enabled: true      # <- example setting
quality_checker_max_parallelism: 4  # <- max parallelism for quality checking multiple tables

e2e_override_clusters:              # <- optional dictionary mapping job cluster names to existing cluster IDs
  default: 0709-132523-cnhxf2p6     # <- existing cluster Id to use
e2e_spark_conf:                     # <- optional spark configuration to use for the end to end workflow
    spark.sql.ansi.enabled: true    # <- example setting

anomaly_override_clusters:          # <- optional dictionary mapping job cluster names to existing cluster IDs (anomaly-trainer workflow)
  default: 0709-132523-cnhxf2p6     # <- existing cluster Id to use
anomaly_spark_conf:                 # <- optional spark configuration to use for the anomaly-trainer workflow
  spark.sql.ansi.enabled: true      # <- example setting

extra_params:                       # <- optional extra parameters to pass to the workflows
    result_column_names:
        errors: dq_errors           # <- default is "_errors"
        warnings: dq_warnings       # <- default is "_warnings"
    user_metadata:
        custom_metadata: custom_value # <- optional user metadata to be added to the results

llm_config:                         # <- configuration for the llm-assisted features
  model:
    model_name: "databricks/databricks-claude-sonnet-4-5"
    api_key: xxx                    # <- optional API key for the model as secret in the format: secret_scope/secret_key. Not required by foundational models
    api_base: xxx                   # <- optional API base for the model as secret in the format: secret_scope/secret_key. Not required by foundational models

run_configs:                        # <- list of run configurations, each run config defines one specific input and output location

- name: default                     # <- unique name of the run config (it can be any string), e.g. input table or job name (use "default" during installation)

  input_config:                     # <- optional input data configuration
    location: s3://iot-ingest/raw   # <- input location of the data (table or cloud path)
    format: delta                   # <- format, required if cloud path provided
    is_streaming: false             # <- whether the input data should be read using streaming (default is false)
    schema: col1 int, col2 string   # <- schema of the input data (optional), applicable if reading csv and json files
    options:                        # <- additional options for reading from the input location (optional)
      versionAsOf: '0'

  output_config:                    # <- optional output data configuration (omit when only the quarantine table is desired)
    location: main.iot.silver       # <- output location (table), used as input for the quality dashboard if quarantine location is not provided
    format: delta                   # <- format of the output table
    mode: append                    # <- write mode for the output table (append or overwrite)
    options:                        # <- additional options for writing to the output table (optional)
      mergeSchema: 'true'
      #checkpointLocation: /Volumes/catalog1/schema1/checkpoint  # <- only applicable if input_config.is_streaming is enabled
    trigger:                        # <- streaming trigger, only applicable if input_config.is_streaming is enabled
      availableNow: true

  quarantine_config:                     # <- optional quarantine data configuration, if specified, bad data is written to quarantine table
    location: main.iot.silver_quarantine # <- quarantine location (table), used as input for quality dashboard
    format: delta                        # <- format of the quarantine table (default: delta)
    mode: append                         # <- write mode for the quarantine table (append or overwrite, default: append)
    options:                             # <- additional options for writing to the quarantine table (optional)
      mergeSchema: 'true'
      #checkpointLocation: /Volumes/catalog1/schema1/checkpoint  # <- only applicable if input_config.is_streaming is enabled
    trigger:                             # <- optional streaming trigger, only applicable if input_config.is_streaming is enabled
      availableNow: true

  metrics_config:                        # <-  optional summary metrics storage
    format: delta
    location: main.iot.dq_metrics
    mode: append

  checks_location: iot_checks.yml        # <- Quality rules (checks) can be stored in a table or defined in JSON or YAML files, located at absolute or relative path within the installation folder or volume file path.

  actions_location: iot_actions.yml      # <- optional. Action definitions (alerting / fail-pipeline) to auto-load and apply after checks. Table, JSON/YAML file (absolute or relative to the installation folder, or volume path), or Lakebase table. Omit to run without actions.
  action_events_location: main.iot.dq_action_events  # <- optional. UC or Lakebase table for action event history and durable alert suppression across runs. Omit to keep suppression state in memory per run.

  anomaly_config:                          # <- optional row anomaly detection configuration (model training via workflow)
    columns: [amount, quantity]            # <- optional, omit to use all supported columns
    model_name: main.iot.orders_monitor    # <- required when using anomaly config, fully qualified
    registry_table: main.iot.dqx_anomaly_models  # <- required when using anomaly config, fully qualified
    # for the full parameter anomaly specification, see the Row Anomaly Detection documentation

  # if wanting to store checks in lakebase table
  # checks_location: dqx.config.checks                        # <- fully qualified Lakebase table for storing quality rules (checks)
  # lakebase_instance_name: my-lakebase-instance              # <- the name of the lakebase instance to use for storing checks
  # lakebase_client_id: 00000000-0000-0000-0000-000000000000  # <- optional service principal client ID to use to connect to Lakebase, if not provided, the caller identity, i.e., user@domain.com, is used
  # lakebase_port: 5432                                       # <- optional port to connect to Lakebase, default is 5432

  custom_check_functions:                               # <- optional mapping of custom check function name to Python file (module) containing check function definition
    my_func: custom_checks/my_funcs.py                  # relative workspace path (installation folder prefix applied)
    my_other: /Workspace/Shared/MyApp/my_funcs.py       # or absolute workspace path
    email_mask: /Volumes/main/dqx_utils/custom/email.py # or UC volume path

  reference_tables:                      # <- optional mapping of reference table names to reference table locations (e.g. required for foreign key check)
    reference_table_1:                   # <- name of the reference table
      input_config:                      # <- input data configuration for the reference table
        format: delta
        location: main.nytaxi.ref
    reference_table_2:                   # <- another reference table
      input_config:
        format: delta
        location: main.nytaxi.ref2       # <- location of the reference table, can be a table or path to files (cloud, volume or workspace path)

  profiler_config:                            # <- profiler configuration
    summary_stats_file: iot_summary_stats.yml # <- relative location within the installation folder of profiling summary stats
    sample_fraction: 0.3                      # <- fraction of data to sample in the profiler (30%)
    sample_seed: 30                           # <- optional seed for reproducible sampling
    limit: 1000                               # <- limit the number of records to profile
    filter: "maintenance_type = 'preventive'" # <- generate profile in a subset of your data source
    criticality: error                        # <- default criticality for generated rules ('error' or 'warn')
    llm_primary_key_detection: false          # <- whether to use llm-assisted pk detection to generate uniqueness check

  checks_user_requirements: business rules description  # <- optional user input for AI-assisted rule generation; only supported if `serverless_clusters` is enabled

  warehouse_id: your-warehouse-id             # <- warehouse id for refreshing dashboard

- name: another_run_config                    # <- unique name of the run config
  ...

Use the —-run-config parameter to specify a particular run configuration when executing DQX Labs CLI commands. If no configuration is provided, the "default" run configuration is used.

Workflows​

Profiling workflow is intended as a one-time operation. It is not scheduled by default, ensuring no costs are incurred.

List all installed workflows in the workspace and their latest run state:

databricks labs dqx workflows

Install DQX on Databricks cluster​

You need to install the DQX package on a Databricks cluster to use it. You can install it either from PYPI or use a wheel file generated during the installation in the workspace.

There are multiple ways to install libraries in a Databricks cluster (see here). For example, you can install DQX directly from a notebook cell as follows:

# Using PYPI package
%pip install databricks-labs-dqx==0.8.0

# Using wheel file, DQX installed for the current user:
%pip install /Workspace/Users/<user-name>/.dqx/wheels/databricks_labs_dqx-*.whl

# Using wheel file, DQX installed globally:
%pip install /Applications/dqx/wheels/databricks_labs_dqx-*.whl

Restart the kernel after the package is installed in the notebook:

# in a separate cell run:
dbutils.library.restartPython()

DQX also integrates seamlessly with Declarative Automation Bundles (DABs). You can add DQX as a library dependency in your bundle configuration (either by using PYPI package or a wheel file) to install it on your cluster:

resources:
  jobs:
    my_job:
      # ...
      tasks:
        - task_key: my_task
          # ...
          libraries:
            # install from wheel file
            - whl: /Workspace/Users/<user-name>/.dqx/wheels/databricks_labs_dqx-*.whl
            # or install from pypi
            #- pypi:
            #    package: databricks-labs-dqx==0.8.0

Installing DQX with company-hosted PyPI mirror​

Some enterprises block the public PyPI index and host a company-controlled PyPI mirror. To install DQX while using a company-hosted PyPI mirror for finding its dependencies, add all DQX dependencies to the company-hosted PyPI mirror (see "dependencies" in pyproject.toml) and set the environment variable PIP_INDEX_URL to the company-hosted PyPI mirror URL while installing DQX:

PIP_INDEX_URL="https://url-to-company-hosted-pypi.internal" databricks labs install dqx

During DQX installation as a tool in the workspace reply yes to the question "Does the given workspace block Internet access"?

If the installation host has no access to GitHub, then dqx installation will not be able to download the files locally and will fail. In order to address that, follow the steps below

  • install dqx on a host which has access to github using the instruction above
  • zip the installation from ~/.databricks/labs/dqx
  • copy the zip file to the target host and unzip Now the installation can be done in offline mode (ensure Databricks CLI is upgraded to version v0.244.0 or higher)
PIP_INDEX_URL="https://url-to-company-hosted-pypi.internal" databricks labs install dqx --offline=true

Upgrade DQX in the Databricks workspace​

Verify that DQX is installed:

databricks labs installed

Upgrade DQX via Databricks CLI:

databricks labs upgrade dqx

Uninstall DQX from the Databricks workspace​

Uninstall DQX via Databricks CLI:

databricks labs uninstall dqx

Databricks CLI will confirm a few options:

  • Whether you want to remove all DQX artifacts from the workspace or not. Defaults to 'no'.

DQX Studio installation​

DQX Studio is the no-code Databricks App that exposes DQX's profiling, rule authoring, approval workflow, scheduled runs, and results tracking through a browser. It is deployed to your Databricks workspace as a Databricks App using a Declarative Automation Bundle (DAB, formerly known as Databricks Asset Bundle).

Coming soon: Databricks Marketplace

We plan to publish DQX Studio to the Databricks Marketplace for one-click installation. Until then, deploy it with the bundle steps below.

Prerequisites​

  • Databricks CLI v1.4.0 or later, authenticated against your workspace (see Authenticate Databricks CLI). v1.4.0 is required for the postgres_projects / postgres_roles resources; lifecycle.prevent_destroy on the studio's stateful schemas / volume / Lakebase project needs only v0.268+.
  • Sufficient workspace permissions to deploy the bundle. Workspace admin is the easiest shortcut, but not strictly required — granular per-permission needs (Databricks Apps Can Manage, Lakebase Manager, UC catalog MANAGE, service principal User role, etc.) are listed in the DQX Studio deployment guide § Required permissions. If you don't hold them yourself, ask an admin to grant them.
  • make available locally, and Databricks CLI v1.4.0+ (required for the postgres_projects / postgres_roles bundle resources).
  • App build toolchain — make app-deploy first builds the wheel via make app-build, which needs uv, Node.js 18+ (provides npm; install via brew install node, nvm, or nodejs.org), and yarn (npm install -g yarn). (bun is only required for the optional make app-check type-check — install via curl -fsSL https://bun.sh/install | bash or brew install oven-sh/bun/bun.)
  • Databricks Apps feature enabled on the workspace.
  • User token passthrough enabled for Databricks Apps — DQX Studio uses On-Behalf-Of (OBO) tokens to access Unity Catalog with the end user's identity.
  • Serverless compute enabled on the workspace — the dqx-studio-task-runner job is serverless-only.
  • An existing Unity Catalog catalog where the studio's schemas and volumes will be provisioned. The bundle does not create the catalog itself; make app-deploy provisions the schemas and wheels volume inside the catalog automatically.
  • A SQL warehouse — the bundle creates and manages a dedicated serverless warehouse for the app; you don't supply one.
  • Lakebase Postgres enabled on the workspace if you keep the default backend layout. DQX Studio stores its OLTP state (rules catalog, app settings, RBAC, comments, schedule configs, scheduler bookkeeping) in a Lakebase Postgres project for sub-millisecond reads. The bundle creates the project (and the app SP's Postgres role) for you. If your workspace doesn't have Lakebase, you can opt out and run everything on Delta — see the Hybrid storage backend admonition under the install steps below.

Install DQX Studio using a Declarative Automation Bundle​

  1. Clone the DQX repository (all make app-* commands run from the repo root):

    git clone https://github.com/databrickslabs/dqx.git
    cd dqx

  2. Create a workspace service principal to run the task-runner job (separate from the app's auto-created SP, which cannot be used outside the Apps framework). In the Databricks workspace UI go to Settings → Identity and Access → Service Principals, create a new SP (for example dqx-task-runner-sp), and on its Permissions tab grant your deploying identity the User role. Note the SP's Application ID — you'll reference it as dqx_service_principal_application_id in the bundle.

  3. Update the deploy target in app/databricks.yml. Only two variables are required:

    • catalog_name — Unity Catalog catalog where the studio's schemas and volume live (must already exist).
    • dqx_service_principal_application_id — the task-runner SP you created in step 2.

    The bundle manages its own SQL warehouse and Lakebase project, so there's nothing else to supply:

    targets:
      dev:
        workspace:
          profile: <your-profile>
        variables:
          # Required
          catalog_name: <your-catalog>
          dqx_service_principal_application_id: <your-sp-application-id>

          # Optional — uncomment and override defaults per target as needed
          # admin_group: <your-admin-group>              # default: admins
          # app_name: <your-app-name>                    # default: dqx-studio
          # sql_warehouse_size: <size>                   # default: Small
          # schema_name: <your-schema-name>              # default: dqx_studio

          # Lakebase backend (default: enabled). To disable Lakebase and run all
          # OLTP tables on Delta, remove the postgres_projects / postgres_roles
          # blocks from databricks.yml and set lakebase_endpoint: "-".
          # lakebase_branch: dqx                         # default branch name
          # lakebase_suspend_timeout: "300s"             # scale-to-zero idle window
        presets:
          trigger_pause_status: PAUSED

    See the DQX Studio deployment guide for the full reference of each variable, including security implications of admin_group and when to override per target.

  4. Grant USE CATALOG on your chosen catalog. This is the only manual grant: the bundle applies all schema/volume permissions natively, but it does not manage the (pre-existing) catalog, so it cannot grant catalog-level access. Run this once per catalog (the app SP's client id is shown by databricks apps get dqx-studio after the first deploy):

    GRANT USE CATALOG ON CATALOG <catalog> TO `account users`;
    GRANT USE CATALOG ON CATALOG <catalog> TO `<app-sp-client-id>`;
    GRANT USE CATALOG ON CATALOG <catalog> TO `<task-runner-sp-application-id>`;

  5. Build, deploy, and start the studio in a single command:

    make app-deploy PROFILE=<your-profile> TARGET=<your-target>

    This runs make app-build, databricks bundle deploy, and databricks bundle run in sequence. bundle deploy provisions the schemas, wheels volume, Lakebase Postgres project (+ endpoint + the app SP's Postgres role), the SQL warehouse, the task-runner job, and the Databricks App in dependency order, and applies all Unity Catalog grants natively — there is no post-deploy grant script and no bind step. The stateful resources (schemas, volume, Postgres project) carry lifecycle.prevent_destroy: true. The app connects to the always-present databricks_postgres admin database via the project endpoint (DQX_LAKEBASE_ENDPOINT) and creates its own dqx_studio Postgres schema there on first start.

  6. Open the deployed app from the Apps page in your Databricks workspace.

Stateful storage is destroy-protected

The studio's schemas (dqx_studio, dqx_studio_tmp), wheels volume, and Lakebase Postgres project are declared as bundle resources with lifecycle.prevent_destroy: true (Databricks CLI 0.268+). databricks bundle destroy is blocked from dropping them, so production data survives accidental destroy/replace operations. The app's dqx_studio Postgres schema (inside the databricks_postgres admin database on the Lakebase project) is created at startup and is not itself a bundle resource, but it is protected transitively by the project-level guard — as long as the project survives, the schema and its tables survive. To intentionally tear something down, remove the flag, databricks bundle deployment unbind <key>, then destroy manually.

For the full walkthrough — including step-by-step commands, the USE CATALOG prerequisite, troubleshooting, and target-specific configuration — see the DQX Studio deployment guide.

Hybrid storage backend

DQX Studio splits its data across two physical backends: high-volume append-mostly tables (dq_validation_runs, dq_profiling_results, dq_quarantine_records, dq_metrics) live in Delta Lake because they're written by Spark; transactional tables (rules catalog, app settings, RBAC, comments, schedule configs) live in Lakebase Postgres for fast row-level reads/writes from the FastAPI request handlers.

First-run wheel upload

On its first start, DQX Studio runs database migrations (Delta and, if enabled, Lakebase) and uploads its DQX wheel files to the Unity Catalog volume. If the task-runner job is triggered before the app has fully started at least once, it will fail to find its wheels. Wait for Uploaded databricks_labs_dqx-<version>... in the app logs before triggering any profiler or dry-run jobs. If Lakebase is enabled, also wait for Lakebase OLTP routing enabled before opening the UI — when Lakebase is configured (DQX_LAKEBASE_ENDPOINT non-empty) and init fails, the app refuses to start and the Databricks Apps platform will restart it; silent fallback to Delta would split-brain OLTP writes across two physical stores and is not safe. To run on Delta only, unset DQX_LAKEBASE_ENDPOINT.

Upgrade DQX Studio​

To upgrade DQX Studio, pull the latest changes from the DQX repository (or check out a specific tag), then redeploy with the same command:

git pull
make app-deploy PROFILE=<your-profile> TARGET=<your-target>

Database migrations run automatically on app startup and preserve existing rules, runs, schedules, and configuration. All stateful resources are declared with lifecycle.prevent_destroy: true, so even an accidental databricks bundle destroy won't drop them.

Uninstall DQX Studio​

databricks bundle destroy removes the app, the task-runner job, and the bundle-managed SQL warehouse. The bundle is blocked from dropping the stateful resources (schemas, volume, Lakebase project) by their prevent_destroy flag. The DQX dqx_studio Postgres schema inside databricks_postgres lives below the resource layer DABs models and is therefore unaffected by bundle destroy — drop it manually if you want a fully clean wipe. To fully uninstall:

  1. Edit app/databricks.yml and remove lifecycle.prevent_destroy: true from each of the stateful resources you want to drop.
  2. Unbind the resources so the bundle no longer tracks them, then destroy: 
    cd app
    databricks bundle deployment unbind main_schema -t <your-target>
    databricks bundle deployment unbind tmp_schema  -t <your-target>
    databricks bundle deployment unbind wheels      -t <your-target>
    databricks bundle deployment unbind dqx_studio  -t <your-target>   # the Lakebase project
    databricks bundle destroy -p <your-profile> -t <your-target>
  3. Drop the now-unbound UC resources manually if desired: 
    DROP VOLUME IF EXISTS <catalog>.dqx_studio.wheels;
    DROP SCHEMA IF EXISTS <catalog>.dqx_studio CASCADE;
    DROP SCHEMA IF EXISTS <catalog>.dqx_studio_tmp CASCADE;
  4. Drop the DQX Postgres schema and the Lakebase project if desired: 
    # DQX schema inside ``databricks_postgres`` (connect to the
    # Lakebase project endpoint with psql via ``databricks_postgres``):
    #   DROP SCHEMA IF EXISTS dqx_studio CASCADE;

    # Lakebase project: delete it from Compute → Database Instances in the
    # workspace UI (or via the Databricks CLI postgres project commands).
Destroy protection is intentional

prevent_destroy is what stops a databricks bundle destroy (or a forced replace from a deploy) from wiping your rules, schedules, role mappings, and comments. The multi-step uninstall above is by design — silent data loss is the bigger risk to protect against.

DQX MCP Server installation​

The DQX MCP Server exposes DQX as Model Context Protocol (MCP) tools for AI agents (Genie Code, Cursor, Claude Code). This section covers the prerequisites and deployment; see the guide for architecture, the available tools, configuration, and usage.

Prerequisites​

Before deploying, set up the following. That's all you need — the deploy creates the app, the job and its UC objects, and applies every grant for you, including the catalog-level ones the bundle cannot declare itself.

  1. Databricks CLI 0.279.0+ — check with databricks --version (the bundle uses the direct deployment engine, which needs this version).
  2. Workspace features enabled (ask a workspace admin if any are off): Databricks Apps, Serverless compute, and Unity Catalog.
  3. A Unity Catalog catalog you own (or hold MANAGE on). The deploy grants privileges on this catalog, which only its owner / a MANAGE holder / a metastore admin can do — plain ALL PRIVILEGES is not enough. So a shared, foreign-owned catalog such as main will not work; use one you administer, or ask an admin to create one and make you owner. Why »
  4. A runner service principal — a dedicated workspace service principal the runner job runs as. Create it once (steps below) and pass its Application ID as runner_service_principal_id at deploy.
Fastest path

If you're a workspace admin with your own catalog, you already meet 1–3; just create the runner SP (step 4) and run make mcp-deploy.

Create the runner service principal​

The runner job runs as a dedicated, least-privilege workspace service principal — not the app's auto-created SP (which is app-scoped and cannot be a job's run_as) and not the deploying user. Create it once:

  1. Create a workspace service principal (Settings → Identity and access → Service principals → Add, or the SCIM API). Note its Application ID.

  2. Grant yourself (the deploying identity) the User role on it (Settings → that SP → Permissions). Do this even if you are a workspace admin — admin membership alone does not confer it, and without it bundle deploy fails partway through with:

    Cannot bind the service principal provided in 'run_as' field (<sp-name>) to the job.
    The user creating or updating the job must have 'servicePrincipal.user' role on the
    service principal. (403 PERMISSION_DENIED)

  3. Give the SP the workspace-access entitlement so it can run serverless jobs.

The deploy grants this SP every UC privilege it needs automatically — no manual grants required.

Deploy the MCP server​

1. Authenticate​

databricks auth login --host https://<your-workspace-url> --profile <profile>

2. Deploy​

Pass your catalog and the runner SP Application ID from the prerequisites:

make mcp-deploy PROFILE=<profile> CATALOG=<catalog> RUNNER_SP=<runner-sp-application-id>

That is the whole deploy, in one command. It runs three steps in an order that matters:

  1. bundle deploy creates the schema, both UC volumes, the runner job and the app, and applies every grant on those objects natively.
  2. catalog grants — the bundle cannot grant on your catalog because it does not own it, so the deploy applies those itself (USE CATALOG for your users group, the runner SP and the app's service principal; plus CREATE SCHEMA for the runner SP, which creates each caller's private output schema). The app's service principal only exists once the app does, which is why this comes after step 1.
  3. bundle run starts the app, which publishes the runner wheel to the volume the runner job installs it from.

Granting before the app first starts is what keeps this a single pass — the app already holds the access it needs to publish the wheel, so there is no "grant, then restart" second step. Re-running is safe (idempotent). Add TARGET=<target> for a non-default bundle target.

Bundle variables​

Pass these at deploy time (as --var name=value, or via BUNDLE_VARS= with make mcp-deploy):

VariableDescriptionDefault
catalog_nameRequired. Catalog the bundle creates its schema and volumes in, and where per-user output schemas live. Passed into the app config as DQX_CATALOG — no secret scope (a catalog name isn't sensitive; UC grants govern access).""
runner_service_principal_idRequired. Application id of the workspace SP the runner job runs as (see Create the runner service principal). The placeholder default is not a real SP. With make mcp-deploy, pass RUNNER_SP= instead — it sets this variable and grants the SP on the catalog, which passing the variable directly does not.00000000-…
name_prefixPrefix for all resource names (app + job). Override to deploy an isolated second copy.mcp-dqx
tmp_schema_nameSchema holding the temp views, the results volume and the runner-wheel volume. Override (together with name_prefix) for a fully isolated second deployment. Must be a valid identifier — no -.dqx_mcp_tmp
users_groupGroup granted CAN_USE on the app, plus USE CATALOG and temp-schema access.account users
runner_wheel_filenameFilename of the runner wheel the app publishes and the runner job installs. Bump together with the version in mcp-server/runner/pyproject.toml — a unit test enforces that they match.dqx_mcp_runner-<version>-py3-none-any.whl

For example, to override the users group:

databricks bundle deploy --var users_group="users" --profile <profile>
Don't have make (e.g. on Windows)? Run these Databricks CLI commands instead

make mcp-deploy is a shortcut for the commands below — only the Databricks CLI is needed, so they work in PowerShell or cmd as well as a POSIX shell. Run them from the mcp-server/ directory. The order matters: the app's service principal does not exist until the app is created, and it needs USE CATALOG before it first starts, or it cannot publish the runner wheel.

# 1. Deploy the bundle: schema, both volumes, the runner job and the app, with their native grants.
databricks bundle deploy --var catalog_name=<catalog> --var runner_service_principal_id=<runner-sp-application-id> --profile <profile>

# 2. Look up the app's service principal (created with the app in step 1).
databricks apps get mcp-dqx --profile <profile> -o json
#    -> copy "service_principal_client_id" from the output

# 3. Grant the catalog privileges the bundle cannot (it does not own your catalog).
#    Idempotent, and no SQL warehouse needed. Replace the three principals.
databricks grants update catalog <catalog> --profile <profile> --json "{\"changes\":[{\"principal\":\"account users\",\"add\":[\"USE_CATALOG\"]},{\"principal\":\"<runner-sp-application-id>\",\"add\":[\"USE_CATALOG\",\"CREATE_SCHEMA\"]},{\"principal\":\"<app-service-principal-client-id>\",\"add\":[\"USE_CATALOG\"]}]}"

# 4. Start the app. It publishes the runner wheel to the volume on startup.
databricks bundle run mcp-dqx --var catalog_name=<catalog> --var runner_service_principal_id=<runner-sp-application-id> --profile <profile>

# 5. Confirm the wheel was published (should list dqx_mcp_runner-<version>-py3-none-any.whl).
databricks fs ls dbfs:/Volumes/<catalog>/dqx_mcp_tmp/dqx_artifacts --profile <profile>

If step 5 lists nothing, check databricks apps logs mcp-dqx --profile <profile> — a failed publish is almost always a missing USE CATALOG grant from step 3. Apply it and re-run step 4.

3. Find your MCP endpoint​

The MCP endpoint is https://<app-url>/mcp. Find <app-url> in the Databricks UI under Compute → Apps → mcp-dqx (the App URL), or via the CLI:

databricks apps get mcp-dqx --profile <profile> -o json | jq -r .url

4. Verify the deployment​

# a) The app is running.
databricks apps get mcp-dqx --profile <profile> -o json | jq -r '.compute_status.state, .app_status.state'

# b) The app published the runner wheel — every data tool needs this.
databricks fs ls dbfs:/Volumes/<catalog>/dqx_mcp_tmp/dqx_artifacts --profile <profile>

# c) The catalog grants are in place (users group, runner SP, app SP).
databricks grants get catalog <catalog> --profile <profile>

# d) The bundle applied its own grants on the schema and volumes.
databricks grants get schema <catalog>.dqx_mcp_tmp --profile <profile>
databricks grants get volume <catalog>.dqx_mcp_tmp.mcp_results --profile <profile>

Then call a tool through the endpoint. save_checks is the most complete check — it exercises the runner SP's CREATE SCHEMA grant and the per-user output schema.

Reference​

Everything below is explanatory — you don't act on it to deploy. It documents what the deploy creates, the grants it applies, and why the catalog-ownership prerequisite exists.

What gets deployed​

The bundle creates these in your workspace (all prefixed by name_prefix, default mcp-dqx):

ResourceNamePurpose
Databricks Appmcp-dqxThe MCP server (serves /mcp); runs as its own app service principal.
App service principalauto-namedAuto-created by the platform when the app is deployed — its application id is assigned at deploy time (not known beforehand). The identity the app process runs as.
Jobmcp-dqx-runnerServerless wheel-task job that installs a pinned, published databricks-labs-dqx release from the registry; runs as the runner SP. Tools submit work to it. See DQX version.
Schema<catalog>.dqx_mcp_tmpBundle-managed, with its grants declared natively. Holds the two volumes below and short-lived temp views.
Volume…dqx_mcp_tmp.mcp_resultsBundle-managed. Each run's JSON result plus staged job inputs.
Volume…dqx_mcp_tmp.dqx_artifactsBundle-managed. The runner wheel, published by the app at startup.

The catalog is injected into the app as a plain DQX_CATALOG config value (no secret scope — a catalog name isn't sensitive; UC grants govern access). Nothing is destroy-protected, so databricks bundle destroy removes all of it — including the schema and both volumes (see Upgrade and uninstall). Two things are not part of the bundle and survive: the runner service principal (you create it yourself) and any dqx_mcp_<user> output schemas.

DQX version​

The runner job installs a pinned, published databricks-labs-dqx release from the package registry — set in the runner job's environment dependencies in mcp-server/databricks.yml. This mirrors DQX Studio, which also pins a released version rather than building from source, so the DQX version the tools run is deterministic and independent of your local working tree.

  • To upgrade DQX, bump the pinned version in mcp-server/databricks.yml and redeploy.
  • Redeploy to pick up changes: the runner is not auto-updated — it keeps using the version from the last deploy until you redeploy (see Upgrade and uninstall).

What the deploy creates inside your catalog​

The catalog you provide is used as the MCP server's working area. The deploy and its tools create the following inside it — nothing else:

ObjectTypeCreated byPurpose & lifecycle
<catalog>.dqx_mcp_tmpSchemabundle deployBundle-managed, with native grants. Holds the two volumes below plus short-lived temp views. Removed by bundle destroy.
<catalog>.dqx_mcp_tmp.dqx_artifactsVolumebundle deployHosts the runner wheel, which the app publishes at startup (skipped when unchanged). Removed by bundle destroy.
<catalog>.dqx_mcp_tmp.mcp_resultsVolumebundle deployEach tool run's JSON result keyed by run_id, plus staged job inputs. The app reads results back from here. Removed by bundle destroy.
Temporary views under <catalog>.dqx_mcp_tmpViewat runtime, per data-tool callDefiner's-rights views created as the calling user for profile_table / run_checks / apply_checks_and_save_to_table. Dropped by the runner job after each run (a periodic sweeper reaps any orphans).
<catalog>.dqx_mcp_<user>Schema (one per user)at runtime, on demandCreated + owned by the runner SP the first time a user calls a persisting tool. Holds that user's save_checks / apply_checks_and_save_to_table output tables. Access is granted only to that user; users never see each other's schemas. See Write access.

So beyond the bundle-managed dqx_mcp_tmp schema (with two volumes), the catalog accumulates one dqx_mcp_<user> schema per user who persists checks or results, plus transient temp views that clean themselves up. databricks bundle destroy removes the app, the runner job and the dqx_mcp_tmp schema with both volumes; the dqx_mcp_<user> schemas are not bundle-managed and survive — drop those manually for a fully clean catalog (see Upgrade and uninstall).

If a `dqx_mcp_tmp` schema already exists in the catalog

The working schema is named dqx_mcp_tmp (DQX-namespaced to avoid clashing with a generic schema name). The bundle creates it, so a pre-existing schema of that name makes the deploy stop with SCHEMA_ALREADY_EXISTS rather than silently taking it over. That is deliberate — adopting a schema the bundle did not create, then deleting it on bundle destroy, would be worse than a clear error. Deploy with --var tmp_schema_name=<another-name> to use a different schema, or drop the existing one if it is not needed.

Catalog grants the deploy applies​

Everything on the objects the bundle owns (the schema, both volumes, the runner job's ACL) is declared natively in the bundle and applied by bundle deploy. The catalog itself is not bundle-managed — the bundle cannot grant on a catalog it does not own — so make mcp-deploy applies these catalog-level grants itself via mcp-server/scripts/grant_catalog_prereqs.sh (idempotent; this is why the deploying user needs MANAGE/ownership on the catalog):

GranteeGrantOnApplied by
Users groupUSE CATALOGcatalogscript (catalog not bundle-managed)
Runner SPUSE CATALOG, CREATE SCHEMAcatalogscript
App SPUSE CATALOGcatalogscript (after the app exists)
Users groupUSE SCHEMA, CREATE TABLE<catalog>.dqx_mcp_tmpnative (bundle deploy)
App SPUSE SCHEMA, SELECT, MODIFY<catalog>.dqx_mcp_tmpnative
App SPREAD VOLUME, WRITE VOLUMEboth volumesnative
App SPCAN_MANAGE_RUNthe runner jobnative
Runner SPUSE SCHEMA, SELECT, MODIFY<catalog>.dqx_mcp_tmpnative
Runner SPREAD VOLUME, WRITE VOLUME…mcp_resultsnative
Runner SPREAD VOLUME…dqx_artifactsnative

Notes: CREATE SCHEMA on the catalog lets the runner SP create each caller's private per-user output schema (dqx_mcp_<user>) on demand — it is a catalog-level grant, so it cannot be native. The app is deliberately given CAN_MANAGE_RUN and not CAN_MANAGE on the runner job: it only submits runs and polls results, and CAN_MANAGE would let it rewrite the job definition (including its run_as) and so escape the least-privilege boundary the dedicated runner SP exists to enforce.

Deploying-user permissions​

If a deploy step is rejected, this table tells you which grant is missing (typical names; exact entitlements vary by workspace):

PermissionOnUsed bySymptom if missing
Allow to create/manage Appsworkspacebundle deploy (the app)app creation rejected
Allow cluster create / serverless entitlementworkspacethe runner jobjob creation rejected
USE CATALOG + CREATE SCHEMAthe target catalogbundle deploy (creates <catalog>.dqx_mcp_tmp + both volumes)User does not have CREATE SCHEMA
MANAGE (or owner) on the catalogthe target catalogthe catalog-level GRANTs the deploy applies (the bundle cannot declare them)User does not have MANAGE on Catalog
Service Principal: User role on the runner SPthe runner SPbundle deploy sets the job's run_asUser is not authorized to use this service principal

Upgrade and uninstall​

Upgrade — pull the latest code and re-deploy; the bundle updates the app and job in place, and the app republishes the runner wheel if it changed (an unchanged wheel is skipped via a content hash):

git pull
make mcp-deploy PROFILE=<profile> CATALOG=<catalog> RUNNER_SP=<runner-sp-application-id>

Uninstall — teardown is a single command:

# macOS / Linux (make)
make mcp-destroy PROFILE=<profile> CATALOG=<catalog>

# Any platform (Databricks CLI only)
cd mcp-server
databricks bundle destroy --auto-approve --var catalog_name=<catalog> --profile <profile>
What destroy removes

The schema and its volumes are bundle-managed, so bundle destroy deletes the app, the runner job, and <catalog>.dqx_mcp_tmp together with the mcp_results and dqx_artifacts volumes inside it — including any result files still in mcp_results. Earlier versions left the schema behind. Back up anything you need first.

What survives, because it is not part of the bundle: any <catalog>.dqx_mcp_<user> output schemas (they hold user data), the catalog and its grants, and the runner service principal — delete those separately if no longer needed.

bundle destroy only removes resources this bundle deployment tracks. If the workspace has an mcp-dqx app or mcp-dqx-runner job from a separate deploy (a colleague's, or an isolated name_prefix copy), remove those separately.

Installing Dashboard​

DQX data quality dashboard can be used to monitor data quality across all tables monitored by DQX. The dashboard is not scheduled to refresh by default, ensuring no costs are incurred.

You have two options to set up the dashboard:

Option 1: Import directly in the workspace​

You can import the dashboard directly from a file. The dashboard file (DQX_Dashboard.lvdash.json) is available in the GitHub repository. To import it:

  1. Navigate to Dashboards in the Databricks UI
  2. Click Import dashboard from file
  3. Select the .lvdash.json file
  4. Provide parameters and refresh the dashboard

Option 2: Use Databricks CLI​

When you install DQX as a tool using the Databricks CLI (see Install DQX using Databricks CLI), the dashboard is automatically deployed to the installation directory.

You can then open it using:

databricks labs dqx open-dashboards

After executing the command:

  1. Locate and click on the dashboard file in the workspace UI
  2. Provide parameters and refresh the dashboard