Skip to main content

databricks.labs.dqx.config

InputConfig Objects​

@dataclass
class InputConfig()

Configuration class for input data sources (e.g. tables or files).

OutputConfig Objects​

@dataclass
class OutputConfig()

Configuration class for output data sinks (e.g. tables or files).

ProfilerConfig Objects​

@dataclass
class ProfilerConfig()

Configuration class for profiler.

summary_stats_file​

file containing profile summary statistics

sample_fraction​

fraction of data to sample (30%)

sample_seed​

seed for sampling

limit​

limit the number of records to profile

RunConfig Objects​

@dataclass
class RunConfig()

Configuration class for the data quality checks

name​

name of the run configuration

quarantine_config​

quarantined data table

warehouse_id​

warehouse id to use in the dashboard

reference_tables​

reference tables to use in the checks

ExtraParams Objects​

@dataclass(frozen=True)
class ExtraParams()

Class to represent extra parameters for DQEngine.

WorkspaceConfig Objects​

@dataclass
class WorkspaceConfig()

Configuration class for the workspace

extra_params​

extra parameters to pass to the jobs, e.g. run_time

get_run_config​

def get_run_config(run_config_name: str | None = "default") -> RunConfig

Get the run configuration for a given run name, or the default configuration if no run name is provided.

Arguments:

  • run_config_name - The name of the run configuration to get.

Returns:

The run configuration.

Raises:

  • ValueError - If no run configurations are available or if the specified run configuration name is not found.

BaseChecksStorageConfig Objects​

@dataclass
class BaseChecksStorageConfig(abc.ABC)

Marker base class for storage configuration.

FileChecksStorageConfig Objects​

@dataclass
class FileChecksStorageConfig(BaseChecksStorageConfig)

Configuration class for storing checks in a file.

Arguments:

  • location - The file path where the checks are stored.

WorkspaceFileChecksStorageConfig Objects​

@dataclass
class WorkspaceFileChecksStorageConfig(BaseChecksStorageConfig)

Configuration class for storing checks in a workspace file.

Arguments:

  • location - The workspace file path where the checks are stored.

TableChecksStorageConfig Objects​

@dataclass
class TableChecksStorageConfig(BaseChecksStorageConfig)

Configuration class for storing checks in a table.

Arguments:

  • location - The table name where the checks are stored.
  • run_config_name - The name of the run configuration to use for checks (default is 'default').
  • mode - The mode for writing checks to a table (e.g., 'append' or 'overwrite'). The overwrite mode will only replace checks for the specific run config and not all checks in the table.

run_config_name​

to filter checks by run config

VolumeFileChecksStorageConfig Objects​

@dataclass
class VolumeFileChecksStorageConfig(BaseChecksStorageConfig)

Configuration class for storing checks in a Unity Catalog volume file.

Arguments:

  • location - The Unity Catalog volume file path where the checks are stored.

InstallationChecksStorageConfig Objects​

@dataclass
class InstallationChecksStorageConfig(WorkspaceFileChecksStorageConfig,
                                      TableChecksStorageConfig,
                                      VolumeFileChecksStorageConfig)

Configuration class for storing checks in an installation.

Arguments:

  • location - The installation path where the checks are stored (e.g., table name, file path). Not used when using installation method, as it is retrieved from the installation config.
  • run_config_name - The name of the run configuration to use for checks (default is 'default').
  • product_name - The product name for retrieving checks from the installation (default is 'dqx').
  • assume_user - Whether to assume the user is the owner of the checks (default is True).

location​

retrieved from the installation config

run_config_name​

to retrieve run config