Skip to main content

databricks.labs.dqx.manager

DQRuleManager Objects​

@dataclass(frozen=True)
class DQRuleManager()

Orchestrates the application of a data quality rule to a DataFrame and builds the final check result.

The manager is responsible for:

  • Executing the rule using the appropriate row or dataset executor.
  • Applying any filter condition specified in the rule to the check result.
  • Combining user-defined and engine-provided metadata into the result.
  • Constructing the final structured output (including check name, function, columns, metadata, etc.) as a DQCheckResult.

The manager does not implement the logic of individual checks. Instead, it delegates rule application to the appropriate DQRuleExecutor based on the rule type (row-level or dataset-level).

Attributes:

  • check - The DQRule instance that defines the check to apply.
  • df - The DataFrame on which to apply the check.
  • engine_user_metadata - Metadata provided by the engine (overridden by check.user_metadata if present).
  • run_time_overwrite - Optional timestamp override. If None, current_timestamp() is used for per-micro-batch timestamps.
  • ref_dfs - Optional reference DataFrames for dataset-level checks.
  • run_id - Optional unique run id.

user_metadata​

@cached_property
def user_metadata() -> dict[str, str]

Returns user metadata as a dictionary.

filter_condition​

@cached_property
def filter_condition() -> Column

Returns the filter condition for the check.

has_unsafe_filter​

@cached_property
def has_unsafe_filter() -> bool

Returns True if the check filter contains unsafe SQL (e.g. a destructive statement keyword).

This is a non-raising check: an unsafe filter causes the check to be skipped (like any other invalid filter) rather than aborting evaluation of the whole rule set. Note this does not apply to the sql_query check, whose query runs via spark.sql and is still hard-rejected in the executor.

invalid_columns​

@cached_property
def invalid_columns() -> list[str]

Returns list of invalid check columns in the input DataFrame.

Column names that contain characters requiring SQL identifier escaping (e.g. spaces, such as "Customer Name") are back-quoted so they are unambiguous in the skip / error messages.

has_invalid_columns​

@cached_property
def has_invalid_columns() -> bool

Returns a boolean indicating whether any of the specified check columns are invalid in the input DataFrame.

has_invalid_filter​

@cached_property
def has_invalid_filter() -> bool

Returns a boolean indicating whether the filter is invalid in the input DataFrame.

An unsafe filter is short-circuited here so that filter_condition (which rejects unsafe SQL by raising) is never compiled for it; the check is reported as skipped via _get_invalid_cols_message.

has_invalid_custom_message​

@cached_property
def has_invalid_custom_message() -> bool

Returns a boolean indicating whether the custom message expression is invalid in the input DataFrame.

invalid_sql_expression​

@cached_property
def invalid_sql_expression() -> str | None

Returns an invalid expression for sql expression check.

process​

def process() -> DQCheckResult

Process the data quality rule (check) and return results as DQCheckResult containing:

  • Column with the check result
  • optional DataFrame with the results of the check

Skip the check evaluation if column or columns, or filter in the check cannot be resolved in the input DataFrame. Return the check result preserving all fields with message identifying invalid fields.