Skip to main content

Data Contract Quality Rules Generation

DQX enables the generation of data quality rules directly from data contracts that follow the Open Data Contract Standard (ODCS). This feature allows teams to implement federated governance strategies by defining data contracts for their Data Products and automatically deriving comprehensive quality checks from those contracts.

Overview​

The data contract-based quality rules generation is available via the DQGenerator class and supports generation of the following rule types:

  • Schema Validation: One dataset-level has_valid_schema rule per ODCS schema when generate_schema_validation=True (default), derived from the contract's schema definition. Use strict_schema_validation=True (default) for exact column set, order, and types; set to False for permissive validation (expected columns with compatible types, extra columns allowed).
  • Predefined Rules: Automatically derived from schema properties and constraints (required, unique, pattern, min/max, enum, format).
  • Explicit Rules: Custom DQX rules embedded directly in the contract's quality section using DQX native format.
  • Text-Based Rules: Natural language quality expectations that are processed using AI/LLM to generate appropriate checks (requires [llm] extra).

This approach is beneficial when you want to:

  • Implement standardised data contracts across your organisation
  • Enable data product owners to define quality expectations in a familiar format
  • Maintain version-controlled quality rules alongside schema definitions
  • Trace quality checks back to contract specifications for compliance and lineage
When to use Data Contract-Based Generation

Data contract-based generation is ideal for:

  • Federated Data Governance: Enable data product teams to define and own their quality contracts
  • Standardisation: Use industry-standard ODCS format for data contracts
  • Version Control: Track data contracts and derived quality rules together
  • Compliance: Maintain auditable links between contracts and quality checks
  • Data Mesh: Support domain-oriented data ownership with clear contracts

For automated discovery of data patterns without contracts, consider using the Data Profiling approach instead. For generating rules from natural language without contracts, see AI-Assisted Generation.

ODCS v3.x Native Support​

DQX natively supports Open Data Contract Standard (ODCS) v3.x contracts. ODCS is part of the Linux Foundation and provides a comprehensive format for defining data product contracts.

Key Features:

  • Native Processing: DQX processes ODCS v3.x contracts directly without conversion
  • Schema Validation: Generates one has_valid_schema rule per schema when enabled (default strict; optional permissive)
  • Full Support: Extracts constraints from logicalTypeOptions (pattern, minimum, maximum, minLength, maxLength)
  • Quality Sections: Handles explicit DQX rules via implementation blocks in quality sections
  • Version Compatibility: Supports ODCS v3.0.x and above (no backward compatibility with older versions)

DQX leverages the datacontract-cli library to parse and validate contracts, ensuring full compatibility with the ODCS ecosystem.

Prerequisites​

To use the data contract-based quality rules generation feature, you need to install DQX with the datacontract extra dependencies:

pip install 'databricks-labs-dqx[datacontract]'

This will install the required packages including datacontract-cli for ODCS contract parsing and validation.

Optional: LLM Support for Text-Based Rules

If your contracts include text-based quality expectations (natural language descriptions), you'll also need to install the LLM dependencies:

pip install 'databricks-labs-dqx[datacontract,llm]'

Without the LLM dependencies, text-based expectations will be skipped with a warning, but predefined and explicit rules will still be generated.

Basic Usage​

Loading from File Path​

Here's a simple example of generating quality rules from a data contract file:

from databricks.labs.dqx.profiler.generator import DQGenerator
from databricks.sdk import WorkspaceClient

# Initialize the generator
ws = WorkspaceClient()
generator = DQGenerator(workspace_client=ws, spark=spark)

# Generate rules from contract file
rules = generator.generate_rules_from_contract(
    contract_file="/Workspace/Shared/contracts/customers.yaml"
)

print(f"Generated {len(rules)} quality rules")
for rule in rules[:3]:
    print(f"- {rule['name']}: {rule['check']['function']}")

Loading from DataContract Object​

You can also pass a pre-loaded DataContract object from datacontract-cli:

from databricks.labs.dqx.profiler.generator import DQGenerator
from databricks.sdk import WorkspaceClient
from datacontract.data_contract import DataContract

ws = WorkspaceClient()
generator = DQGenerator(workspace_client=ws, spark=spark)

# Load contract using datacontract-cli
contract = DataContract(data_contract_file="/path/to/contract.yaml")

# Generate rules from contract object
rules = generator.generate_rules_from_contract(contract=contract)

print(f"Generated {len(rules)} quality rules")

Schema validation​

When the contract defines a schema and generate_schema_validation=True (default), DQX generates one dataset-level rule per ODCS schema that calls has_valid_schema with the expected schema derived from the contract. Set generate_schema_validation=False to skip these rules. The strictness of schema validation is controlled by strict_schema_validation: default True (exact columns, order, and types); set to False for permissive validation (expected columns must exist with compatible types; extra columns are allowed).

Physical types required for schema validation

Schema validation requires every property in the contract to have a physicalType field set to a valid Unity Catalog data type. Properties with only logicalType will cause an InvalidPhysicalTypeError.

# Default: schema validation rules generated, strict mode
rules = generator.generate_rules_from_contract(contract_file="/path/to/contract.yaml")

# Skip schema validation rules
rules = generator.generate_rules_from_contract(
    contract_file="/path/to/contract.yaml",
    generate_schema_validation=False,
)

# Permissive schema validation (allow extra columns)
rules = generator.generate_rules_from_contract(
    contract_file="/path/to/contract.yaml",
    strict_schema_validation=False,
)
Type validation rules
  • Recursive validation: Inner types of ARRAY<T>, MAP<K,V>, and STRUCT<...> are validated at every nesting level. Invalid inner types (e.g. ARRAY<NOT_A_TYPE>) raise InvalidPhysicalTypeError at contract load time.
  • Nesting depth limit: Type nesting is limited to 50 levels to avoid stack overflow on malformed or excessively deep types. Exceeding this limit raises InvalidPhysicalTypeError.

Generated schema validation rules include user_metadata.rule_type: "schema_validation" and the expected DDL string built from the contract's physicalType for each property.

Predefined Rule Generation​

Predefined rules are automatically derived from schema field constraints defined in your data contract. DQX analyzes field properties and generates appropriate quality checks without requiring explicit rule definitions.

Supported ODCS v3.x Property Constraints​

ConstraintLocationGenerated DQX RuleExample
required: trueDirect property attributeis_not_nullAll mandatory properties
unique: trueDirect property attributeis_uniquePrimary keys, identifiers
patternlogicalTypeOptionsregex_matchEmail, phone, ID formats
minimum + maximumlogicalTypeOptionsis_in_range or sql_expression (floats)Age, amount limits
minimum onlylogicalTypeOptionsis_aggr_not_less_thanPositive amounts
maximum onlylogicalTypeOptionsis_aggr_not_greater_thanUpper bounds
minLength onlylogicalTypeOptionssql_expression (LENGTH >= N)Minimum code length
maxLength onlylogicalTypeOptionssql_expression (LENGTH <= N)Maximum field size
minLength + maxLengthlogicalTypeOptionssql_expression (both checks)Fixed-length codes
formatlogicalTypeOptionsLogged (not yet implemented)Date/timestamp formats

Note: In ODCS v3.x, enum values are typically handled via pattern constraints or quality rules with type: library.

Example Contract with Predefined Rules​

kind: DataContract
apiVersion: v3.0.2
id: urn:datacontract:ecommerce:orders
name: E-Commerce Orders
version: 1.0.0
status: active
domain: ecommerce
dataProduct: orders_data_product

schema:
  - name: orders
    physicalType: table
    properties:
      - name: order_id
        logicalType: string
        physicalType: string
        required: true              # → is_not_null
        unique: true                # → is_unique
        logicalTypeOptions:
          pattern: '^ORD-[0-9]{8}$'   # → regex_match
      
      - name: customer_email
        logicalType: string
        physicalType: string
        required: true              # → is_not_null
        logicalTypeOptions:
          pattern: '^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'  # → regex_match
      
      - name: order_total
        logicalType: number
        physicalType: decimal(10,2)
        required: true              # → is_not_null
        logicalTypeOptions:
          minimum: 0.01               # → sql_expression (floats)
          maximum: 100000.00          # → sql_expression (floats)
      
      - name: order_status
        logicalType: string
        physicalType: string
        required: true              # → is_not_null
        logicalTypeOptions:
          pattern: '^(pending|confirmed|shipped|delivered|cancelled)$'  # → regex_match

Controlling Predefined Rule Generation​

You can control whether predefined rules are generated:

# Generate only explicit rules, skip predefined ones
rules = generator.generate_rules_from_contract(
    contract_file="contract.yaml",
    generate_predefined_rules=False
)

# Generate all rules (default behavior)
rules = generator.generate_rules_from_contract(
    contract_file="contract.yaml",
    generate_predefined_rules=True
)

Setting Default Criticality​

All predefined rules inherit a default criticality level:

# Generate predefined rules with 'warn' criticality (default is 'error')
rules = generator.generate_rules_from_contract(
    contract_file="contract.yaml",
    default_criticality="warn"  # or "error"
)

# All predefined rules will have criticality: "warn"
# Explicit rules can still define their own criticality

Explicit Rule Generation​

Explicit rules allow you to embed custom DQX quality checks directly within your ODCS v3.x data contract. This is useful for business logic rules that cannot be expressed as simple property constraints.

Defining Explicit Rules in ODCS v3.x Contracts​

Explicit rules are defined in the quality section of a property or schema using type: custom with engine: dqx and an implementation block:

kind: DataContract
apiVersion: v3.0.2
id: urn:datacontract:finance:transactions
name: Financial Transactions
version: 1.0.0
status: active
domain: finance
dataProduct: transactions_data_product

schema:
  - name: transactions
    physicalType: table
    
    properties:
      - name: transaction_id
        logicalType: string
        physicalType: string
        required: true
        unique: true
        
      - name: account_id
        logicalType: string
        physicalType: string
        required: true
        quality:
          # Property-level explicit check
          - type: custom
            engine: dqx
            description: Account ID must not be empty
            implementation:
              name: account_id_format
              criticality: error
              check:
                function: is_not_null_and_not_empty
                arguments:
                  column: account_id
                  trim_strings: true
      
      - name: amount
        logicalType: number
        physicalType: decimal(10,2)
        required: true
        logicalTypeOptions:
          minimum: 0
        quality:
          # Business logic rule
          - type: custom
            engine: dqx
            description: Flag suspicious high-value transactions
            implementation:
              name: suspicious_transaction_flag
              criticality: warn
              check:
                function: sql_expression
                arguments:
                  expression: amount < 10000 OR verified = true
                  columns:
                    - amount
                    - verified
    
    # Schema-level (dataset-level) quality checks
    quality:
      # Check data freshness
      - type: custom
        engine: dqx
        description: Ensure transaction data is recent
        implementation:
          name: transaction_data_freshness
          criticality: error
          check:
            function: is_data_fresh_per_time_window
            arguments:
              column: transaction_timestamp
              window_minutes: 60
              min_records_per_window: 1
              lookback_windows: 2
      
      # Check minimum row count
      - type: custom
        engine: dqx
        description: Ensure dataset contains transactions
        implementation:
          name: minimum_transactions
          criticality: warn
          check:
            function: is_aggr_not_less_than
            arguments:
              column: transaction_id
              limit: 1
              aggr_type: count

Explicit Rule Structure for ODCS v3.x​

Each explicit DQX rule must follow this structure:

- type: custom              # Required: marks as custom quality check
  engine: dqx              # Required: specifies DQX engine
  description: ...         # Optional: human-readable description
  implementation:          # Required: contains the DQX rule (ODCS v3.x)
    name: rule_name        # Required: unique rule name
    criticality: error     # Required: "error" or "warn"
    check:                 # Required: the actual check definition
      function: check_function_name
      arguments:
        # function-specific arguments
Explicit Rule Criticality in ODCS v3.x

Unlike predefined rules (which use default_criticality), explicit rules must define their own criticality within the implementation block. This allows fine-grained control over different business rules.

Text-Based Rule Generation​

Text-based rules allow you to define quality expectations in natural language, which are then processed by an LLM to generate appropriate DQX checks. This combines the power of data contracts with AI-assisted rule generation.

Prerequisites for Text-Based Rules​

Text-based rule generation requires both the datacontract and llm extras:

pip install 'databricks-labs-dqx[datacontract,llm]'

Defining Text-Based Expectations​

Text-based expectations are defined using type: text in the quality section:

kind: DataContract
apiVersion: v3.0.2
id: urn:datacontract:compliance:user_data
name: User Data (GDPR Compliant)
version: 1.0.0
status: active

schema:
  - name: users
    physicalType: table
    
    # Dataset-level text expectation
    quality:
      - type: text
        description: |
          The dataset should not contain duplicate user records based on
          email address. Users with the same email are considered duplicates
          and should be flagged for review.
    
    properties:
      - name: email
        logicalType: string
        physicalType: string
        required: true
        quality:
          # Field-level text expectation
          - type: text
            description: |
              Email addresses must be valid and from approved corporate
              domains only (@company.com or @partner.com). Personal email
              domains like gmail.com or yahoo.com should be rejected.
      
      - name: consent_date
        logicalType: date
        physicalType: date
        required: true
        quality:
          # Another text expectation
          - type: text
            description: |
              Consent date must not be in the future and should be within
              the last 7 years to comply with GDPR retention policies.

Generating Rules with Text Expectations​

from databricks.labs.dqx.profiler.generator import DQGenerator
from databricks.sdk import WorkspaceClient

ws = WorkspaceClient()
generator = DQGenerator(workspace_client=ws, spark=spark)

# Generate rules including text-based expectations (requires LLM)
rules = generator.generate_rules_from_contract(
    contract_file="user_data_contract.yaml",
    generate_predefined_rules=True,   # Include schema-derived rules
    process_text_rules=True         # Process text expectations with LLM
)

# The generated rules will include:
# - Predefined rules from schema constraints
# - Explicit DQX rules (if any)
# - Rules generated from text expectations via LLM

Disabling Text Rule Processing​

If you don't have LLM dependencies installed or want to skip text processing:

# Skip text-based rules, generate only predefined and explicit rules
rules = generator.generate_rules_from_contract(
    contract_file="contract.yaml",
    process_text_rules=False  # Skip LLM processing
)

# Text expectations will be logged as warnings and skipped

Metric Rule Generation​

ODCS quality entries with type: library (or, per the spec, no type at all as long as metric is set) reference a named, standard data quality metric instead of a natural-language description or a hand-written DQX check. DQX recognizes five of these metrics — rowCount, nullValues, missingValues, invalidValues, and duplicateValues — and maps each one onto the DQX check(s) that best express it.

A fast on-ramp, not the long-term home for your checks

Library metrics are for teams who are already ODCS-native, or who have existing ODCS contracts and want DQX enforcing them straight away. Point DQX at a contract you already have and you get checks immediately, with no rewrite — and the same five-metric vocabulary is understood by any other ODCS-compliant tool.

Treat that as a starting point. The five metrics cover the common cases, but not the rest of what DQX can check: no outlier detection, freshness, foreign keys, schema validation, or dataset comparison. Longer term you will get more out of defining checks natively, either as explicit rules with type: custom and engine: dqx or as text-based expectations — both give you the full DQX check surface and far more flexibility than a fixed metric vocabulary.

This five-metric surface is intentionally closed: DQX does not plan to grow this mapping into a general ODCS interpreter for arbitrary quality vocabularies.

Every type: library entry is processed whenever generate_rules_from_contract runs, unless disabled with process_library_rules=False (default True), matching the generate_predefined_rules/process_text_rules opt-out pattern used elsewhere. An entry that DQX cannot map is skipped with a logged warning instead of failing the whole contract (see Malformed and Unrecognized Entries).

Where the ODCS spec is silent

Several of the mapping decisions below are not settled by the ODCS spec text. Where that's the case, DQX follows the interpretation already established by other ODCS-compliant tooling rather than inventing its own reading, so a contract produces the same verdict regardless of which tool evaluates it. Each judgment call is called out explicitly where it applies: duplicateValues's counting unit and invalidValues's combined-condition handling of validValues + pattern. Bound inclusivity for mustBeBetween/mustNotBeBetween is a different case — the spec is inconsistent rather than silent — and is covered under Threshold Fields and the SQL Fallback.

Defining Metric Rules in ODCS v3.x Contracts​

A library quality entry sets a metric name (and optionally type: library) in the quality section of a property (for nullValues, missingValues, invalidValues, and the single-column form of duplicateValues) or a schema (for rowCount and the composite-key form of duplicateValues):

quality:
  - type: library
    metric: nullValues
    mustBe: 0
    dimension: completeness   # optional; defaults per metric, see below
    severity: critical        # optional; recorded verbatim, DQX defines no vocabulary for it

Contracts written against ODCS 3.0.x may use the deprecated rule key instead of metric; DQX accepts both.

Every metric shares the same threshold fields (one of which must be set) and an optional unit — see Threshold Fields and the SQL Fallback and Rows vs. Percent below. dimension and severity are recorded in user_metadata for every metric rule; dimension falls back to a per-metric ODCS default when the contract omits it, and severity is only present when the contract sets it.

Supported Metrics​

MetricLevelDefault dimensionWhat it validates
rowCountSchemacompletenessThe total row count of the dataset.
nullValuesPropertycompletenessThe count or percentage of NULL values in a column.
missingValuesPropertycompletenessThe count or percentage of NULLs plus contract-supplied sentinel values (e.g. "", "N/A") in a column.
invalidValuesPropertyconformityThe count or percentage of values in a column that fail an allowlist and/or a regex pattern.
duplicateValuesProperty (single column) or Schema (composite key)uniquenessThe count or percentage of distinct values (or key combinations) that occur more than once.

nullValues, missingValues, and invalidValues are property-level only; a schema-level entry for one of these is skipped with a warning. rowCount is schema-level only. duplicateValues supports both: a property-level entry with no arguments checks that single column, while a schema-level entry reads a composite key from arguments.properties.

Threshold Fields and the SQL Fallback​

Every library metric is evaluated against the eight ODCS threshold fields: mustBe, mustNotBe, mustBeGreaterOrEqualTo, mustBeLessOrEqualTo, mustBeGreaterThan, mustBeLessThan, mustBeBetween, mustNotBeBetween. Each field set on an entry produces its own rule, so a range written as two fields on one entry (e.g. mustBeGreaterOrEqualTo: 10 and mustBeLessOrEqualTo: 1000) enforces both bounds.

  • mustBe: 0 is special-cased for nullValues, missingValues, invalidValues, and duplicateValues: it maps onto a cheap row-level check (is_not_null, is_not_in_list, is_in_list/regex_match, or is_unique) that pinpoints the offending rows, rather than a dataset-level count.
  • mustBe, mustNotBe, mustBeGreaterOrEqualTo, mustBeLessOrEqualTo (including mustBe with a non-zero value) map onto exact-fit dataset-level aggregate checks — is_aggr_equal, is_aggr_not_equal, is_aggr_not_less_than, is_aggr_not_greater_than — over the metric's count or percentage, for rowCount, nullValues, missingValues, and invalidValues.
  • mustBeGreaterThan, mustBeLessThan, mustBeBetween, mustNotBeBetween have no strict-inequality or range equivalent among DQX's aggregate checks, so they fall back to a dataset-level sql_query check. For mustBeBetween/mustNotBeBetween, both bounds are inclusive — a value exactly equal to either bound counts as being "between" them. The ODCS spec is inconsistent here: its operator table writes mustBeBetween: [0, 100] as ∈ (a closed interval), while a note calls it equivalent to mustBeGreaterThan + mustBeLessThan. DQX follows SQL BETWEEN, so mustBeBetween: [0, 5] passes a column with no nulls.
  • duplicateValues is the one exception: every non-mustBe: 0 threshold falls back to sql_query, not just the four above. See duplicateValues below for exactly what it counts.
No recognized threshold field

An entry with none of the eight fields set is skipped with a warning naming all eight, so a contract author can spot a typo (e.g. mustbe instead of mustBe) without reading DQX source. Threshold values must be numbers (a numeric string such as "0" is accepted), and mustBeBetween/mustNotBeBetween must be two numbers, smallest first; anything else is skipped with a warning.

Rows vs. Percent​

The optional unit field controls whether a non-mustBe: 0 threshold is evaluated as a raw row count (unit: rows, the default) or a percentage of all rows (unit: percent). This applies to nullValues, missingValues, invalidValues, and duplicateValues. rowCount is inherently a count and accepts only unit: rows. Any other unit value causes the entry's thresholds to be skipped with a warning, except mustBe: 0, which ignores the unit, since zero rows and zero percent are the same fact. On an empty dataset a percentage is treated as 0.

rowCount​

schema:
  - name: orders
    physicalType: table
    quality:
      - type: library
        metric: rowCount
        mustBeGreaterOrEqualTo: 1   # → is_aggr_not_less_than(column="*", aggr_type="count")

A mustBeGreaterThan, mustBeLessThan, mustBeBetween, or mustNotBeBetween threshold on rowCount falls back to a sql_query check counting COUNT(*) over the whole dataset.

nullValues​

properties:
  - name: customer_email
    logicalType: string
    physicalType: string
    quality:
      - type: library
        metric: nullValues
        mustBe: 0                # → row-level is_not_null(column="customer_email")

  - name: middle_name
    logicalType: string
    physicalType: string
    quality:
      # `type: library` can be omitted when `metric` is set, which is how the ODCS
      # spec's own examples are written -- DQX recognizes both forms.
      - metric: nullValues
        mustBeLessOrEqualTo: 5
        unit: percent            # → dataset-level check on the null percentage

missingValues​

missingValues combines real NULLs with a contract-supplied sentinel list, so it needs arguments.missingValues — a list that may include a literal null alongside sentinel strings:

properties:
  - name: country_code
    logicalType: string
    physicalType: string
    quality:
      - type: library
        metric: missingValues
        mustBe: 0
        arguments:
          missingValues: [null, "", "N/A", "UNKNOWN"]
        # → row-level is_not_null (for the null entry)
        #   + row-level is_not_in_list(forbidden=["", "N/A", "UNKNOWN"])

NULL is always counted as missing, whether or not a literal null appears in arguments.missingValues — listing it is redundant, not required. With mustBe: 0, a row-level is_not_null rule is always generated, plus a row-level is_not_in_list rule when non-null sentinels are present; both rules share the same user_metadata and are OR'd together via DQX's own per-row error/warning union, since is_not_in_list's forbidden list can never itself match a real SQL NULL. Any other threshold routes through a dataset-level count/percentage of rows matching column IS NULL OR column IN (...) — the same unconditional-NULL condition, so the metric means the same thing at every threshold.

invalidValues​

invalidValues accepts an allowlist (arguments.validValues), a regex (arguments.pattern), or both — a row fails if it matches neither. When both are present, DQX combines them into a single "invalid" condition (NOT (in the allowlist) OR NOT (matches the pattern)) rather than treating them as two independent thresholds, so mustBeLessOrEqualTo: 5 means at most 5 rows failing either criterion, not up to 5 failing each independently. This combination is a DQX judgment call: other ODCS tooling commonly honors only validValues and ignores arguments.pattern for invalidValues, so pattern support here goes beyond what most implementations cover, even though the spec documents the field:

properties:
  - name: order_status
    logicalType: string
    physicalType: string
    quality:
      - type: library
        metric: invalidValues
        mustBe: 0
        arguments:
          validValues: ["pending", "confirmed", "shipped", "delivered", "cancelled"]
        # → row-level is_in_list(allowed=[...], case_sensitive=true)

  - name: postal_code
    logicalType: string
    physicalType: string
    quality:
      - type: library
        metric: invalidValues
        mustBeLessOrEqualTo: 0.5
        unit: percent
        arguments:
          pattern: '^[0-9]{5}$'
        # → dataset-level check on the invalid percentage
Pattern safety check

arguments.pattern is a Java regular expression (Spark evaluates it with Java's regex engine) and is checked at generation time. A pattern that fails a check is treated as absent — its own warning is logged, and the entry still proceeds using validValues alone if present. A pattern is rejected when it:

  • Could cause catastrophic backtracking (ReDoS): it is longer than 200 characters, or it repeats (+, *, {m,}, {m,n}) a group that contains a variable quantifier or an alternation, at any nesting depth — e.g. (a+)+, ((a+))+, (\w+\s?)+ or (a|ab)*. Optional groups ((a+)?) and exact counts ((\d{3})+) are allowed.
  • Uses Python-only syntax that Java rejects: (?P<name>...) (write (?<name>...)), (?#...) comments, conditionals or {,n}.
  • Is not a valid regular expression, e.g. an unclosed group or class.

Java syntax such as Unicode classes (\p{L}), possessive quantifiers (\d++), atomic groups and inline flags is accepted.

duplicateValues​

The single-column, argument-less form is a property-level entry; the composite-key form is a schema-level entry with arguments.properties.

Counts distinct recurring values, not rows in a duplicated group

For a non-mustBe: 0 threshold, duplicateValues counts the number of distinct values (or key combinations) that recur — for a column holding [A, A, A, B, B, C] that's 2 (A and B each recur), not 5 (the rows sitting in those two groups). unit: percent divides that count by the total row count. This matches how other ODCS-compliant tooling counts duplicates; the ODCS spec text ("Counts duplicate values in a column") doesn't settle it either way, so DQX follows the established interpretation rather than counting affected rows. mustBe: 0 is unaffected by this choice, since both readings agree at zero.

properties:
  - name: order_id
    logicalType: string
    physicalType: string
    quality:
      - type: library
        metric: duplicateValues
        mustBe: 0   # → row-level is_unique(columns=["order_id"])

Rule Names and Row Filters​

Each library entry produces rules named <property>_<metric>_<thresholdField> (or <schema>_<metric>_<thresholdField> for schema-level entries), e.g. email_nullValues_mustBeLessOrEqualTo. Where mustBe: 0 produces more than one row-level rule, each carries a suffix naming its part (_null/_sentinel for missingValues, _allowed/_pattern for invalidValues). Two entries that would still share a name, such as two mustBeLessOrEqualTo entries on the same property, get _2, _3, ... suffixes, so a range expressed as two entries never collides. Suffixes also skip names already used by other rules generated from the contract (for example an explicit rule), so every generated rule name is unique.

A row_filter (or rowFilter) custom property on a library entry is honoured the same way as on an explicit rule: it is emitted as the rule's filter, so the metric is computed over, and reported on, the matching rows only.

quality:
  - metric: nullValues
    mustBeLessOrEqualTo: 5
    customProperties:
      - property: row_filter
        value: "country = 'UK'"

Malformed and Unrecognized Entries​

DQX warns and skips an individual type: library entry — logging the reason and continuing to process the rest of the contract — rather than failing generation, whenever it encounters:

  • A missing or unrecognized metric name (the warning names all five supported metrics).
  • No recognized threshold field set (none of the eight mustBe.../mustNotBe... fields). A threshold field whose value is not a number is skipped on its own; the entry's other valid threshold fields still produce rules.
  • An unrecognized unit (anything other than rows or percent, or anything other than rows for rowCount). Only the entry's thresholds other than mustBe: 0 are skipped, since mustBe: 0 ignores the unit; the warning is still logged, and a kept mustBe: 0 rule records unit: rows — see Rows vs. Percent.
  • A schema-level entry for a property-only metric (nullValues, missingValues, invalidValues), or a property-level rowCount entry.
  • Malformed arguments — not a dict, a missing or wrong-typed key (e.g. arguments.missingValues not a list of scalar values, or containing NaN/infinity), or an empty list where a non-empty one is required.
  • For invalidValues, an entry with neither a usable arguments.validValues list nor a usable arguments.pattern string (after the pattern safety check, described above).

This per-entry warn-and-skip behavior is independent of process_library_rules: even with process_library_rules=True (the default), an individual malformed entry is skipped rather than failing the whole contract. Set process_library_rules=False to skip all type: library entries instead — e.g. if a contract's library entries need to be ignored altogether.

No dedup against predefined rules

DQX does not deduplicate library metric rules against predefined rules generated from schema constraints. A property marked required: true (which generates a predefined is_not_null rule) that also carries a nullValues/mustBe: 0 entry produces two separate is_not_null rules under different names. Both are harmless (they agree on every row), but if you want only one, disable one of the two sources — e.g. generate_predefined_rules=False or omit the redundant nullValues entry.

Disabling Library Metric Processing​

If a contract's type: library entries misbehave, or you'd rather express quality entirely through explicit rules:

# Skip type: library processing, generate only predefined/explicit/text rules
rules = generator.generate_rules_from_contract(
    contract_file="contract.yaml",
    process_library_rules=False  # Skip library metric mapping
)

Complete Usage Example​

Here's a complete example showing contract-based rule generation with every rule type. Each generated rule records how it was produced in user_metadata.rule_type:

  • Predefined (rule_type: "predefined"): derived from the schema, e.g. required, unique and logicalTypeOptions such as pattern, minimum and maximum.
  • Explicit (rule_type: "explicit"): DQX-native checks declared as type: custom with engine: dqx.
  • Text (rule_type: "text_llm"): type: text expectations translated into checks by the LLM (requires process_text_rules=True).
  • Metric (rule_type: "metric"): ODCS type: library metrics such as rowCount or nullValues, see Metric Rule Generation.
  • Schema validation (rule_type: "schema_validation"): one has_valid_schema rule per schema (disable with generate_schema_validation=False).
kind: DataContract
apiVersion: v3.0.2
id: urn:datacontract:iot:sensor_readings
name: IoT Sensor Readings
version: 2.1.0
status: active
description:
  purpose: Quality contract for IoT sensor data
team:
  - username: iot-platform-team
    role: owner

schema:
  - name: sensor_readings
    physicalType: table
    description: Real-time sensor readings from IoT devices
    
    quality:
      # Explicit (rule_type: explicit): DQX-native dataset-level check
      - type: custom
        engine: dqx
        description: Ensure sensor data is fresh and recent
        implementation:
          name: sensor_data_freshness
          criticality: error
          check:
            function: is_data_fresh_per_time_window
            arguments:
              column: reading_timestamp
              window_minutes: 60
              min_records_per_window: 1
              lookback_windows: 24
      
      # Text (rule_type: text_llm): expectation translated into a check by the LLM
      - type: text
        description: |
          The dataset should not contain duplicate sensor readings for the
          same sensor_id and reading_timestamp combination.

      # Metric (rule_type: metric): ODCS library metric, the table must not be empty
      - type: library
        metric: rowCount
        mustBeGreaterOrEqualTo: 1

    # Predefined (rule_type: predefined): derived from the property definitions marked below
    properties:
      - name: sensor_id
        logicalType: string
        physicalType: string
        required: true           # Predefined: is_not_null
        unique: true             # Predefined: is_unique
        logicalTypeOptions:
          pattern: '^SENSOR-[A-Z]{2}-[0-9]{4}$'  # Predefined: regex_match
      
      - name: temperature
        logicalType: decimal
        physicalType: decimal(5,2)
        required: true           # Predefined: is_not_null
        logicalTypeOptions:
          minimum: -40.0           # Predefined: sql_expression (range check)
          maximum: 125.0           # Predefined: sql_expression (range check)
        quality:
          # Text (rule_type: text_llm): property-level expectation
          - type: text
            description: |
              Temperature readings outside normal operating range (-10°C to 50°C)
              should be flagged as warnings for manual review.
      
      - name: status
        logicalType: string
        physicalType: string
        required: true           # Predefined: is_not_null
        logicalTypeOptions:
          pattern: '^(active|maintenance|offline)$'  # Predefined: regex_match

      - name: battery_level
        logicalType: integer
        physicalType: int
        quality:
          # Metric (rule_type: metric): at most 5% of readings may lack a battery level
          - type: library
            metric: nullValues
            mustBeLessOrEqualTo: 5
            unit: percent

Contract Metadata and Lineage​

All generated rules include rich metadata that traces them back to the source contract for lineage and governance:

{
  "name": "sensor_id_not_null",
  "criticality": "error",
  "check": {
    "function": "is_not_null",
    "arguments": {"column": "sensor_id"}
  },
  "user_metadata": {
    "contract_id": "urn:datacontract:iot:sensor_readings",
    "contract_version": "2.1.0",
    "odcs_version": "v3.0.2",
    "schema": "sensor_readings",
    "field": "sensor_id",
    "rule_type": "predefined"  # or "explicit", "text_llm", "metric" or "schema_validation"
  }
}

Metadata Fields​

FieldDescriptionExample
contract_idUnique identifier from contracturn:datacontract:iot:sensor_readings
contract_versionContract version2.1.0
odcs_versionODCS API version (standard version)v3.0.2
schemaSchema/table name (ODCS v3.x)sensor_readings
fieldProperty/column name (single-field property-level rule)sensor_id
fieldsComposite key columns as a JSON-encoded list, since user_metadata is a string map (schema-level duplicateValues metric rule only)'["order_id", "line_number"]'
rule_typeHow rule was generatedpredefined, explicit, text_llm, metric (see Metric Rule Generation), or schema_validation
text_expectationOriginal text (for text-based rules only)Natural language description
metricODCS library metric name (rule_type: "metric" only)rowCount, nullValues, missingValues, invalidValues, or duplicateValues
threshold_fieldODCS threshold field that produced the check (rule_type: "metric" only)mustBe, mustBeGreaterOrEqualTo, mustBeBetween, etc.
unitThreshold unit for the metric (rule_type: "metric" only)rows (default) or percent
dimensionData quality dimension — the contract's own dimension (whitespace trimmed) if set and not blank, else the metric's ODCS defaultcompleteness, uniqueness, or conformity
severityContract's own severity value, recorded verbatim for audit only (never used to derive criticality); present only when the contract sets itAny contract-defined string

This metadata enables:

  • Traceability: Link quality check results back to contract specifications
  • Impact Analysis: Understand which rules are affected by contract changes
  • Governance: Demonstrate compliance with data contracts
  • Debugging: Identify the source of quality rules during troubleshooting

Combining with Other Rule Sources​

Data contract-based rules can be combined with rules from other sources for comprehensive coverage:

from databricks.labs.dqx.profiler.profiler import DQProfiler
from databricks.labs.dqx.profiler.generator import DQGenerator
from databricks.sdk import WorkspaceClient

ws = WorkspaceClient()

# Step 1: Profile the data to discover statistical patterns
profiler = DQProfiler(workspace_client=ws, spark=spark)
df = spark.read.table("production.sales_data")
summary_stats, profiles = profiler.profile(df)

generator = DQGenerator(workspace_client=ws, spark=spark)

# Step 2: Generate profiler-based rules (technical/statistical)
profiler_rules = generator.generate_dq_rules(profiles)

# Step 3: Generate contract-based rules (business/compliance)
contract_rules = generator.generate_rules_from_contract(
    contract_file="/Shared/contracts/sales_data_contract.yaml"
)

# Step 4: Combine all rules
all_rules = profiler_rules + contract_rules

print(f"Total rules: {len(all_rules)}")
print(f"  - Profiler (discovered): {len(profiler_rules)}")
print(f"  - Contract (specified): {len(contract_rules)}")

# Apply combined rules
from databricks.labs.dqx.engine import DQEngine

engine = DQEngine(ws)
result_df = engine.apply_checks_by_metadata(df, all_rules)

Best Practices​

  1. Version Your Contracts: Store data contracts in version control (Git) alongside your code and track changes over time.

  2. Validate Contracts Externally: Use datacontract-cli to validate contracts before using them with DQX:

    pip install datacontract-cli
    datacontract lint /path/to/contract.yaml

  3. Start with Predefined Rules: Begin with schema-based predefined rules, then add explicit rules for business logic that cannot be expressed as constraints.

  4. Use Text Expectations Judiciously: Text-based rules are powerful but require LLM processing. Use them for complex business logic that's easier to express in natural language.

  5. Review Generated Rules: Always review the generated rules before applying to production data, especially text-based rules generated by LLM.

  6. Set Appropriate Criticality: Use error for critical violations that should block pipelines and warn for informational checks.

  7. Combine Approaches: Use profiling for discovering technical patterns and contracts for codifying business requirements.

  8. Document in Contracts: Use the description fields in contracts to explain business context and rationale for constraints.

  9. Track Lineage: Leverage the contract metadata in generated rules to maintain traceability from checks back to contracts.

  10. Iterate on Contracts: Treat contracts as living documents that evolve with your data and business requirements.

Example Use Cases​

Use Case 1: Federated Data Governance​

Enable domain teams to define and own quality contracts:

# Data product team defines their contract
contract_yaml = """
kind: DataContract
apiVersion: v3.0.2
id: urn:datacontract:marketing:customer_360
name: Customer 360 View
version: 1.2.0
status: active
team:
  - username: marketing-analytics-team
    role: owner

schema:
  - name: customer_360
    physicalType: table
    properties:
      - name: customer_id
        logicalType: string
        physicalType: string
        required: true
        unique: true
      - name: email
        logicalType: string
        physicalType: string
        required: true
        logicalTypeOptions:
          pattern: '^[a-zA-Z0-9._%+-]+@.+$'
      - name: lifetime_value
        logicalType: decimal
        physicalType: decimal(18,2)
        logicalTypeOptions:
          minimum: 0
"""

# Central platform team generates and applies rules
generator = DQGenerator(workspace_client=ws, spark=spark)
rules = generator.generate_rules_from_contract(contract_file="customer_360_contract.yaml")

# Rules are automatically tracked back to the owning team's contract

Use Case 2: GDPR Compliance​

Define compliance requirements in contracts:

contract_yaml = """
kind: DataContract
apiVersion: v3.0.2
id: urn:datacontract:gdpr:personal_data
name: GDPR Personal Data
version: 1.0.0
status: active
  
schema:
  - name: user_data
    physicalType: table
    properties:
      - name: email
        logicalType: string
        physicalType: string
        required: true
      - name: consent_date
        logicalType: date
        physicalType: date
        required: true
        quality:
          - type: text
            description: |
              Consent date must be within the last 7 years to comply with
              GDPR data retention policies. Records older than 7 years should
              be flagged for deletion.
"""

# Generate compliance-focused rules
rules = generator.generate_rules_from_contract(
    contract_file="gdpr_contract.yaml",
    process_text_rules=True
)

# Apply rules and track violations
engine = DQEngine(ws)
result_df = engine.apply_checks_by_metadata(df, rules)

Use Case 3: Data Mesh with Contracts​

Implement data mesh principles with contracts as interfaces:

# Producer team defines data product contract
producer_contract = "/Volumes/catalog/schemas/contracts/sales_transactions_v2.yaml"

# Consumer team generates rules from producer's contract
generator = DQGenerator(workspace_client=ws, spark=spark)
rules = generator.generate_rules_from_contract(contract_file=producer_contract)

# Apply rules on consumer side to validate producer data
df = spark.read.table("sales_domain.published.transactions")
result_df, bad_df = engine.apply_checks_by_metadata_and_split(df, rules)

# Alert if producer violates contract
if bad_df.count() > 0:
    print(f"Contract violation detected! {bad_df.count()} rows failed quality checks")
    # Trigger alert to producer team

Troubleshooting​

Error: datacontract-cli not installed​

Problem: You receive an error saying "Data contract functionality requires datacontract-cli".

Solution: Install the datacontract dependencies:

pip install 'databricks-labs-dqx[datacontract]'

Contract Validation Errors​

Problem: Contract fails validation or generates unexpected rules.

Solution:

  • Validate your contract externally using datacontract-cli: 
    datacontract lint /path/to/contract.yaml
  • Ensure your contract follows ODCS v3.x specification (kind: DataContract, apiVersion: v3.0.x)
  • Check that constraints are in logicalTypeOptions for ODCS v3.x
  • Review the ODCS specification

Text Rules Not Generated​

Problem: Text-based expectations are skipped with warnings.

Solution:

  • Install LLM dependencies: pip install 'databricks-labs-dqx[datacontract,llm]'
  • Ensure process_text_rules=True is set (default)
  • Check LLM model access and credentials
  • See AI-Assisted Generation Troubleshooting

Missing Rules for Some Fields​

Problem: Expected predefined rules are not generated for certain fields.

Solution:

  • Verify field constraints are properly defined in the contract
  • Check that generate_predefined_rules=True (default)
  • Review supported constraint types in the Predefined Rule Generation section
  • Some constraints may require specific formats (e.g., format: '%Y-%m-%d' for dates)

Explicit Rules Not Recognized​

Problem: Explicit DQX rules defined in contract are not generated.

Solution:

  • Ensure you're using type: custom and engine: dqx
  • Verify the rule structure matches the required format (see Explicit Rule Generation)
  • Check that the rule is wrapped in an implementation block (ODCS v3.x)
  • Validate YAML syntax and indentation

Contract Metadata Not Preserved​

Problem: Generated rules don't include contract metadata.

Solution:

  • Ensure your contract includes id and info.version fields
  • Verify you're examining the user_metadata field of generated rules
  • Check that you're using a recent version of DQX (≥0.8.0)

Limitations​

  • The data contract feature requires the datacontract-cli library which increases package size.
  • Text-based rule generation requires LLM dependencies and network access to the model endpoint.
  • Only ODCS v3.0.x format is currently supported (earlier versions are not supported).
  • Dataset-level checks beyond the rowCount and duplicateValues library metrics must be defined as explicit DQX rules (custom quality checks).
  • Generated rules quality depends on the completeness and correctness of the contract definition.
  • Complex business logic (text based rules) may still require manual rule definition or refinement.
  • type: library covers five metrics only (see Metric Rule Generation) and is not intended to grow into a general ODCS interpreter; use type: custom with engine: dqx for anything beyond simple row-count/null/duplicate/allowlist checks.