Skip to main content

Reconcile Configuration Reference

This page covers the full configuration schema for Lakebridge Reconcile. For setup steps and prerequisites, see the Reconcile Guide.

Report Types​

report typedescriptionkey outputs
schemaReconcile schema of source and target — validate that data types are the same or compatibleschema_comparison, schema_difference
rowReconcile data at row level (hash value comparison). Use when there are no join columns.missing_in_src, missing_in_tgt
dataReconcile data at row and column level using join_columns to identify per-row, per-column mismatchesmismatch_data, missing_in_src, missing_in_tgt, threshold_mismatch, mismatch_columns
allCombination of data + schemaAll outputs above

See Reconcile Data Flow Examples for visualizations of each report type.

Config File Naming​

recon_config_<DATA_SOURCE>_<UNITY_CATALOG_CONNECTION_NAME_OR_CATALOG>_<REPORT_TYPE>.json

Place the file in .lakebridge/ in your Databricks workspace home folder.

The filename must match the case of <UNITY_CATALOG_CONNECTION_NAME_OR_CATALOG> exactly. For example, if the source connection is conn-oracle-prod, the filename is recon_config_oracle_conn-oracle-prod_data.json.

Examples by source:

recon_config_snowflake_sample_data_all.json
source:
    dialect: snowflake
    catalog: sample_data
    schema: default
    uc_connection_name: example_connection_snowflake
target:
    catalog: migrated
    schema: migrated
report_type: all

TABLE Config Schema​

@dataclass
class Table:
    source_name: str
    target_name: str
    aggregates: list[Aggregate] | None = None
    join_columns: list[str] | None = None
    jdbc_reader_options: JdbcReaderOptions | None = None
    select_columns: list[str] | None = None
    drop_columns: list[str] | None = None
    column_mapping: list[ColumnMapping] | None = None
    transformations: list[Transformation] | None = None
    column_thresholds: list[ColumnThresholds] | None = None
    filters: Filters | None = None
    table_thresholds: list[TableThresholds] | None = None
    sampling_options: SamplingOptions | None = None

Configuration Parameters​

ParameterTypeRequiredDescription
source_namestringYesSource table name
target_namestringYesTarget table name
join_columnslist[string]NoPrimary key columns for row-level joins
aggregateslist[Aggregate]NoAggregate reconciliation rules (see Aggregates)
jdbc_reader_optionsobjectNoJDBC read parallelization (see JDBC Reader Options)
select_columnslist[string]NoColumns to include in reconciliation
drop_columnslist[string]NoColumns to exclude from reconciliation
column_mappinglist[ColumnMapping]NoSource-to-target column name mapping
transformationslist[Transformation]NoColumn-level SQL transformation expressions
column_thresholdslist[ColumnThresholds]NoAcceptable variance per column
table_thresholdslist[TableThresholds]NoAcceptable mismatch rate at table level
filtersFiltersNoWHERE-clause filters for source and/or target
sampling_optionsSamplingOptionsNoCaps the rows pulled into each sample bucket — mismatch, missing-in-source, missing-in-target (see Sampling Options)

JDBC Reader Options​

Use JDBC reader options to parallelize reads from large tables.

@dataclass
class JdbcReaderOptions:
    num_partitions: int
    partition_column: str
    lower_bound: str
    upper_bound: str
    fetchsize: int = 0
FieldTypeRequiredDescription
num_partitionsintYesNumber of Spark partitions for parallel reads
partition_columnstringYesColumn used for partitioning (choose high-cardinality column for composite keys)
lower_boundstringYesMinimum value for partitioning
upper_boundstringYesMaximum value for partitioning
fetchsizeintNo (default: 100)Rows per JDBC round-trip

Column Mapping​

Map source column names to target column names when they differ.

@dataclass
class ColumnMapping:
    source_name: str
    target_name: str

Drop Columns​

Exclude specific columns from reconciliation (for data or all report types).

"drop_columns": ["comment", "audit_timestamp"]

Transformations​

Apply SQL expressions to source or target columns before comparison. Use this when data types or formats differ between systems.

@dataclass
class Transformation:
    column_name: str
    source: str
    target: str | None = None
Handling Nulls

When you provide a transformation expression, Reconcile uses it as-is. You must handle nulls and cast to string explicitly in the expression. Use coalesce(cast(... as string), '_null_recon_') to avoid null mismatches and incompatibility with the internal transformations that rely on strings.

Timestamp Columns

Convert timestamps to Unix epoch strings for cross-platform comparison:

Transformation(
    column_name='UPDATE_TIMESTAMP',
    source="coalesce(cast(EXTRACT(epoch_millisecond FROM UPDATE_TIMESTAMP) as string), '_null_recon_')",
    target="coalesce(cast(unix_millis(UPDATE_TIMESTAMP) as string), '_null_recon_')"
)

Hash Expression​

Override the row-hash function with your own SQL. Each value is raw SQL containing a single {} placeholder that Lakebridge replaces with the concatenated hash input.

  • source is required when the block is set: the source-side expression Lakebridge will emit on the source dialect.
  • target defaults to sha2({}, 256) (the target is always Databricks); override it if you use a non-SHA-256 hash.

The block itself is optional for most dialects, omit it to use the source dialect's default. For Teradata sources the block is mandatory, since Teradata has no out-of-the-box cryptographic hash.

@dataclass
class HashExpressionOverrides:
    source: str
    target: str = "sha2({}, 256)"
Same hash on both sides

The source and target expressions must produce the same string for the same logical row — that is what the comparison query joins on. Lakebridge wraps both sides with LOWER(...) automatically, so case alone won't cause mismatches; the bytes produced by the hash itself must match across systems.

Column Thresholds​

Allow a bounded variance between source and target column values.

"column_thresholds": [
  {
    "column_name": "price",
    "lower_bound": "-5%",
    "upper_bound": "5%",
    "type": "float"
  }
]
FieldTypeRequiredDescription
column_namestringYesColumn to apply the threshold to
lower_boundstringYesLower bound of acceptable difference
upper_boundstringYesUpper bound of acceptable difference
typestringYesColumn type (supports SQLGlot DataType.NUMERIC_TYPES and DataType.TEMPORAL_TYPES)

Table Thresholds​

Allow a bounded mismatch rate at the table level.

"table_thresholds": [
  {
    "lower_bound": "0%",
    "upper_bound": "5%",
    "model": "mismatch"
  }
]

Bounds must be non-negative, and lower bound must not exceed upper bound. Only "mismatch" is supported for model.

Filters​

Apply WHERE-clause filters to source and/or target before reconciliation.

"filters": {
  "source": "lower(dept_name) = 'finance'",
  "target": "lower(department_name) = 'finance'"
}
note

Filters must use dialect-specific SQL expressions — they are applied directly without any transformation by Reconcile.

Sampling Options​

For data and all report types, Reconcile captures sample rows for each mismatch / missing-in-source / missing-in-target bucket. sampling_options controls how many rows are sampled and how they are drawn.

note

sampling_options applies to row-level reconcile (data / all report types) only. Aggregates Reconciliation always samples up to 50 (DEFAULT_SAMPLE_ROWS) rows per missing-in-source / missing-in-target set and ignores sampling_options (both the sampling method and size).

SamplingSpecifications​

Controls the sample-size policy.

@dataclass
class SamplingSpecifications:
    type: SamplingSpecificationsType = SamplingSpecificationsType.COUNT
    value: int | float = 50
fieldtypedescriptiondefault
typecount | fractioncount = absolute row cap. fraction is reserved and currently disabled — passing it raises a config error.count
valueint (float for fraction)Maximum rows per sample set (mismatch / missing-in-source / missing-in-target).50

Engine-aware bounds on value:

Each source has a max_sample_size (a property on its connector):

  • Databricks source: honored up to 50_000.
  • Non-Databricks source: honored up to 400.
  • Values above the source's limit are capped to it with a WARNING log (exceeds the limit of <max> for <connector>; capping to <max>).
  • value <= 0 falls back to 50.

SamplingOptions​

Wraps SamplingSpecifications and selects the sampling method.

@dataclass
class SamplingOptions:
    method: SamplingOptionMethod = SamplingOptionMethod.RANDOM
    specifications: SamplingSpecifications = field(default_factory=SamplingSpecifications)
    stratified_columns: list[str] | None = None
    stratified_buckets: int | None = None
fieldtypedescriptiondefault
methodrandom | stratifiedSampling strategy.random
specificationsSamplingSpecificationsSize policy (see above).count / 50
stratified_columnslist[string]Required when method = stratified. Columns whose values define the strata.null
stratified_bucketsintRequired when method = stratified. Number of strata buckets. Clamped to [2, 50].null

When method = stratified, rows are bucketed by pmod(abs(hash(stratified_columns)), stratified_buckets) and sampling proceeds within each bucket. Missing stratified_columns or stratified_buckets raises a ValueError.

Examples

Random sampling (Databricks source) — limit each sample set (mismatch, missing-in-source, missing-in-target) to 500 rows:

"sampling_options": {
  "method": "random",
  "specifications": { "type": "count", "value": 500 }
}

Shortest form (relies on defaults of random and count):

"sampling_options": { "specifications": { "value": 500 } }

Stratified sampling — 10 strata on ss_store_sk, up to 500 sample rows total (allocated proportionally across buckets):

"sampling_options": {
  "method": "stratified",
  "specifications": { "type": "count", "value": 500 },
  "stratified_columns": ["ss_store_sk"],
  "stratified_buckets": 10
}

Aggregates Reconciliation​

Reconcile aggregate metrics (MIN, MAX, COUNT, SUM, AVG, etc.) between source and target instead of comparing raw rows.

note

Aggregates reconcile always samples up to 50 (DEFAULT_SAMPLE_ROWS) rows per missing-in-source / missing-in-target set; sampling_options does not apply here.

Supported Functions​

MIN, MAX, COUNT, SUM, AVG, MEAN, MODE, STDDEV, VARIANCE, MEDIAN

Aggregate Config​

@dataclass
class Aggregate:
    agg_columns: list[str]
    type: str
    group_by_columns: list[str] | None = None
note

select_columns and drop_columns are ignored for aggregate reconciliation. column_mapping, transformations, jdbc_reader_options, and filters are applied.

Complete Examples​

Standard Reconcile Config​

{
  "tables": [
    {
      "source_name": "product_prod",
      "target_name": "product",
      "jdbc_reader_options": {
        "num_partitions": 10,
        "partition_column": "p_id",
        "lower_bound": "0",
        "upper_bound": "10000000"
      },
      "join_columns": ["p_id"],
      "drop_columns": ["comment"],
      "column_mapping": [
        { "source_name": "p_id", "target_name": "product_id" },
        { "source_name": "p_name", "target_name": "product_name" }
      ],
      "transformations": [
        {
          "column_name": "creation_date",
          "source": "creation_date",
          "target": "to_date(creation_date,'yyyy-mm-dd')"
        }
      ],
      "column_thresholds": [
        { "column_name": "price", "lower_bound": "-50", "upper_bound": "50", "type": "float" }
      ],
      "table_thresholds": [
        { "lower_bound": "0%", "upper_bound": "5%", "model": "mismatch" }
      ],
      "filters": {
        "source": "p_id > 0",
        "target": "product_id > 0"
      },
      "sampling_options": {
        "method": "random",
        "specifications": { "type": "count", "value": 100 }
      }
    }
  ]
}

Aggregates Reconcile Config​

{
  "tables": [
    {
      "source_name": "product_prod",
      "target_name": "product",
      "join_columns": ["p_id"],
      "aggregates": [
        { "type": "MIN", "agg_columns": ["discount"], "group_by_columns": ["p_id"] },
        { "type": "AVG", "agg_columns": ["discount"], "group_by_columns": ["p_id"] },
        { "type": "MAX", "agg_columns": ["p_id"], "group_by_columns": ["creation_date"] },
        { "type": "SUM", "agg_columns": ["p_id"] }
      ],
      "jdbc_reader_options": {
        "num_partitions": 10,
        "partition_column": "p_id",
        "lower_bound": "0",
        "upper_bound": "10000000"
      }
    }
  ]
}
Key Considerations
  1. Column names are always converted to lowercase.
  2. Case insensitivity and collation are not currently supported.
  3. Always reference source column names in all configs, except transformations and filters — those use dialect-specific SQL applied directly.
  4. When no user transformation is provided, Reconcile applies default transformations based on the source data type.