Silver Transformations Schema
The silver transformations file defines the SQL logic applied when writing from a bronze table to a silver table. Each entry maps one source to one silver target.
The file path is referenced in the onboarding file via the silver_transformation_json_{env} field (e.g. silver_transformation_json_prod).
Both JSON and YAML formats are supported.
File Structure
The file is a top-level array of transformation objects. Each object defines the transformation for one silver table.
JSON Example
[
{
"target_table": "customers_silver",
"source_format": "delta",
"select_exp": [
"id as customer_id",
"name",
"upper(email) as email"
],
"where_clause": "is_active = true"
},
{
"target_table": "transactions_silver",
"source_format": "delta",
"select_exp": [
"transaction_id",
"customer_id",
"amount",
"cast(transaction_date as date) as transaction_date"
]
}
]
YAML Equivalent
- target_table: customers_silver
source_format: delta
select_exp:
- "id as customer_id"
- "name"
- "upper(email) as email"
where_clause: "is_active = true"
- target_table: transactions_silver
source_format: delta
select_exp:
- transaction_id
- customer_id
- amount
- "cast(transaction_date as date) as transaction_date"
Fields
| Field | Type | Required | Description |
|---|---|---|---|
target_table | string | Yes | Name of the silver target table. Must match the silver_table value in the onboarding file. |
source_format | string | No | Source format hint — typically delta for bronze-to-silver flows |
select_exp | array of strings | Yes | SQL column expressions applied to the source data. Each element is a valid Spark SQL expression, e.g. "id as customer_id", "upper(email) as email" |
where_clause | string | No | Optional SQL filter expression applied before writing to the silver table. Rows that do not match are excluded from the silver output. |
target_partition_cols | array of strings | No | Partition columns for the silver table output |
Usage Notes
select_exp entries support any Spark SQL expression valid in a SELECT clause. This includes functions, casts, conditional expressions (CASE WHEN), and column aliases.
If where_clause is used in combination with a silver fanout scenario, each transformation entry can have its own filter, effectively splitting the bronze data into multiple filtered silver tables.