Sometimes upgrading from one version to the next requires a schema change. In these cases, the CHANGELOG will be explicit. Upgrades MUST be executed WITH the new library (jar) and before the pipeline is executed. The general upgrade process is:
Basic pseudocode can be found below as a reference. For actual version upgrade scripts please reference the upgrade scripts linked to your target version in the Changelog.
When a schema upgrade is required between versions, this step cannot be skipped. Overwatch will not allow you to continue on a version that requires a newer schema.
A sample notebook is provided below for reference.
import com.databricks.labs.overwatch.utils.Upgrade import com.databricks.labs.overwatch.pipeline.Initializer val prodArgs = """<myConfigCompactString>""" // from runner notebook // A more verbose example is available in the example notebooks referenced above val prodWorkspace = Initializer(prodArgs) val upgradeReport = Upgrade.upgradeTo060(prodWorkspace, ...) display(upgradeReport) Upgrade.finalize060Upgrade("<overwatch_etl_db_name>")