Sometimes upgrading from one version to the next requires a schema change. In these cases, the CHANGELOG will be explicit. Upgrades MUST be executed WITH the new library (jar) and before the pipeline is executed. The general upgrade process is:
Basic pseudocode can be found below as a reference. For actual version upgrade scripts please reference the upgrade scripts linked to your target version in the Changelog.
When a schema upgrade is required between versions, this step cannot be skipped. Overwatch will not allow you to continue on a version that requires a newer schema.
A sample notebook is provided below for reference.
import com.databricks.labs.overwatch.utils.Upgrade
import com.databricks.labs.overwatch.pipeline.Initializer
val prodArgs = """<myConfigCompactString>""" // from runner notebook
// A more verbose example is available in the example notebooks referenced above
val prodWorkspace = Initializer(prodArgs)
val upgradeReport = Upgrade.upgradeTo060(prodWorkspace, ...)
display(upgradeReport)
Upgrade.finalize060Upgrade("<overwatch_etl_db_name>")