Upgrade_0610_91LTS(Scala)

Import Notebook

Upgrade Procedure

  • ALERT -- You are choosing to remaing on DBR 9.1LTS. This is ok but requires a full rebuild of the underlying spark_events_bronze table which can be compute intensive. If possible upgrade all Overwatch related jobs to 10.4LTS+ and use the standard Upgrade notebook
  • Complete the upgrade steps below
    • Only required to be executed from a single workspace
  • Upgrade all jobs to run using the new JAR "0610"
  • If you have an external optimize job running -- be sure to upgrade the JAR there as well

Technical Details

  • Upgrade job_status_silver and job_gold schema to repair potential issues with schema size described in further detail in Issue 323
  • Upgrade spark_events_bronze schema to minimize schema size on new fields being produced in later versions of DBR.
    • If using 10.4LTS, the problematic column[s] will be deprecated and the table doesn't require a rebuild due to new features (column mapping) in the 10.4LTS.
    • If unable to upgrade DBR to 10.4LTS, a full table rebuild will occur and can be compute intensive depending on table size.
import com.databricks.labs.overwatch.utils._
import com.databricks.labs.overwatch.pipeline.Initializer
val compactString = """compact string"""
 
val prodWorkspace = Initializer(compactString)
val upgradeReport = Upgrade.upgradeTo0610(prodWorkspace.getConfig.databaseName, enableUpgradeBelowDBR104 = true)
display(upgradeReport)
Upgrade.finalize0610Upgrade("overwatch_etl")

Upgrade All Other Metastores

If using an external metastore -- you may skip this step

Otherwise, run the following on all workspaces on a cluster with the Overwatch Jar attached

This will upgrade the schema -- if you skip this step Overwatch will fail due to incompatible schema.

import com.databricks.labs.overwatch.utils._
SchemaTools.modifySchemaVersion("overwatch_etl", "0.610")