UC Pre-Requisites

Unity Catalog Prerequisites

After all UC Pre-requisites are completed, please continue to Deploy Overwatch section.

This section will walk you through the steps necessary as a prerequisite to deploy Overwatch on Unity Catalog.

  • Workspace should be UC enabled.
  • Overwatch Pipeline Cluster must be UC enabled (single user and runtime version > 11.3+).
  • UC Storage Requirements
  • Create Storage Credentials to be used by the external locations provisioned with appropriate read/write access to the UC External Location (AWS | GCP | AZURE) with privileges:
    • READ FILES
    • WRITE FILES
    • CREATE EXTERNAL TABLE
  • Create UC External location where Overwatch data is to be stored (AWS | GCP | AZURE). Provide ALL PRIVILEGES permission to the principal (user/SP) that is going to run the Overwatch Pipeline.
    • Ensure the external location path is pointing to the same storage location to which the storage credential’s identity was authorized
  • Create a Catalog or identify an existing catalog where overwatch data will be stored. Overwatch code WILL NOT create the catalog, it must be pre-existing.
  • Principal (user/SP) executing the Overwatch Pipeline must have access to the catalog with privileges:
    • USE CATALOG
    • USE SCHEMA
    • SELECT
  • Create ETL and Consumer Schemas (i.e. databases). Overwatch WILL NOT create the Schemas in a UC Deployment. Principal (user/SP) executing the Overwatch Pipeline must have the following privileges on the Schema AND must be an Owner of the Schema.
    • IS OWNER – required since schema metadata is edited and requires schema ownership
      • The schema owner can be a user, service principal, or group. It’s recommended that you provision an Overwatch_Maintainers group and place the Overwatch Admin users in this group along with any service principals that will be writing data to the output and assign this group as owner of the Overwatch schemas.
    • USE SCHEMA
    • CREATE TABLE
    • MODIFY
    • SELECT
  • Overwatch latest version(0.7.2.0+) should be deployed in the workspace
    • 0.7.1.1+ is ok so long as the migration process is completed before executing with Unity Catalog configurations.
  • Other overwatch prerequisites can be found here

SQL Command to Grant Permissions to various UC Objects

The following can be done through the UI or via commands as shown below

SQL Command to grant permissions on Storage Credentials

GRANT READ FILES, WRITE FILES, CREATE EXTERNAL TABLE ON STORAGE CREDENTIAL `<storage-credential-name>` TO `<principal>`;

SQL Command to grant permissions on External Locations

GRANT ALL PRIVILEGES ON EXTERNAL LOCATION `<external-location-name>` TO `<principal>`;

SQL Command to grant permissions on Catalog

GRANT USE CATALOG, USE SCHEMA, SELECT
  ON CATALOG <catalog-name>
  TO `<principal>`

SQL Command to grant permissions on ETL Database

GRANT USE SCHEMA, CREATE TABLE, MODIFY
ON SCHEMA <catalog-name>.<etl-database>
TO `<principal>`;

SQL Command to grant permissions on Consumer Database

GRANT USE SCHEMA, CREATE TABLE, MODIFY
ON SCHEMA <catalog-name>.<consumer-database>
TO `<principal>`;