Skip to main content

Governance & observability

Governance and Observability partners build catalog, lineage, data quality, security, and privacy solutions.

  • Leverage Unity Catalog native capabilities to simplify your integration.
  • Review the data governance documentation for your cloud to understand key concepts: AWS | Azure | GCP
  • Explore and understand Unity Catalog privileges and securable objects to ensure Principle of Least Privilege.
  • Ensure complete integration coverage across all your products' capabilities to avoid a broken user experience.

Documentation: Privileges and Securable Objects

Catalog & lineage products​

Requirements​

  • Use bulk extraction patterns as the first approach for metadata extraction:
    • Use INFORMATION_SCHEMA views to extract business and technical metadata in bulk.
    • Use System tables to extract operational metadata (lineage, billing, audit logs, data classification results).
  • Use Information Schema, System tables, and REST API to extract specific entity details.
  • Use REST API as the last approach if others don't meet detail or latency needs.

Best practices​

  • Write metadata back including lineage using the writing metadata patterns.
  • For model metadata, use REST API until available in system tables.
  • Use native Data Classification from Unity Catalog. Use the Data Classification Results system table to extract this metadata.

Documentation: Metadata Patterns | Information Schema | System Tables | Lineage System Tables | External Lineage (BYOL) | Data Classification

Observability products​

Requirements​

  • Use System tables to extract operational metadata (Audit logs, Query History, Lineage, Billing, Jobs) for account monitoring and cost observability.
  • Use REST API if system tables don't meet detail or latency needs.
  • Run all Data Profiling and Data Quality processing on Databricks to ensure optimal performance by avoiding unnecessary data movement. See data transformation patterns.

Best practices​

  • Leverage native Data Quality Monitoring (Data Profiling, Anomaly detection) as a starting point.
  • If enabled by customer, extract results from System tables for anomaly detection and data profiling metrics.

Documentation: Operational Metadata | System Tables | Data Quality Monitoring

Data security products​

Requirements​

  • Use the access control patterns for authentication and authorization.
  • Use Governed tags with ABAC to manage fine-grained access at scale.
  • Push Data Security policies to Unity Catalog and leave enforcement to Unity Catalog.

Best practices​

  • Leverage native Data Classification from Unity Catalog. Use the Data Classification Results system table to extract this metadata.

Documentation: Access Control | Governed Tags | ABAC | Data Classification

Data privacy products (encryption/tokenization)​

Requirements​

  • Process tokenization/encryption on Databricks to avoid unnecessary data movement.
  • For remote policy or key retrieval, use external APIs capability on Python UDFs.
  • Use Governed tags with ABAC to manage fine-grained protection at scale.
  • Ensure the integration works with Serverless SQL Warehouses.

Best practices​

Documentation: Python UDFs with External APIs | UDFs | Serverless SQL Warehouses

What's next​