System Table Integration

Databricks system tables offer a robust mechanism for managing and understanding the intricacies of a Databricks environment. From metadata exploration to performance optimization and security auditing, these tables provide a comprehensive foundation for users and administrators to derive valuable insights and make informed decisions in a collaborative and data-driven environment. Follow these documents to know more about System Tables Databricks Docs for System Tables

In our continuous efforts to enhance the efficiency and robustness of Overwatch, we have transitioned to a system table integration for audit logs. This strategic move brings forth numerous advantages, further optimizing the monitoring and logging capabilities of the tool. Currently, our integration focuses on leveraging system table audit logs, but in future releases, we will be expanding our integration to include other system tables as well. Follow these documents to know more about System Table Audit Logs

Advantages of Moving to System Table Integrations:

  • Seamless Integration with Overwatch: System table integration seamlessly aligns with Overwatch. The synergy between system tables and Overwatch ensures a harmonious integration, providing a unified platform for comprehensive monitoring and analysis.

  • Simplified Setup within Overwatch: Integration with system tables leads to a simplified setup within Overwatch. The need to configure audit log settings independently within the Overwatch configurations is eliminated. This simplicity ensures a more user-friendly experience for both setup and ongoing management.

  • Effortless Migration: The migration process to system tables is designed to be effortless within the Overwatch framework. The transition is smooth, minimizing disruptions and ensuring continuous monitoring capabilities without requiring extensive reconfigurations.

  • Extended Data Retention for Azure Deployments in Overwatch: Particularly advantageous for Overwatch Azure deployments, the use of system tables extends the audit log retention period beyond 30 days for audit log bronze. This eliminates the previous dependency on Events Hub’s retention period.

System Table Integration Specific Configuration Details

As mentioned earlier, the deployment process is almost the same as other deployments. Therefore, the Deployment Guide contains relevant information that will guide your deployment. Specific differences are referenced below and detailed in this section of the documentation.

Migrating to System Table

Once the deployment is migrated to System Tables, it cannot be reverted back to source the audit logs from S3, GC Bucket or Azure Event Hubs i.e Legacy Deployment.

The migration to system tables is straightforward. Organizations can effortlessly transition from existing configurations to system table integrations, minimizing downtime and ensuring a smooth migration experience. Follow the below steps to migrate to System Table Integration: Configuration Details

Fetching data from Multi Account System Table

With the integration of System Table in Overwatch, fetching system table data from multiple different accounts becomes easy, enabling seamless multi-account deployments. This setup requires minimal adjustments in configuration and additional infrastructure setup in the workspaces of other accounts. Follow the link to configure Multi Account System Table Integration