Databricks system tables offer a robust mechanism for managing and understanding the intricacies of a Databricks environment. From metadata exploration to performance optimization and security auditing, these tables provide a comprehensive foundation for users and administrators to derive valuable insights and make informed decisions in a collaborative and data-driven environment. Follow these documents to know more about System Tables Databricks Docs for System Tables
In our continuous efforts to enhance the efficiency and robustness of Overwatch, we have transitioned to a system table integration for audit logs. This strategic move brings forth numerous advantages, further optimizing the monitoring and logging capabilities of the tool. Currently, our integration focuses on leveraging system table audit logs, but in future releases, we will be expanding our integration to include other system tables as well. Follow these documents to know more about System Table Audit Logs
Advantages of Moving to System Table Integrations:
Seamless Integration with Overwatch: System table integration seamlessly aligns with Overwatch. The synergy between system tables and Overwatch ensures a harmonious integration, providing a unified platform for comprehensive monitoring and analysis.
Simplified Setup within Overwatch: Integration with system tables leads to a simplified setup within Overwatch. The need to configure audit log settings independently within the Overwatch configurations is eliminated. This simplicity ensures a more user-friendly experience for both setup and ongoing management.
Effortless Migration: The migration process to system tables is designed to be effortless within the Overwatch framework. The transition is smooth, minimizing disruptions and ensuring continuous monitoring capabilities without requiring extensive reconfigurations.
Extended Data Retention for Azure Deployments in Overwatch: Particularly advantageous for Overwatch Azure deployments, the use of system tables extends the audit log retention period beyond 30 days for audit log bronze. This eliminates the previous dependency on Events Hub’s retention period.
As mentioned earlier, the deployment process is almost the same as other deployments. Therefore, the Deployment Guide contains relevant information that will guide your deployment. Specific differences are referenced below and detailed in this section of the documentation.
Once the deployment is migrated to System Tables, it cannot be reverted back to source the audit logs from S3, GC Bucket or Azure Event Hubs i.e Legacy Deployment.