Skip to main content

Metadata and access control

Partner integrations should use serverless SQL warehouses for all metadata operations—reading catalogs, schemas, tables, and access policies—to ensure consistent governance and optimal performance. Review Unity Catalog privileges and securable objects, apply least-privilege access, and filter out system assets for a cleaner user experience.

Reading metadata​

General metadata lookup​

Use Databricks SQL driver metadata APIs, GET calls on SDKs, or SHOW commands to retrieve high-level metadata such as catalogs, schemas, tables, and other assets.

Detailed metadata for specific objects​

For more detailed metadata, use SHOW and DESCRIBE EXTENDED SQL commands.

Bulk metadata extraction​

Business and technical metadata​

Use INFORMATION_SCHEMA views to query metadata. The information schema provides a SQL-based, self-describing API to the metadata.

Operational metadata​

Use system tables to extract operational metadata in bulk, including lineage, billing, audit logs, and data classification results. System tables are Databricks-hosted analytical stores of your account's operational data, located in the system catalog.

Documentation: SHOW statements | DESCRIBE statements | Information Schema | System tables

Writing metadata​

  1. The recommended starting point for creating and modifying metadata (e.g., tables, views, catalogs, tags) is via SQL DDL statements.
  2. To include external assets in data lineage, use the bring-your-own-lineage API.

Documentation: SQL DDL statements | External Lineage (BYOL)

Managing access control​

Databricks provides multiple layers of access control. Use SQL to write policies to Unity Catalog and let Unity Catalog handle enforcement. To manage access at scale without granting permissions on individual objects, use ABAC with governed tags. For advanced scenarios, use the Databricks REST API.

Documentation: Access control layers | ABAC | Data access control | REST API

What's next​