Skip to main content

Apache Iceberg REST Catalog Integration

This guide describes how ISV partners integrate with Unity Catalog Managed Apache Iceberg tables using the Apache Iceberg REST Catalog.

Supported Table Type: Unity Catalog Managed Apache Iceberg Tables

Overview​

The Apache Iceberg REST Catalog (IRC) enables partner applications to access Unity Catalog Managed Apache Iceberg tables using the open Apache Iceberg REST specification. By integrating through the Iceberg REST Catalog, partners can use standard Apache Iceberg APIs while Unity Catalog manages metadata, governance, authorization, and storage access.

This guide describes the recommended integration architecture and Partner Integration Validation requirements for ISV partner integrations. Configuration examples, supported Apache Iceberg client versions, and API reference are maintained in the official Databricks documentation.

The Apache Iceberg REST Catalog is recommended for partner products that directly create, read, or write Unity Catalog Managed Apache Iceberg tables, including query engines, ETL/ELT platforms, and data integration products.

Scope​

This guide covers partner integrations with Unity Catalog Managed Apache Iceberg tables using the Apache Iceberg REST Catalog. Integrations using foreign Iceberg tables or other table formats are outside the scope of this guide.

Integration overview​

Before a partner application connects through the Apache Iceberg REST Catalog, the customer is responsible for configuring the Unity Catalog environment. This includes:

  • Configuring a Managed Storage Location at the metastore, catalog, or schema level.
  • Enabling External Data Access for the Unity Catalog metastore.
  • Configuring OAuth authentication for a Databricks Service Principal (recommended) or user.
  • Granting the required Unity Catalog permissions, including EXTERNAL USE SCHEMA and the necessary catalog, schema, and table privileges required by the partner application.

These are customer-managed administrative tasks and are outside the scope of the partner integration.

The Unity Catalog Apache Iceberg REST Catalog endpoint is:

/api/2.1/unity-catalog/iceberg-rest

Once configured, the partner application authenticates with Databricks using OAuth and connects to the Apache Iceberg REST Catalog. Unity Catalog authenticates and authorizes the request through the Apache Iceberg REST Catalog. When supported by the selected Apache Iceberg client, Unity Catalog returns Iceberg metadata and temporary cloud storage credentials (Credential Vending), enabling the client to access the underlying managed table directly.

Integration architecture​

Implementation note​

Unlike traditional cloud storage integrations, partner applications do not access customer-managed cloud credentials directly. Unity Catalog authorizes access and, when supported by the selected Apache Iceberg client, vends temporary cloud storage credentials for the requested operation.

Integration requirements​

All standard Connected Partner integration requirements apply. In addition, for Apache Iceberg REST Catalog integrations, the following capabilities are required for Partner Integration Validation.

RequirementDescription
AuthenticationSupport OAuth Machine-to-Machine (recommended) or User-to-Machine authentication. See the Databricks OAuth documentation.
Apache Iceberg REST CatalogIntegrate with Unity Catalog using the Apache Iceberg REST Catalog. See Access Databricks Tables from Apache Iceberg Clients.
Supported Table TypeValidate against Unity Catalog Managed Apache Iceberg tables.
Credential VendingSupport Unity Catalog Credential Vending when available in the selected Apache Iceberg client. See Credential Vending for External Engines.
User-Agent TelemetrySet the HTTP User-Agent header on all Apache Iceberg REST Catalog requests. See REST API Telemetry Attribution – Iceberg Integration.

Best practices​

The following best practices are recommended when designing partner integrations with the Apache Iceberg REST Catalog.

Recommended Iceberg client version

Databricks recommends Apache Iceberg client version 1.9.2 or later. Refer to the official Databricks documentation for the latest supported versions and capabilities. Apache Iceberg client 1.11.0 and later supports automatic OAuth token refresh.

Authentication​

  • Use OAuth (mandatory requirement) instead of Personal Access Tokens (PATs) for production deployments.
  • Use a Databricks Service Principal for service-to-service integrations whenever possible.
  • Implement OAuth token refresh according to the capabilities of the selected Apache Iceberg client. Iceberg client version 1.11.0 and above supports automatic token refresh; earlier versions require manual token refresh before expiry (default: 1 hour).
  • When configuring OAuth credentials in partner UIs, avoid displaying client secrets in plain text. Use masked input fields and secure credential storage.

Metadata and authorization​

  • Treat Unity Catalog as the system of record for metadata, governance, and authorization.
  • Use the Apache Iceberg REST Catalog for metadata operations instead of directly interacting with Unity Catalog APIs.
  • Respect Unity Catalog authorization and avoid caching permissions beyond the authenticated session's lifetime.

Storage access​

  • Rely on Unity Catalog-issued temporary credentials instead of requiring customers to manage long-lived cloud storage credentials.
  • Do not require customers to provide cloud IAM credentials directly to the partner application.
  • Validate Credential Vending across all supported cloud providers before certifying your integration.
  • For AWS integrations, ensure the AWS region is configured via environment variable (AWS_REGION) or Spark config (spark.hadoop.fs.s3a.region), as vended credentials do not include region information.
  • For Azure integrations, use Apache Iceberg client version 1.9.2 or above, which includes fixes for Azure ADLS credential vending compatibility.
  • For GCP integrations, no additional region configuration is required. See Credential Vending for GCP.

What's next​

ResourceDescription
Access Databricks Tables from Apache Iceberg ClientsOfficial Databricks documentation for Iceberg REST Catalog integration
Configure Managed StorageUnity Catalog managed storage configuration
Enable External Data AccessAdministrative setup for external engine access
Credential Vending for External EnginesTemporary credential vending documentation
Apache Iceberg REST Catalog SpecificationOpen specification for Iceberg REST Catalog