Databricks Design Patterns

This section provides guidance on common implementation patterns for partner integrations with Databricks. These patterns cover the full data lifecycle—from authentication and access control through ingestion, transformation, serving, and AI capabilities.

Pattern overview

Pattern	Description
Access and Authentication	Networking architecture, OAuth flows, token federation, and authentication best practices
Metadata and Access Control	Unity Catalog integration, reading/writing metadata, and managing access policies
Data Ingestion	File-based ingestion, streaming, CDC, and data loading patterns
Data Transformation	Incremental and batch transformations, medallion architecture, and orchestration
Serving and Consumption	SQL warehouses, model serving, Delta Sharing, and streaming output
Lakebase (OLTP)	PostgreSQL-compatible transactional workloads with Lakebase
AI Capabilities	AI agents, Genie, AI Functions, Vector Search, and foundation models

Data lifecycle

Partners typically implement patterns across these phases:

┌─────────────┐     ┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Ingest    │ ──▶ │  Transform  │ ──▶ │    Serve    │ ──▶ │   Consume   │
│             │     │             │     │             │     │             │
│ • Files     │     │ • Medallion │     │ • SQL WHs   │     │ • BI Tools  │
│ • Streaming │     │ • SDP       │     │ • APIs      │     │ • Apps      │
│ • CDC       │     │ • Jobs      │     │ • Sharing   │     │ • AI/ML     │
└─────────────┘     └─────────────┘     └─────────────┘     └─────────────┘

Foundational requirements

All patterns build on these foundational requirements:

1. Unity Catalog - All data assets must be governed through Unity Catalog
2. OAuth Authentication - Use OAuth 2.0 flows for secure authentication
3. User-Agent Telemetry - Implement telemetry for usage attribution

See Integration Requirements for complete guidance.

Getting started

Choose your starting point based on your integration type:

Integration Type	Start Here
BI / Analytics tools	Serving and Consumption → Metadata
Data ingestion / ETL	Data Ingestion → Transformation
AI / Agent platforms	AI Capabilities → Access and Authentication
Governance / Catalog	Metadata and Access Control → Access and Authentication
OLTP / Transactional	Lakebase (OLTP)

What's next

• Start with authentication: Understand access and authentication patterns
• Review requirements: See the integration requirements for foundational guidance
• Configure telemetry: Set up User-Agent telemetry for attribution
• Explore reference architectures: Browse the Databricks Architecture Center for detailed diagrams and patterns