Skip to main content

Agentic AI

Overview​

Partners increasingly deliver AI assistants and agents as customer-facing product capabilities. This pattern describes how to build a production agent whose intelligence layer runs on Databricks while the partner retains ownership of the user experience, application session, and product branding. The required authorization model follows the deployment model: per-user, on-behalf-of-user authorization for Customer Managed deployments, and per-tenant service-principal authorization for Partner Hosted (SaaS) deployments.

The resulting agent can answer natural-language questions over governed data, invoke tools, perform multi-step workflows, maintain context across sessions, and stream responses into the partner application. Databricks endpoints, credentials, and administrative interfaces are not exposed to the end user. When Genie contributes to answers shown in the partner UI, satisfy the applicable Genie attribution requirements; Genie Space citations and preserved response links apply in Customer Managed deployments where end users can access the workspace.

This page covers architecture, authorization, Genie integration, memory, model serving, and production tracing for agents deployed as Databricks apps.

Use this pattern when the product requires one or more of the following:

  • A natural-language interface over governed customer data, such as analytics, reporting, or "ask your data."
  • An assistant that invokes tools, performs multi-step workflows, and streams responses.
  • Conversation state or durable memory that persists across sessions.
  • A Databricks-hosted agent embedded in the partner's own branded application.
  • Authorization aligned to the deployment model — each end user's Unity Catalog permissions (Customer Managed) or per-tenant service-principal isolation (Partner Hosted).
  • Production tracing, evaluation, monitoring, and model-cost governance.

For related ISV validation requirements, see AI Agents and Tools and integration requirements.

Core principles​

Databricks credentials remain server-side​

Under this reference design, the browser communicates only with the partner backend. For every interactive user-facing request, the backend calls the Databricks app using a Databricks OAuth token — one representing the authenticated end user (Customer Managed) or the tenant's mapped service principal (Partner Hosted). Databricks credentials, access tokens, workspace endpoints, and internal service URLs are not returned to the browser.

Documentation: Connect to an API Databricks app using token authentication | Best practices for Databricks Apps

Authorization follows the deployment model​

The required identity depends on who owns the Databricks environment. In Customer Managed deployments, every interactive request must carry a valid Databricks OAuth token representing the authenticated end user; the backend cannot substitute a service principal, and the request must fail closed if a user token cannot be obtained or validated. In Partner Hosted (SaaS) deployments, end users have no Databricks identity, so each request executes as the tenant's mapped service principal, and a single service principal shared across tenants is prohibited. In both models the partner backend cannot establish authorization by asserting a username or passing a custom identity header — it must present a Databricks-recognized OAuth token.

Documentation: Configure authorization in a Databricks app | Connect to an API Databricks app using token authentication | Authenticate access to Databricks using OAuth token federation

Governance is enforced by Databricks​

Because user authorization is required, Databricks evaluates access using the end user's existing permissions, including Unity Catalog grants, row filters, and column masks. The application's declared OAuth scopes further restrict which Databricks APIs the agent can call on the user's behalf. Application logic must not attempt to reproduce Unity Catalog authorization rules, but it must still enforce product-level controls such as tenant routing, workflow permissions, and safe tool selection.

Documentation: Configure authorization in a Databricks app

Tenant and memory isolation are explicit​

Unity Catalog governs access to the memory store. A trusted server-side scope partitions memory within the store by user, tenant, or another stable security boundary. The scope must be derived from authenticated identity in server-side code; it must never be accepted from the browser or selected by the model.

Documentation: Managed agent memory

Observable execution is traced​

Model calls, agent execution, tool invocations, inputs, outputs, latency, and errors must be traced through MLflow. Tracing captures observable execution rather than private model reasoning or hidden chain-of-thought.

Documentation: Trace agents deployed on Databricks

Required identity model

The authorization model is selected by deployment model. Customer Managed: per-user, on-behalf-of-user authorization is required (OAuth U2M or account-wide token federation); a service principal is never a fallback for interactive requests. Partner Hosted (SaaS): per-tenant service-principal authorization is required because end users have no Databricks identity; never share one service principal across tenants.

Reference architecture​

The pattern extends a standard layered application architecture with an agent and AI orchestration layer running as a Databricks app. The required identity path depends on the deployment model, as shown in the diagram below.

Architecture layers​

Client layer​

The partner browser frontend renders the chat interface, citations, intermediate status, tool confirmations, errors, and streamed responses. The browser calls only the partner backend and does not directly hold Databricks access tokens under this architecture. When an answer includes Genie-generated content, the UI must display the required "Powered by Genie" attribution. In Customer Managed deployments, also surface Genie Space citations and navigable response links where the end user has workspace access.

Documentation: AI Agents and Tools | Integration requirements

Authentication layer​

The partner authenticates the user through its own OAuth or OIDC identity provider and creates the product application session. For every interactive request, the partner backend must obtain a Databricks OAuth token. In Customer Managed deployments this token represents the end user (interactive OAuth U2M or account-wide token federation), and a service principal is not an acceptable fallback. In Partner Hosted deployments the token represents the tenant's mapped service principal (machine-to-machine OAuth). Whichever party owns the Databricks account configures the relevant users, groups, service principals, federation policies, workspace access, and Unity Catalog permissions.

Documentation: Authorize user access to Databricks with OAuth | Authenticate access to Databricks using OAuth token federation | Authenticate with an identity provider token

Agent and AI orchestration layer​

The agent runs as a Databricks app and owns the agent loop, model invocation, tool registration, conversation-state access, long-term memory access, tool confirmation requirements, streaming behavior, MLflow tracing, evaluation integration, retries, cancellation, and error handling.

Documentation: Author an AI agent and deploy it on Databricks Apps

Data and AI services layer​

This layer provides governed access to foundation models through Unity AI Gateway (required), Genie managed MCP servers, SQL warehouses, managed memory, optional Lakebase state, and MLflow tracing and evaluation.

Documentation: Configure AI Gateway on model serving endpoints | Databricks Foundation Model APIs | Databricks managed MCP servers | Agent memory | Trace agents deployed on Databricks

Governed data layer​

Unity Catalog is the governance boundary for the data and securable objects the agent accesses. Unity Catalog evaluates access using the querying identity — the end user (Customer Managed) or the tenant service principal (Partner Hosted) — subject to the app's declared OAuth scopes. In Partner Hosted deployments, tenant isolation is enforced by per-tenant service-principal grants, reinforced by schema or row separation.

Documentation: Configure authorization in a Databricks app

Key components​

Agent runtime​

Build the agent using any agent framework and deploy it as a Databricks app.

A minimal analytics agent requires:

  • A governed structured-data question tool; Genie is preferred.
  • Conversation-state support.
  • Optional long-term memory.
  • Utility tools required by the application.
  • Domain-specific read tools.
  • Carefully controlled write or action tools.

Tools that modify data, trigger workflows, communicate externally, or create financial or operational impact must require explicit confirmation and stronger authorization than read-only tools.

Documentation: Author an AI agent and deploy it on Databricks Apps | Connect agents to tools

Genie as the structured-data query engine​

For natural-language analytics, delegate data questions to Genie through a Databricks managed MCP server rather than prompting the model to generate arbitrary SQL directly. The agent presents Genie as an "ask a data question" tool; Genie interprets the request, generates and executes governed SQL, and returns an answer with deep links to the cited Databricks sources.

Genie One MCP server​

Use the Genie One MCP server when the agent must answer across enterprise data represented through Genie One and its ontology, subject to Unity Catalog permissions and supported resources. It is available at /api/2.0/mcp/genie and uses the genie OAuth scope for on-behalf-of-user authentication.

Per-agent Genie MCP server​

Use the per-agent server when the agent must operate against a curated semantic scope with known tables, metadata, instructions, and example questions. Its endpoint remains /api/2.0/mcp/genie/{'{genie_space_id}'}, even though the current product terminology is Genie Agent.

Genie attribution and output handling​

Partner integrations that surface Genie results must follow the Genie attribution requirements.

This includes:

  • Display a "Powered by Genie" attribution label.
  • Present Genie results as required by the validation framework rather than silently replacing the underlying result.

In Customer Managed deployments — where end users can access the customer's Databricks workspace — you must also:

  • Include a citation linking to the source Genie Space.
  • Preserve links returned by Genie.

Documentation: Genie One MCP server | Databricks managed MCP servers | AI Agents and Tools | Integration requirements

Conversation state and durable memory​

Conversation state and long-term memory are related but distinct:

  • Conversation state contains the running messages, tool calls, and outputs required to continue an interaction.
  • Long-term memory contains distilled information intended for reuse across conversations, such as preferences, stable facts, or prior decisions.

Managed agent memory​

Managed agent memory stores long-term memory in a Unity Catalog memory store. A memory store is a governed Unity Catalog securable; each memory entry belongs to a required scope and path. The trusted scope must be derived from server-side authenticated identity — the end user (Customer Managed) or the tenant (Partner Hosted). Scope is the isolation boundary between users, tenants, or groups and must be configured in trusted code. Databricks explicitly warns that the model must never set it, and the app service principal can read every scope.

from databricks.sdk import WorkspaceClient
from databricks_openai import DatabricksOpenAI

workspace_client = WorkspaceClient()
user_id = str(workspace_client.current_user.me().id)

client = DatabricksOpenAI(
    workspace_client=workspace_client,
    use_ai_gateway=True,
)

conversation = client.conversations.create(
    extra_body={
        "memory_store": {"name": "main.agent_data.customer_memory"},
        "scope": {"kind": "user", "value": user_id},
    },
)

Memory-entry APIs provide long-term memory only. Use server-side conversations, framework sessions, a checkpointer, or self-managed memory for short-term state.

Self-managed memory with Lakebase​

Use Lakebase when the application requires direct SQL access, a custom schema, framework-managed checkpoints, transactional application state, or custom short-term and long-term memory. This option provides more control but requires the application team to manage schema design, isolation, access controls, retention, and lifecycle behavior.

Selecting a memory approach​

A deployment does not necessarily require both managed memory and Lakebase. Use managed memory for governed, scoped, cross-session memory. Use Lakebase when direct SQL access or custom transactional state is required.

Documentation: Agent memory | Managed agent memory | Self-managed agent memory with Lakebase

Model serving and Unity AI Gateway​

It is required to access foundation models through Unity AI Gateway. Route all agent model calls through the gateway — for example, by setting use_ai_gateway=True on the Databricks model client. Foundation models are available through the gateway with OpenAI-compatible access. Keep the model identifier configurable so the model can be upgraded without changing agent business logic.

The gateway centralizes model permissions, per-app or per-user cost attribution, traffic inspection, and model changes without rotating provider credentials or changing agent code. Do not call foundation models outside the gateway on the agent request path.

Documentation: Configure AI Gateway on model serving endpoints | Databricks Foundation Model APIs | Author an AI agent and deploy it on Databricks Apps

Tracing and evaluation​

Bind the application to an MLflow experiment and instrument the agent and custom tools. Production traces must include request and session identifiers, pseudonymous user and tenant context, model calls, tool names and outcomes, latency, token usage, deployment version, evaluation results, and user feedback where appropriate.

Do not log OAuth tokens, credentials, or sensitive request headers. Trace custom tools and remote services explicitly where automatic instrumentation does not cover the full execution path.

Documentation: Trace agents deployed on Databricks | Author an AI agent and deploy it on Databricks Apps

Required identity and authorization model​

The required authorization model is determined by the deployment model, because the deployment model determines whether end users are Databricks principals. Select one model per deployment and document it.

  • Customer Managed (and hybrid deployments where end users federate into the Databricks account that holds the data): end users are Databricks principals, so per-user, on-behalf-of-user authorization is required. A service principal is never a fallback on the interactive request path.
  • Partner Hosted (SaaS): the partner owns the Databricks account and customers have no Databricks identity, so per-tenant service-principal authorization is required. A single service principal shared across tenants is prohibited.

Customer Managed: required end-user authorization​

Where end users are Databricks principals, partners must use Databricks apps user authorization for all interactive customer-facing agent requests. Any request that reads governed data, invokes Genie or another Databricks service in user context, reads or writes user-scoped memory, or performs an action for a user must execute with a Databricks OAuth token representing that user. The app declares specific OAuth scopes, and Databricks forwards the validated user identity and access token to the app through request headers. Scopes restrict what the app can do even when the user has broader permissions outside the app.

resources:
  apps:
    partner_agent:
      name: partner-agent
      source_code_path: ./
      user_api_scopes:
        - sql
        - dashboards.genie
        - serving.serving-endpoints

Retrieve the user-scoped workspace client inside the request handler rather than at application startup, because user credentials are request-specific. Confirm the current scope required by each API and MCP endpoint during deployment.

Documentation: Configure authorization in a Databricks app | Author an AI agent and deploy it on Databricks Apps

Partner Hosted: required per-tenant service-principal authorization​

Where customers consume the product as a service and have no Databricks identity, there is no end-user Databricks principal to represent, and end-user authorization does not apply. Partners must authenticate to Databricks with service principals mapped per tenant. A single service principal shared across tenants is prohibited, because it gives every tenant the same data view.

  • Map each tenant to a dedicated service principal. The partner backend resolves the authenticated application tenant and selects that tenant's service principal; a request for one tenant must never execute with another tenant's service principal.
  • Grant each tenant service principal only that tenant's Unity Catalog objects, warehouse CAN_USE, Genie access, and memory-store scope. Tenant isolation is enforced by per-service-principal Unity Catalog grants, reinforced by schema or row separation in the data model.
  • Authenticate with machine-to-machine OAuth (client credentials). Prefer workload identity federation to eliminate stored secrets; where a client secret is unavoidable, store it in a managed secret store — never in plaintext, source, or the application database.
  • Derive memory scope and audit context from the partner's authenticated application identity (tenant and application user), because no Databricks user identity exists on the request path.
  • Do not share a tenant service principal with unrelated partner workloads.
Service principals and Genie free usage

Databricks Genie pricing includes a monthly free DBU allowance for identified human users. Service principals do not receive this free allowance — Genie usage under these per-tenant service principals is billed from the first request. Free usage applies to Genie LLM usage only; underlying Genie compute (for example, SQL serverless) is billed separately.

Documentation: Configure authorization in a Databricks app | Authenticate access to Databricks using OAuth token federation | Best practices for Databricks Apps

OAuth user-to-machine authorization​

Use OAuth U2M when the user can complete an interactive Databricks authorization flow. Databricks OAuth issues a short-lived access token after the user signs in and grants consent; supported SDKs and tools can manage token generation and refresh.

Documentation: Authorize user access to Databricks with OAuth

Account-wide token federation​

Account-wide token federation lets a trusted customer IdP JWT be exchanged for a Databricks OAuth token representing the user. It is commonly paired with SCIM so identities in the trusted IdP are synchronized into the Databricks account. This approach allows the customer IdP to remain the primary login experience without treating a custom partner session header as Databricks identity.

Documentation: Authenticate access to Databricks using OAuth token federation | Authenticate with an identity provider token

Service-principal use in Customer Managed deployments​

In Customer Managed deployments, service-principal authorization is prohibited for interactive end-user requests and must never be used as a fallback when user authorization fails. It may be used only for isolated machine-only operations that do not read or write user-governed data, invoke tools on behalf of a user, or depend on the user's permissions. Examples include shared telemetry, scheduled maintenance, or system-level configuration. Any exception must be separately isolated, least-privileged, documented, and explicitly approved.

A Customer Managed deployment that cannot support user-based authorization does not meet that path until an approved user-token path is established; where per-user identity is not available at all, use the Partner Hosted per-tenant service-principal model instead. For permitted external machine workloads, prefer workload identity federation over stored client secrets where supported.

Documentation: Configure authorization in a Databricks app | Authenticate access to Databricks using OAuth token federation | Best practices for Databricks Apps

OAuth and authorization security requirements​

  • Request only the minimum OAuth scopes or Unity Catalog privileges required.
  • Never print, log, persist unnecessarily, or return access tokens or service-principal secrets to the browser.
  • Use short-lived tokens and support expiration, revocation, and reauthorization.
  • Use PKCE for interactive OAuth U2M implementations where applicable.
  • Use account-wide token federation for trusted user federation (Customer Managed).
  • Use workload identity federation for machine and service-principal authentication where possible; reserve stored client secrets for cases where federation is unavailable, and keep them in a managed secret store.
  • Record structured audit events identifying the effective actor for every customer-facing Databricks operation — the end user (Customer Managed) or the tenant and application user (Partner Hosted).
  • Customer Managed: fail closed when user authorization cannot be completed; never fall back to the app service principal. Keep permitted machine-only service-principal operations on separate endpoints, jobs, or components from interactive user request paths.
  • Partner Hosted: fail closed when a request cannot be resolved to its tenant service principal; never fall back to a shared or default service principal.

Documentation: Best practices for Databricks Apps | Authorize user access to Databricks with OAuth | Authenticate access to Databricks using OAuth token federation

Request dataflow​

Steps 1–4 are common to both deployment models. Steps 5–7 branch by model. Step 8 onward is common.

Common

  1. The end user authenticates to the partner application through the partner or customer identity provider.
  2. The partner backend establishes and validates the product application session.
  3. The user submits a message from the partner chat interface.
  4. The partner backend determines the authenticated tenant and user.

Customer Managed (steps 5–7)

  1. The backend obtains a Databricks user OAuth token through U2M OAuth or account-wide token federation. This step is required for every interactive request.
  2. The backend calls the Databricks app using the end user's Databricks OAuth bearer token. It never uses a service principal on this request path.
  3. Databricks validates the token and makes the user authorization context available to the app. If validation fails, the request is denied.

Partner Hosted (steps 5–7)

  1. The backend selects the service principal mapped to the authenticated tenant and obtains a Databricks OAuth token through client credentials or workload identity federation.
  2. The backend calls the Databricks app using the tenant service principal's bearer token. A request for one tenant never uses another tenant's service principal.
  3. Databricks validates the token and makes the tenant service principal's authorization context available to the app. If validation or tenant resolution fails, the request is denied.

Common (step 8 onward)

  1. Inside the request handler, the agent obtains the appropriate Databricks workspace client — user-scoped (Customer Managed) or the tenant service-principal client (Partner Hosted).
  2. The agent must invoke the configured foundation model through Unity AI Gateway.
  3. When the request requires structured-data analysis, the agent invokes the appropriate Genie managed MCP tool.
  4. Genie accesses the relevant resources using the authorized identity and permitted scopes and executes SQL against governed Unity Catalog data.
  5. The agent reads or updates conversation state and, where configured, long-term memory using a server-derived scope.
  6. Model calls, tool activity, outputs, errors, and metadata are traced through MLflow.
  7. The app streams the response to the partner backend.
  8. The partner backend streams the response to the browser and renders required Genie attribution; in Customer Managed deployments, also render Genie Space citations and navigable response links where permitted.

Documentation: Configure authorization in a Databricks app | Connect to an API Databricks app using token authentication | Genie One MCP server | Managed agent memory | Trace agents deployed on Databricks

What's Next​

  • AI Agents and Tools — ISV agent requirements, Genie MCP, and validation
  • Governance — Unity Catalog patterns for multi-tenant agent workloads
  • Cost Management — Per-customer model and compute attribution
  • Automation — Deploy agent apps with Databricks Asset Bundles and Terraform