Skip to contents

Running Code Remotely

brickster provides mechanisms to run code against Databricks, below is an overview of the available of those in the package:

Method Compatible Compute Notes
db_sql_query SQL Warehouse Simple and efficient function to run SQL
SQL execution API
(db_sql_exec_*)
SQL Warehouse Lower level functions that align 1:1 with API endpoints
Command execution context manager
(db_context_manager)
Clusters
(Shared, Single User)
Higher level R6 class for command execution contexts.
Command execution API
(db_context_*)
Clusters
(Shared, Single User)
Lower level functions that align 1:1 with API endpoints
Databricks REPL
(db_repl)
Clusters
(Shared, Single User)
Supports all notebook languages, R is only supported on single user clusters.

Databricks REPL (db_repl()) will be the focus of this article.

What is the Databricks REPL?

The REPL temporarily connects the existing R console to a Databricks cluster (via command execution APIs) and allows code in all supported languages to be sent interactively - as if it were running locally.

Getting Started

Using the REPL is simple, to start just provide cluster_id:

# start REPL
db_repl(cluster_id = "<insert cluster id>")

The REPL will check the clusters state and start the cluster if inactive. The default language is R.

After successfully connecting to the cluster you can run commands against the remote compute from the local session.

Switching Languages

The REPL has a shortcut you can enter :<language> to change the active language. You can change between the following languages:

Language Shortcut
R :r
Python :py
SQL :sql
Scala :scala
Shell :sh

When you change between languages all variables should persist unless REPL is exited.

Limitations

  • Development environments (e.g. RStudio, Positron) won’t display variables from the remote contexts in the environment pane

  • HTML content will only render for Python, htmlwidgets rendering is restricted due to notebook limitations that require a workaround currently

  • Not designed to work with interactive serverless compute

  • Cannot persist or recover sessions

  • Multi-line expressions are only supported for R. Python, Scala, and SQL are limited to single line expressions.