Spark Python Task

Spark Python Task

Usage

spark_python_task(python_file, parameters = list())

Arguments

python_file

The URI of the Python file to be executed. DBFS and S3 paths are supported.

parameters

List. Command line parameters passed to the Python file. Use Task parameter variables to set parameters containing information about job runs.

See also

Other Task Objects: condition_task(), email_notifications(), for_each_task(), libraries(), new_cluster(), notebook_task(), pipeline_task(), python_wheel_task(), run_job_task(), spark_jar_task(), spark_submit_task(), sql_file_task(), sql_query_task()