Spark Jar Task

Spark Jar Task

Usage

spark_jar_task(main_class_name, parameters = list())

Arguments

main_class_name

The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library. The code must use SparkContext.getOrCreate to obtain a Spark context; otherwise, runs of the job fail.

parameters

Named list. Parameters passed to the main method. Use Task parameter variables to set parameters containing information about job runs.

See also

Other Task Objects: condition_task(), email_notifications(), for_each_task(), libraries(), new_cluster(), notebook_task(), pipeline_task(), python_wheel_task(), run_job_task(), spark_python_task(), spark_submit_task(), sql_file_task(), sql_query_task()