SubmitRun

Submit a Databricks run. Optionally, set waitForCompletion to a desired maximum duration to wait for the run completion.

yaml
type: "io.kestra.plugin.databricks.job.SubmitRun"

Submit a Databricks run and wait up to 5 minutes for its completion.

yaml
id: databricks_job_submit_run
namespace: company.team

tasks:
  - id: submit_run
    type: io.kestra.plugin.databricks.job.SubmitRun
    host: "{{ secret('DATABRICKS_HOST') }}"
    authentication:
      token: "{{ secret('DATABRICKS_TOKEN') }}"
    runTasks:
      - existingClusterId: <your-cluster>
        taskKey: pysparkTask
        sparkPythonTask:
          pythonFile: /Shared/hello.py
          sparkPythonTaskSource: WORKSPACE
    waitForCompletion: PT5M

SubType

Min items 1

The run tasks, if multiple tasks are defined you must set dependsOn on each task.

Databricks account identifier.

Databricks authentication configuration.

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one.

Databricks host.

The name of the run.

Format duration

If set, the task will wait for the run completion.

The run identifier.

Format uri

The run URI on the Databricks console.

SubType string

Map of task base parameters.

Can be a map of string/string or a variable that binds to a JSON object.

Possible Values

GITWORKSPACE

SubType string

Map of task named parameters.

Can be a map of string/string or a variable that binds to a JSON object.

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

SubType string

Task dependencies, set this if multiple tasks are defined on the run.

SubType

Task libraries.

Notebook task settings.

Pipeline task settings.

Python Wheel task settings.

Run job task settings.

Spark JAR task settings.

Spark Python task settings.

Spark Submit task settings.

Possible Values

GITWORKSPACE

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

SubType string

​Submit​Run

SubmitRun