PluginsTasksJob

SubmitRun​Submit​Run

yaml
type: "io.kestra.plugin.databricks.job.SubmitRun"

Submit a Databricks run. Optionally, set waitForCompletion to a desired maximum duration to wait for the run completion.

Examples

Submit a Databricks run and wait up to 5 minutes for its completion.

yaml
id: databricks_job_submit_run
namespace: company.team

tasks:
  - id: submit_run
    type: io.kestra.plugin.databricks.job.SubmitRun
    authentication:
      token: <your-token>
    host: <your-host>
    runTasks:
      - existingClusterId: <your-cluster>
        taskKey: taskKey
        sparkPythonTask:
          pythonFile: /Shared/hello.py
          sparkPythonTaskSource: WORKSPACE
    waitForCompletion: PT5M

Properties

runTasks

The run tasks, if multiple tasks are defined you must set dependsOn on each task.

accountId

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Databricks account identifier.

authentication

Databricks authentication configuration.

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

configFile

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one.

host

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Databricks host.

runName

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The name of the run.

waitForCompletion

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Format: duration

If set, the task will wait for the run completion.

Outputs

runId

  • Type: integer
  • Required: ❌

runURI

  • Type: string
  • Required: ❌
  • Format: uri

Definitions

io.kestra.plugin.databricks.job.task.NotebookTaskSetting

  • baseParameters
    • Type:
      • string
      • object
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • notebookPath
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • source
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Possible Values:
      • GIT
      • WORKSPACE

io.kestra.plugin.databricks.job.task.PythonWheelTaskSetting

  • entryPoint
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • namedParameters
    • Type:
      • string
      • object
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • packageName
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • parameters
    • Type:
      • string
      • array
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.databricks.job.task.LibrarySetting-CranSetting

  • _package
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • repo
    • Type: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.databricks.job.task.SparkSubmitTaskSetting

  • parameters
    • Type:
      • string
      • array
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.databricks.job.SubmitRun-RunSubmitTaskSetting

io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig

  • authType
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • azureClientId
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • azureClientSecret
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • azureTenantId
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • clientId
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • clientSecret
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • googleCredentials
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • googleServiceAccount
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • password
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • token
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • username
    • Type: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.databricks.job.task.RunJobTaskSetting

  • jobId
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • jobParameters
    • Type: object
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.databricks.job.task.SparkPythonTaskSetting

  • pythonFile
    • Type: string
    • Dynamic: ✔️
    • Required: ✔️
  • sparkPythonTaskSource
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Possible Values:
      • GIT
      • WORKSPACE
  • parameters
    • Type:
      • string
      • array
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.databricks.job.task.PipelineTaskSetting

  • fullRefresh
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌
  • pipelineId
    • Type: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.databricks.job.task.LibrarySetting

io.kestra.plugin.databricks.job.task.LibrarySetting-PypiSetting

  • _package
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • repo
    • Type: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.databricks.job.task.SparkJarTaskSetting

  • jarUri
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • mainClassName
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • parameters
    • Type:
      • string
      • array
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.databricks.job.task.LibrarySetting-MavenSetting

  • coordinates
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • exclusions
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • repo
    • Type: string
    • Dynamic: ✔️
    • Required: ❌