CreateJob
yaml
type: "io.kestra.plugin.databricks.job.CreateJob"Create a Databricks job and run it. Set waitForCompletion to the desired maximum duration if you want the task to wait for the job completion (e.g., PT1H to wait up to one hour).
Examples
Create a Databricks job, run it, and wait for completion for five minutes.
yaml
id: databricks_job_create
namespace: company.team
tasks:
- id: create_job
type: io.kestra.plugin.databricks.job.CreateJob
authentication:
token: <your-token>
host: <your-host>
jobTasks:
- existingClusterId: <your-cluster>
taskKey: taskKey
sparkPythonTask:
pythonFile: /Shared/hello.py
sparkPythonTaskSource: WORKSPACE
waitForCompletion: PT5M
Properties
jobTasks
- Type: array
- SubType: CreateJob-JobTaskSetting
- Dynamic: ❌
- Required: ✔️
- Min items:
1
The job tasks, if multiple tasks are defined you must set dependsOn on each task.
accountId
- Type: string
- Dynamic: ✔️
- Required: ❌
Databricks account identifier.
authentication
- Type: AbstractTask-AuthenticationConfig
- Dynamic: ❌
- Required: ❌
Databricks authentication configuration.
This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.
configFile
- Type: string
- Dynamic: ✔️
- Required: ❌
Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one.
host
- Type: string
- Dynamic: ✔️
- Required: ❌
Databricks host.
jobName
- Type: string
- Dynamic: ✔️
- Required: ❌
The name of the job.
waitForCompletion
- Type: string
- Dynamic: ❌
- Required: ❌
- Format:
duration
If set, the task will wait for the job run completion for up to the waitForCompletion duration before timing out.
Outputs
jobId
- Type: integer
- Required: ❌
jobURI
- Type: string
- Required: ❌
- Format:
uri
runId
- Type: integer
- Required: ❌
runURI
- Type: string
- Required: ❌
- Format:
uri
Definitions
io.kestra.plugin.databricks.job.task.SqlTaskSetting
parameters- Type:
- string
- object
- SubType: string
- Dynamic: ✔️
- Required: ❌
- Type:
queryId- Type: string
- Dynamic: ✔️
- Required: ❌
warehouseId- Type: string
- Dynamic: ✔️
- Required: ❌
io.kestra.plugin.databricks.job.task.NotebookTaskSetting
baseParameters- Type:
- string
- object
- SubType: string
- Dynamic: ✔️
- Required: ❌
- Type:
notebookPath- Type: string
- Dynamic: ✔️
- Required: ❌
source- Type: string
- Dynamic: ❌
- Required: ❌
- Possible Values:
GITWORKSPACE
io.kestra.plugin.databricks.job.CreateJob-JobTaskSetting
dbtTask- Type: DbtTaskSetting
- Dynamic: ❌
- Required: ❌
dependsOn- Type: array
- SubType: string
- Dynamic: ❌
- Required: ❌
description- Type: string
- Dynamic: ✔️
- Required: ❌
existingClusterId- Type: string
- Dynamic: ✔️
- Required: ❌
libraries- Type: array
- SubType: LibrarySetting
- Dynamic: ❌
- Required: ❌
notebookTask- Type: NotebookTaskSetting
- Dynamic: ❌
- Required: ❌
pipelineTask- Type: PipelineTaskSetting
- Dynamic: ❌
- Required: ❌
pythonWheelTask- Type: PythonWheelTaskSetting
- Dynamic: ❌
- Required: ❌
runJobTask- Type: RunJobTaskSetting
- Dynamic: ❌
- Required: ❌
sparkJarTask- Type: SparkJarTaskSetting
- Dynamic: ❌
- Required: ❌
sparkPythonTask- Type: SparkPythonTaskSetting
- Dynamic: ❌
- Required: ❌
sparkSubmitTask- Type: SparkSubmitTaskSetting
- Dynamic: ❌
- Required: ❌
sqlTask- Type: SqlTaskSetting
- Dynamic: ❌
- Required: ❌
taskKey- Type: string
- Dynamic: ✔️
- Required: ❌
timeoutSeconds- Type: integer
- Dynamic: ❌
- Required: ❌
io.kestra.plugin.databricks.job.task.PythonWheelTaskSetting
entryPoint- Type: string
- Dynamic: ✔️
- Required: ❌
namedParameters- Type:
- string
- object
- SubType: string
- Dynamic: ✔️
- Required: ❌
- Type:
packageName- Type: string
- Dynamic: ✔️
- Required: ❌
parameters- Type:
- string
- array
- Dynamic: ✔️
- Required: ❌
- Type:
io.kestra.plugin.databricks.job.task.LibrarySetting-CranSetting
_package- Type: string
- Dynamic: ✔️
- Required: ❌
repo- Type: string
- Dynamic: ✔️
- Required: ❌
io.kestra.plugin.databricks.job.task.SparkSubmitTaskSetting
parameters- Type:
- string
- array
- Dynamic: ✔️
- Required: ❌
- Type:
io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig
authType- Type: string
- Dynamic: ✔️
- Required: ❌
azureClientId- Type: string
- Dynamic: ✔️
- Required: ❌
azureClientSecret- Type: string
- Dynamic: ✔️
- Required: ❌
azureTenantId- Type: string
- Dynamic: ✔️
- Required: ❌
clientId- Type: string
- Dynamic: ✔️
- Required: ❌
clientSecret- Type: string
- Dynamic: ✔️
- Required: ❌
googleCredentials- Type: string
- Dynamic: ✔️
- Required: ❌
googleServiceAccount- Type: string
- Dynamic: ✔️
- Required: ❌
password- Type: string
- Dynamic: ✔️
- Required: ❌
token- Type: string
- Dynamic: ✔️
- Required: ❌
username- Type: string
- Dynamic: ✔️
- Required: ❌
io.kestra.plugin.databricks.job.task.RunJobTaskSetting
jobId- Type: string
- Dynamic: ✔️
- Required: ❌
jobParameters- Type: object
- Dynamic: ✔️
- Required: ❌
io.kestra.plugin.databricks.job.task.SparkPythonTaskSetting
pythonFile- Type: string
- Dynamic: ✔️
- Required: ✔️
sparkPythonTaskSource- Type: string
- Dynamic: ❌
- Required: ✔️
- Possible Values:
GITWORKSPACE
parameters- Type:
- string
- array
- Dynamic: ✔️
- Required: ❌
- Type:
io.kestra.plugin.databricks.job.task.PipelineTaskSetting
fullRefresh- Type: boolean
- Dynamic: ❌
- Required: ❌
pipelineId- Type: string
- Dynamic: ✔️
- Required: ❌
io.kestra.plugin.databricks.job.task.LibrarySetting
cran- Type: LibrarySetting-CranSetting
- Dynamic: ❌
- Required: ❌
egg- Type: string
- Dynamic: ✔️
- Required: ❌
jar- Type: string
- Dynamic: ✔️
- Required: ❌
maven- Type: LibrarySetting-MavenSetting
- Dynamic: ❌
- Required: ❌
pypi- Type: LibrarySetting-PypiSetting
- Dynamic: ❌
- Required: ❌
whl- Type: string
- Dynamic: ✔️
- Required: ❌
io.kestra.plugin.databricks.job.task.DbtTaskSetting
catalog- Type: string
- Dynamic: ✔️
- Required: ❌
commands- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
schema- Type: string
- Dynamic: ✔️
- Required: ❌
warehouseId- Type: string
- Dynamic: ✔️
- Required: ❌
io.kestra.plugin.databricks.job.task.LibrarySetting-PypiSetting
_package- Type: string
- Dynamic: ✔️
- Required: ❌
repo- Type: string
- Dynamic: ✔️
- Required: ❌
io.kestra.plugin.databricks.job.task.SparkJarTaskSetting
jarUri- Type: string
- Dynamic: ✔️
- Required: ❌
mainClassName- Type: string
- Dynamic: ✔️
- Required: ❌
parameters- Type:
- string
- array
- Dynamic: ✔️
- Required: ❌
- Type:
io.kestra.plugin.databricks.job.task.LibrarySetting-MavenSetting
coordinates- Type: string
- Dynamic: ✔️
- Required: ❌
exclusions- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
repo- Type: string
- Dynamic: ✔️
- Required: ❌