PluginsDataHubTasks

Ingestion​Ingestion

yaml
type: "io.kestra.plugin.datahub.Ingestion"

DataHub ingestion

Examples

Run DataHub ingestion

yaml
id: datahub_cli
namespace: company.name

tasks:
  - id: cli
    type: io.kestra.plugin.datahub.Ingestion
    recipe:
      source:
        type: mysql
        config:
          host_port: 127.0.0.1:3306
          database: dbname
          username: root
          password: "{{ secret('MYSQL_PASSWORD') }}"
      sink:
        type: datahub-rest
        config:
          server: http://datahub-gms:8080

Run DataHub ingestion using local recipe file

yaml
id: datahub_cli
namespace: company.name

tasks:
  - id: cli
    type: io.kestra.plugin.datahub.Ingestion
    recipe: "{{ input('recipe_file') }}"

Properties

recipe

  • Type: object
  • Dynamic: ❌
  • Required: ✔️

The Ingestion DataHub Recipe.

containerImage

  • Type: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: acryldata/datahub-ingestion:head

The Ingestion DataHub docker image.

env

  • Type: object
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

The environments for Ingestion DataHub.

inputFiles

  • Type:
    • object
    • string
  • Dynamic: ✔️
  • Required: ❌

The files to create on the local filesystem. It can be a map or a JSON object.

namespaceFiles

Inject namespace files.

Inject namespace files to this task. When enabled, it will, by default, load all namespace files into the working directory. However, you can use the include or exclude properties to limit which namespace files will be injected.

outputFiles

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

The files from the local filesystem to send to Kestra's internal storage.

Must be a list of glob) expressions relative to the current working directory, some examples: my-dir/, my-dir/*/ or my-dir/my-file.txt.

taskRunner

  • Type: TaskRunner
  • Dynamic: ❌
  • Required: ❌
  • Default: { "type": "io.kestra.plugin.scripts.runner.docker.Docker" }

The task runner to use.

Outputs

exitCode

  • Type: integer
  • Required: ✔️
  • Default: 0

outputFiles

  • Type: object
  • SubType: string
  • Required: ❌

vars

  • Type: object
  • Required: ❌

Definitions

io.kestra.core.models.tasks.NamespaceFiles

  • enabled
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌
    • Default: true
  • exclude
    • Type: array
    • SubType: string
    • Dynamic: ❌
    • Required: ❌
  • include
    • Type: array
    • SubType: string
    • Dynamic: ❌
    • Required: ❌

io.kestra.core.models.tasks.runners.TaskRunner

  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Validation RegExp: \p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*(\.\p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*)*
    • Min length: 1