Ingestion
DataHub ingestion
type: "io.kestra.plugin.datahub.Ingestion"Run DataHub ingestion
id: datahub_cli
namespace: company.name
tasks:
- id: cli
type: io.kestra.plugin.datahub.Ingestion
recipe:
source:
type: mysql
config:
host_port: 127.0.0.1:3306
database: dbname
username: root
password: "{{ secret('MYSQL_PASSWORD') }}"
sink:
type: datahub-rest
config:
server: http://datahub-gms:8080
Run DataHub ingestion using local recipe file
id: datahub_cli
namespace: company.name
tasks:
- id: cli
type: io.kestra.plugin.datahub.Ingestion
recipe: "{{ input('recipe_file') }}"
The Ingestion DataHub Recipe.
The Ingestion DataHub docker image.
The environments for Ingestion DataHub.
The files to create on the local filesystem. It can be a map or a JSON object.
Inject namespace files.
Inject namespace files to this task. When enabled, it will, by default, load all namespace files into the working directory. However, you can use the include or exclude properties to limit which namespace files will be injected.
The files from the local filesystem to send to Kestra's internal storage.
Must be a list of glob expressions relative to the current working directory, some examples: my-dir/**, my-dir/*/** or my-dir/my-file.txt.
The task runner to use.
The exit code of the entire flow execution.
The output files' URIs in Kestra's internal storage.
The value extracted from the output of the executed commands.
A list of filters to exclude matching glob patterns. This allows you to exclude a subset of the Namespace Files from being downloaded at runtime. You can combine this property together with include to only inject a subset of files that you need into the task's working directory.
A list of filters to include only matching glob patterns. This allows you to only load a subset of the Namespace Files into the working directory.