Trigger multiple Airbyte Cloud syncs in parallel, then run a dbt job

Source

yaml

id: airbyte-cloud-dbt
namespace: company.team

tasks:
  - id: data_ingestion
    type: io.kestra.plugin.core.flow.Parallel
    tasks:
      - id: salesforce
        type: io.kestra.plugin.airbyte.cloud.jobs.Sync
        connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12ab

      - id: google_analytics
        type: io.kestra.plugin.airbyte.cloud.jobs.Sync
        connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12cd

      - id: facebook_ads
        type: io.kestra.plugin.airbyte.cloud.jobs.Sync
        connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12ef

  - id: dbt
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: clone_repository
        type: io.kestra.plugin.git.Clone
        url: https://github.com/kestra-io/dbt-demo
        branch: main

      - id: dbt_build
        type: io.kestra.plugin.dbt.cli.Build
        taskRunner:
          type: io.kestra.plugin.scripts.runner.docker.Docker
        dbtPath: /usr/local/bin/dbt
        dockerOptions:
          image: ghcr.io/kestra-io/dbt-bigquery:latest
        inputFiles:
          .profile/profiles.yml: |
            jaffle_shop:
              outputs:
                dev:
                  type: bigquery
                  dataset: your_big_query_dataset_name
                  project: your_big_query_project
                  fixed_retries: 1
                  keyfile: sa.json
                  location: EU
                  method: service-account
                  priority: interactive
                  threads: 8
                  timeout_seconds: 300
              target: dev
          sa.json: "{{ secret('GCP_CREDS') }}"

pluginDefaults:
  - type: io.kestra.plugin.airbyte.cloud.jobs.Sync
    values:
      token: "{{ secret('AIRBYTE_CLOUD_API_TOKEN') }}"

About this blueprint

Parallel Ingest BigQuery Git dbt

This flow runs Airbyte Cloud syncs in parallel and then runs dbt Core's CLI commands. It's recommended to configure the GCP service account and Airbyte API token, referenced in the pluginDefaults, as a secret.

Parallel

Sync

Working Directory

Clone

Build

Docker

More Related Blueprints

Parallel Ingest BigQuery Git dbt

Trigger multiple Airbyte syncs, then run a dbt job

Parallel Ingest BigQuery Git dbt

Trigger multiple Fivetran syncs in parallel, then run a dbt job

New to Kestra?

Use blueprints to kickstart your first workflows.

Get started with Kestra