Blueprints

Clone a GitHub repository and run a Python ETL script

Source

yaml
id: git-python
namespace: company.team

tasks:
  - id: python_scripts
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: clone_repository
        type: io.kestra.plugin.git.Clone
        url: https://github.com/kestra-io/scripts
        branch: main

      - id: python
        type: io.kestra.plugin.scripts.python.Commands
        warningOnStdErr: false
        taskRunner:
          type: io.kestra.plugin.scripts.runner.docker.Docker
        containerImage: ghcr.io/kestra-io/pydata:latest
        commands:
          - python etl/global_power_plant.py
See more

About this blueprint

CLI Python Data Git

This flow clones a git repository and runs a Python ETL script.

  • The Python task can install additional pip packages before running the script using beforeCommands.
  • The image used in this flow is public and you can use it in your workflows.
  • You can add username and password on the git.Clone task but note that they are required only for private repositories. When using public GitHub repositories such as this one, you can only reference the url and branch name.

Working Directory

Clone

Commands

Docker

New to Kestra?

Use blueprints to kickstart your first workflows.

Get started with Kestra