Moving from Development to Production
Common patterns to deploy your flows from development to production environments.
Development Environment
One best practice with Kestra is to have one development instance where users can write their flow directly in UI. This instance can be seen as a "sandbox" where flows can be tested and executed without the fear to break critical business operations.
We usually encourage two types of development environment:
- installing Kestra on your local machine (usually with Docker Compose installation)
- installing Kestra on a Kubernetes cluster accessible by users and separated from production matters.
Production Environment
The production instance should be safeguarded. Especially as this environment supports critical operations and engages your responsibilities for end users.
One common best practice here is to limit the access of the production environment. In this case, there two elements to consider:
- User access
- Flow deployments
User Access
For Kestra Enterprise users, this is streamlined with RBAC and SSO features. With role policies such as "Admin" or "Viewer", one administrator can manage all user access with fine-grain control over all Kestra resources. You can learn more in the dedicated documentation.
For open-source users it's usually a good idea to have a restricted instance, meaning an instance only accessible by CI/CD and administrators.
Flows Deployment
Kestra offers many strategies to deploy flows to an instance:
Choosing one way or the other depends of your preferences and your current deployment patterns.
One recurring pattern is moving flows from the development to the production instance through version control system and CI/CD.
When users have developed flows, they will usually commit changes to a version control system (Git). Then, upon validated pull request, the CI/CD engine will deploy the corresponding flows to the production instance.
The way users can commit flow changes to Git can be addressed with the following patterns:
- Export or copy-paste flows from the user interface
- Using the
git.PushFlowstask
The way CI/CD deploy flows to production instance can be addressed with the following patterns:
- GitHub Action, GitLab CI/CD, Jenkins, Azure DevOps, etc.
- Terraform deployment
- Kestra CLI
You can find more about CI/CD pattern with Kestra here.
Git Example
We can use the git.SyncFlows task combined with a Trigger to automatically pull Flows from the main branch of the Git repository.
This means Kestra will manage the process, reducing the number of systems needed to automate the dev to prod process.
You can use either a Schedule Trigger to pull on a regular routine, e.g. nightly, or the Webhook Trigger to pull when the main branch receives new commits. Check out this dedicated guide on setting them up.
If we combine this with the git.PushFlows task, we can push our Flows to our repository from our local development environment, ensuring they are validated as Kestra will not let you save an invalid flow.
On top of that, we can automatically open a Pull Request with the create.Pulls task to main to speed up the process of getting the Flows to production.
While we can be sure that our flows are valid, this will not check for logical errors. We'd recommend testing flows separately to check this before deploying to production.
CI/CD Example
We can use CI/CD to automatically deploy our flows from our Git repository to our production instance of Kestra when they are merged to the main branch.
With GitHub, we can use the official Deploy Action, which uses the Kestra Server CLI under the hood, to deploy when a Pull Request is merged to main.
We can combine the Deploy Action with the Validate Action, which runs a validate check on Flows using the Kestra Server CLI.
This means we can configure the Git Repository to require status checks to pass before a Pull Request can be merged - preventing any invalid flows from being deployed to production.
Note: If a flow is in an invalid format, the Deploy Action will fail.
Was this page helpful?