​Troubleshooting

Something doesn't work as expected? Check out our advice to help you troubleshoot.

CrashLoopBackoff when restarting all pods

Question from a user: "When I restart all Kubernetes pods at once, they get stuck in a CrashLoopBackoff for a number of minutes before eventually resolving - Why does it happen?"

You are likely hitting the startup issue of a Java application causing failure of liveness probes. The startup can consume a lot of resources, since Java will load many classes at startup. Setting the CPU limit to 0.5 can improve the startup time of the server and should resolve the failing healthchecks.

Unprocessable execution

Sometimes, things can go wrong in an unattended manner; in such situations, you can skip an execution that Kestra is not able to process.

To do so, you can start the executor server (or the standalone server if not using a deployment with separate components) with a list of execution identifiers to skip.

kestra server executor --skip-executions 6FSPERUe1JwbYmMmdwRlgV,5iLGjTLOHAVGUGlsesFaMb

For skipping all executions of a:

1. Flows
   You can skip all executions of one or more flows by using the --skip-flows flag:

kestra server executor --skip-flows flowA,flowB

Example: Suppose you're dealing with a flow called daily-data-sync. If it's causing issues and you want to skip its execution, you can run:

kestra server executor --skip-flows daily-data-sync

2. Namespace
   To skip all executions within specific namespaces, use the --skip-namespaces flag:

kestra server executor --skip-namespaces myNamespaceA,myNamespaceB

Example: If you have a namespace called production-data, and you want to skip all executions under it, use:

kestra server executor --skip-namespaces production-data

3. Tenant
   To skip all executions associated with specific tenants, you can use the --skip-tenants flag:

kestra server executor --skip-tenants tenantA,tenantB

Example: Let's say you have a tenant named companyA, and you want to skip all executions for that tenant:

kestra server executor --skip-tenants companyA

Docker in Docker (DinD)

If you face some issues using Docker in Docker e.g. with Script tasks using DOCKER runner, start troubleshooting by attaching the terminal: docker run -it --privileged docker:dind sh. Then, use docker logs container_ID to get the container logs.

Also, try docker inspect container_ID to get more information about your Docker container. The output from this command displays details about the container, its environments, network settings, etc. This information can help you identify what might be wrong.

Docker in Docker using Helm charts

On some Kubernetes deployments, using DinD with our default Helm charts can lead to:

Device "ip_tables" does not exist.
ip_tables              24576  4 iptable_raw,iptable_mangle,iptable_nat,iptable_filter
modprobe: can't change directory to '/lib/modules': No such file or directory
error: attempting to run rootless dockerd but need 'kernel.unprivileged_userns_clone' (/proc/sys/kernel/unprivileged_userns_clone) set to 1

To fix this, use root to launch the DinD container by setting the following values:

dind:
  image:
    tag: dind
  args:
    - --log-level=fatal
  securityContext:
    runAsUser: 0
    runAsGroup: 0

securityContext:
  runAsUser: 0
  runAsGroup: 0

Docker in Docker (DinD) on a Mac with ARM-based Apple silicon chip

If you are getting an error similar to: java.io.IOException: com.sun.jna.LastErrorException: [111] Connection refused, it might be related to a Docker in Docker (DinD) issue.

Try using an embedded Docker server as shown below:

tmp directory "No such file or directory"

If you're encountering errors associated to the tmp directory such as "No such file or directory", there's a good chance your tmp directory isn't mounted correctly in your Kestra configuration.

When configuring your docker-compose.yml, it's important to make sure that the tmpDir has the same path as the volume otherwise Kestra won't know what directory to mount for the tmp directory.

kestra:
  tasks:
    tmpDir:
      path: /home/kestra/tmp

In this example, /home/kestra:/home/kestra matches the tasks tmpDir field.

volumes:
  - kestra-data:/app/storage
  - /var/run/docker.sock:/var/run/docker.sock
  - /home/kestra:/home/kestra