Helm: How to wait for the deployment before assuming it has succeeded

helm kubernetes wait

4 min read | by Jordi Prats

When you upgrade your application using helm what it really does is to just apply the deployments and other objects upgrades. If your new deployment fails to start for some reason (such as missing image) you won't notice it until you actually check the kubernetes cluster.

Helm install and upgrade commands include a couple of options to assist in checking the deployments: --wait and --timeout

  • --wait: It tells helm to wait for the deplyment to succeed before setting as successful
  • -timeout: We can also set a timeout, by default is 300 seconds (5 minutes)

Let's test it out, first we can take a look at the current helm history:

$ helm history pet2cattle REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION  84 Sat Jan 9 12:04:35 2021 superseded pet2cattle-2 3.22 Upgrade complete 85 Sun Jan 10 13:10:58 2021 superseded pet2cattle-2 3.23 Upgrade complete 86 Sun Jan 10 13:14:59 2021 superseded pet2cattle-2 3.24 Upgrade complete 87 Fri Jan 15 21:25:06 2021 superseded pet2cattle-2 3.25 Upgrade complete 88 Sat Jan 30 12:51:53 2021 superseded pet2cattle-2 3.27 Upgrade complete 89 Sun Jan 31 14:18:37 2021 superseded pet2cattle-2 3.28 Upgrade complete 90 Sun Jan 31 17:00:43 2021 superseded pet2cattle-2.1 3.28 Upgrade complete 91 Sun Jan 31 17:05:41 2021 superseded pet2cattle-2.1 3.29 Upgrade complete 92 Sun Jan 31 18:37:20 2021 superseded pet2cattle-2.1 3.31 Upgrade complete 93 Sun Jan 31 18:47:19 2021 deployed pet2cattle-2.1 3.32 Upgrade complete

To test a failing upgrade I have modified the image tag to a non existing so it will fail:

$ helm upgrade pet2cattle -f pet2cattle_values.yaml --wait . Error: UPGRADE FAILED: timed out waiting for the condition

After timing out, on the helm history we will see that the deploy is marked as failed:

$ helm history pet2cattle REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION  85 Sun Jan 10 13:10:58 2021 superseded pet2cattle-2 3.23 Upgrade complete  86 Sun Jan 10 13:14:59 2021 superseded pet2cattle-2 3.24 Upgrade complete  87 Fri Jan 15 21:25:06 2021 superseded pet2cattle-2 3.25 Upgrade complete  88 Sat Jan 30 12:51:53 2021 superseded pet2cattle-2 3.27 Upgrade complete  89 Sun Jan 31 14:18:37 2021 superseded pet2cattle-2 3.28 Upgrade complete  90 Sun Jan 31 17:00:43 2021 superseded pet2cattle-2.1 3.28 Upgrade complete  91 Sun Jan 31 17:05:41 2021 superseded pet2cattle-2.1 3.29 Upgrade complete  92 Sun Jan 31 18:37:20 2021 superseded pet2cattle-2.1 3.31 Upgrade complete  93 Sun Jan 31 18:47:19 2021 deployed pet2cattle-2.1 3.32 Upgrade complete  94 Tue Feb 2 19:43:06 2021 failed pet2cattle-2.1 3.33 Upgrade "pet2cattle" failed: timed out waiting for the condition

Nevertheless, deployments and their pods are going to be still in place:

$ kubectl get pods NAME READY STATUS RESTARTS AGE pet2cattle-s3sync-d8488f8c-9q8j9 1/1 Running 0 2d pet2cattle-845645df85-k2s8k 1/1 Running 0 43h pet2cattle-s3sync-d7948c5f8-zp9w9 0/1 ImagePullBackOff 0 10m pet2cattle-7bf686db6d-qmmt9 0/1 ImagePullBackOff 0 10m

So we will have to rollback the upgrade using helm rollback to a working release:

$ helm rollback pet2cattle 93 Rollback was a success! Happy Helming!

Once we have rolled back the upgrade we won't have the failed pods on the kubernetes cluster:

$ kubectl get pods NAME READY STATUS RESTARTS AGE pet2cattle-s3sync-d8488f8c-9q8j9 1/1 Running 0 2d pet2cattle-845645df85-k2s8k 1/1 Running 0 43h pet2cattle-s3sync-d7948c5f8-zp9w9 0/1 Terminating 0 11m

Back on the helm history we will be able to see that the app version 3.33 failed to deploy and we had to rollbacl to version 3.32:

$ helm history pet2cattle REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION  85 Sun Jan 10 13:10:58 2021 superseded pet2cattle-2 3.23 Upgrade complete  86 Sun Jan 10 13:14:59 2021 superseded pet2cattle-2 3.24 Upgrade complete  87 Fri Jan 15 21:25:06 2021 superseded pet2cattle-2 3.25 Upgrade complete  88 Sat Jan 30 12:51:53 2021 superseded pet2cattle-2 3.27 Upgrade complete  89 Sun Jan 31 14:18:37 2021 superseded pet2cattle-2 3.28 Upgrade complete  90 Sun Jan 31 17:00:43 2021 superseded pet2cattle-2.1 3.28 Upgrade complete  91 Sun Jan 31 17:05:41 2021 superseded pet2cattle-2.1 3.29 Upgrade complete  92 Sun Jan 31 18:37:20 2021 superseded pet2cattle-2.1 3.31 Upgrade complete  93 Sun Jan 31 18:47:19 2021 superseded pet2cattle-2.1 3.32 Upgrade complete  94 Tue Feb 2 19:43:06 2021 failed pet2cattle-2.1 3.33 Upgrade "pet2cattle" failed: timed out waiting for the condition 95 Tue Feb 2 19:54:10 2021 deployed pet2cattle-2.1 3.32 Rollback to 93

Posted on 24/02/2021

Categories