Updating secrets using the External Secrets Operator

3 min read | by Jordi Prats

When we are using an ExternalSecret the actual secret is stored elsewhere, being the External Secret Operator in charge of updating the Secret object with the datata if fetches from the SecretStore. What happens when the Secret is updated?

When the Secret is updated, depending on how we are using the secret we might need to restart the deployments using it:

• If we are using the Secret as an environment variable, we'll have to rollout restart the Pods (via the Deployment, ...)
• If we are mounting the Secret as a file, we'll be able to read the updated data

Let's test it using the following ExternalSecret:

apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata:  name: es-aws-demo spec:  refreshInterval: 1m  secretStoreRef:  name: demo-ssm  kind: SecretStore  target:  name: es-aws-demo  data:  - secretKey: demo  remoteRef:  key: demo 

Once applied it's going to fetch the secret from the AWS Parameter Store and create the secret:

$ kubectl get externalsecret NAME STORE REFRESH INTERVAL STATUS READY es-aws-demo demo-ssm 1m SecretSynced True $ kubectl get secret NAME TYPE DATA AGE es-aws-demo Opaque 1 15m 

We can now deploy the following that is going to create a Pod with the Secret mounted as a file and an environment variable:

kind: Deployment apiVersion: apps/v1 metadata:  name: test-eso  labels:  app: test-eso spec:  selector:  matchLabels:  app: test-eso  replicas: 1  strategy:  type: RollingUpdate  template:  metadata:  labels:  app: test-eso  spec:  containers:  - name: test-eso  image: alpine:latest  imagePullPolicy: Always  command: ["sh", "-c", "while true; do echo ===; echo env; echo $ESO_ENV; echo; echo volume; cat /secret/demo; echo; sleep 1; done"]  env:  - name: ESO_ENV  valueFrom:  secretKeyRef:  name: es-aws-demo  key: demo  volumeMounts:  - name: secret  mountPath: /secret  volumes:  - name: secret  secret:  secretName: es-aws-demo 

Once we have the Pod running we can update the secret on the parameter store:

$ awstools ssm set demo newvalue --description "test ESO" --overwrite 6647ac56-494c-431a-a4f7-f3b60c68c4bf 

As soon as the External Secret Operator fetches the data and updates the secret we'll be able to see on the Pod's log that it is able to read the new value from the file (but not from the environment variable)

$ kubectl logs test-eso-559f9fcd4c-b9cgl -f (...) === env value  volume value === env value  volume newvalue 

It won't be able to read the new value even if we create a new process:

$ kubectl exec -it test-eso-559f9fcd4c-b9cgl -- sh / # echo $ESO_ENV value 

To be able to use the new value when it is pushed using an environment variable we'll have to refresh the Pods.

Posted on 11/10/2022