6 min read | by Jordi Prats
To be able to modify a request to the Kubernetes API server prior to persist the object (to, for example, inject a sidecar) we can use a Mutating Webhook. The admission controller makes a requests using all the MutatingWebhookConfiguration objects that matches the request and processes them in serial:
apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration (...) Let's take a look on how to configure a mutating webhook from scratch
First we will have to create an application to make the heavy lifting: It will receive the object that is going to add to the cluster on the /mutate URL:
@app.route('/mutate', methods=HTTP_METHODS) def mutate(): (...) It will send as a response a JSON-patch that will add the label powered-by to the pod:
@app.route('/mutate', methods=HTTP_METHODS) def mutate(): (...) try: request_json = json.loads(request.data.decode('utf-8')) except Exception as e: print(str(e), flush = True) print(str(request_json), flush = True) patch = "[{ \"op\": \"add\", \"path\": \"/metadata/labels/powered-by\", \"value\": \"pet2cattle.com\" }]" (...) response = { "apiVersion": "admission.k8s.io/v1", "kind": "AdmissionReview", "response": { "uid": request_json['request']['uid'], "allowed": True, 'patch': patch_base64, 'patchType': "JSONPatch", } } return json.dumps(response), 200, {'ContentType':'application/json-patch+json'} Then we'll need to decide which URL it is going to be to generate the SSL certificates. If we create the following Kubernetes Service on the webhookdemo namespace:
kind: Service apiVersion: v1 metadata: name: pet2cattle-hook spec: selector: component: pet2cattle-hook ports: - name: http port: 443 targetPort: 8443 The URL the admission controller that will need to call is going to be pet2cattle-hook.webhookdemo.svc. It is going to expect to be a SSL certificate with a SAN record, we can create a new private key and the self-signed certificate as follows:
openssl req -new -sha256 \ -newkey rsa:2048 \ -subj "/C=RC/ST=Barcelona/O=pet2cattle/CN=pet2cattle-hook.webhookdemo.svc" \ -nodes -x509 \ -days 365 \ -out server.crt \ -addext "subjectAltName = DNS:pet2cattle-hook.webhookdemo.svc" To be able to make it the certificate and the private key to the Pod we can use a ConfigMap mounted on /ssl as follows:
apiVersion: v1 kind: ConfigMap metadata: name: ssl-pet2cattle-webhook data: server.crt: | -----BEGIN CERTIFICATE----- MIIDzTCCArWgAwIBAgIUaQt6HUWmFD+MdQl8nWOyx+hlrMEwDQYJKoZIhvcNAQEL BQAwYDELMAkGA1UEBhMCUkMxEjAQBgNVBAgMCUJhcmNlbG9uYTETMBEGA1UECgwK cGV0MmNhdHRsZTEoMCYGA1UEAwwfcGV0MmNhdHRsZS1ob29rLndlYmhvb2tkZW1v LnN2YzAeFw0yMTA4MTEyMDA2MzJaFw0yMjA4MTEyMDA2MzJaMGAxCzAJBgNVBAYT AlJDMRIwEAYDVQQIDAlCYXJjZWxvbmExEzARBgNVBAoMCnBldDJjYXR0bGUxKDAm BgNVBAMMH3BldDJjYXR0bGUtaG9vay53ZWJob29rZGVtby5zdmMwggEiMA0GCSqG SIb3DQEBAQUAA4IBDwAwggEKAoIBAQC1mYz3tQdYciUzdm0kY22lGfhRLFs4y0Et vgm0Icktt2k+pWhuG532lKFIjtmL0/G+lmPZUlRm7S7DUITy2yP2RK1zO5gUce2r IOr4Ag5zWpYlYAUpKzPxQ3Igoi9l9tXa77Lsf0FXMLO47vfUEBDea+ekJZqjvxti fyE1Xc7BLDENpRIv3GweXvVgBEtio1rCQ230vFuxGaHgQU3Qt28dyD8N1P2RbXQ4 M5D74ahBIfbCeyBvOmyv5Bm0BT6YJrEVAY62ZeyKCW6SNM5psbfnoBG4bO62sk0z K+7zWWQm8/4S+k1FgKXt8ZuNhtRoBpATAK8mcp5c8F9w/7x1QG/FAgMBAAGjfzB9 MB0GA1UdDgQWBBTnfBu9wPj1XZeZGZ7tmQ8X7fsU8DAfBgNVHSMEGDAWgBTnfBu9 wPj1XZeZGZ7tmQ8X7fsU8DAPBgNVHRMBAf8EBTADAQH/MCoGA1UdEQQjMCGCH3Bl dDJjYXR0bGUtaG9vay53ZWJob29rZGVtby5zdmMwDQYJKoZIhvcNAQELBQADggEB AEf5FdToi9YURf5bf7NfVAU6l70RbFgfFWv4dzMFCp+jsgMOeL3O57IU80rRxHBI FEkxIfIhiTnR+adDU+eiXm8OjM4bQUIlioMhoDBfuDkFvksWCDqRfR+3iBX7y7Qe UDvGp+NBiBm7+kBuDG4xuP41r+GUuw59LLzNt0+kRQp6HxtzL+mJ1V8beWMrXFrU M4ZwGdmTSb4swP+hC2wGPI5EOOsBkDc2eqXgRR2n63eSTVC8Eo0UWJQw/6RGuzBW q4IyHUh5xoZ7Txea78truVgs413Duhdvp09Vn8GgUgRYXefA0hEM6RRO9sDLIK9f UVIT2Fzb2xETIM3IMOjB6sU= -----END CERTIFICATE----- server.key: | -----BEGIN PRIVATE KEY----- MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC1mYz3tQdYciUz dm0kY22lGfhRLFs4y0Etvgm0Icktt2k+pWhuG532lKFIjtmL0/G+lmPZUlRm7S7D UITy2yP2RK1zO5gUce2rIOr4Ag5zWpYlYAUpKzPxQ3Igoi9l9tXa77Lsf0FXMLO4 7vfUEBDea+ekJZqjvxtifyE1Xc7BLDENpRIv3GweXvVgBEtio1rCQ230vFuxGaHg QU3Qt28dyD8N1P2RbXQ4M5D74ahBIfbCeyBvOmyv5Bm0BT6YJrEVAY62ZeyKCW6S NM5psbfnoBG4bO62sk0zK+7zWWQm8/4S+k1FgKXt8ZuNhtRoBpATAK8mcp5c8F9w /7x1QG/FAgMBAAECggEAQ8ClImnM8serb3bYo4HhD38P8SEOa7MRf0JulmEgkMjk IDZQLvxow+2R+uMo8Q1DHSs414Tq7nfBQaeR4pW15hSbbemnBMG4vWcLozoJMCp0 6D7ZzhFLUNEsDFbWPkGIaiWR6MBVnXUTKIUnu1u/H2y8wLYy6rLLQcVSm3mDQPhd gYNOVPIMHF2Y4axYdlkuAnKjI4LDPJohFeT5qD/hCdP3s6M95CI6g9F+YGLIEwhD 9sj8b9//BEe/A94Dl6usIA2MHlp9e8vqapF/R0CfXSuZQXATri75U5JwkHTzYnpE qgTnBb3ozJEA6L9zquMmrH7zG0ppaKAZnV9iIZSQhQKBgQDoaDgOeNZxmmKIfSHN JWEfd9UA3C64d9imdupx9WnVtc2u1ul2t4S60okyB3b9piaS0fLHNL5B15IZRJxY 27fpDwvYG7azdWJISRbQD7CIoJo3Isixz4Xsq53W7kpMqdltDyp4iTLYYVEzyShV CLEW4Cut71DbDDwEHYZKCAuq1wKBgQDICPRxzbJ9qE8qoTm6IWQ78Zu2f7xAl9sv srDfl5FOIWlUXLTg848jBGZd41la1kex+KsTL6eTqA51CEKChsLbX+Ouq0iL0kpX iqDb0wxkOdojMWuFZ4x0RSLmv9juuYzddpZvORMdBYD9mB6Vti8K/vJzSFl3UE/S cqdiOuFiwwKBgQCQx9QcF+UnslCtzJ5RCXc+vk0gkwo7+tUppq0YvxTmgLKYt+OL BHqYU+4KD6JuE6K2FjqTJOVdaSjnutlXddFVS/1J7MHdfEP02itvBEcqZjqMHIxA URKSRLs4mQwKRElh6m+/1WCqcb2/cBJDHv4LTS2I1qxdOXrt6WKuHeL+0wKBgHFk 2iU1JMykv5P75zyDN03fzZRr3qyDKQZl9mwZgI5Y1Fu1XffzOZ3xHZJ1ka6zr9rM izYKGqXSa7eeIg3aFNXFCs12XV6dq/TqKfvTLMAYJ3cxybDLHUy/8GP8Nx5E4vyb //U21oXqG9AmDphxuUMzeP8u8UB4r3ct9YLyu9d/AoGBAIoAlWlwAsMjm0KTs2kt 7yyaLhxX55cCRnW1adedrqLPUXOZEkF7nHCOh4W6iYWnFvnZdGY5YqlPOVHoCsA3 xcKGxsOvoEZnDyNmRu8Y7i1Ta7iZ8BWq7hw0nNxNfVYGgw0UQaKDQZNx0V/BUP4y 6YmjXNyII6Av3qMqzqpIaNOI -----END PRIVATE KEY----- Then we will need to create a HTTPS enabled container to use these certificates, we can configure gunicorn to use the SSL certificates present on the /ssl directory:
FROM python:3.8-alpine WORKDIR /code # GUNICORN - not an actual dependency RUN pip install gunicorn COPY requirements.txt . RUN pip install -r requirements.txt COPY app /code/app EXPOSE 8443 USER 1001:1001 CMD [ "/usr/local/bin/gunicorn", "app:app", "--certfile=/ssl/server.crt", "--keyfile=/ssl/server.key", "--bind", "0.0.0.0:8443", "--keep-alive", "1" ] To be able to have a pod that mounts the ConfigMap with the SSL certificates we will create a Deployment object as follows:
apiVersion: apps/v1 kind: Deployment metadata: name: pet2cattle-hook spec: replicas: 1 selector: matchLabels: component: pet2cattle-hook template: metadata: labels: component: pet2cattle-hook spec: containers: - name: pet2cattle image: jordiprats/pet2cattle-mutating-webhook:1 ports: - name: http containerPort: 8443 volumeMounts: - name: ssl-pet2cattle-webhook mountPath: /ssl volumes: - name: ssl-pet2cattle-webhook configMap: name: ssl-pet2cattle-webhook Once this is in place we will have to create the actual MutatingWebhookConfiguration that will make Kubernetes make the request to the Deployment to retrieve the JSON-patch. On this object we will have to specify the CA, since we are using a self-signed vertificate we can just use the self-signed certificate itself in base64.
To get the base64 encoded string of the certificate file on a single file we can just disable wrapping using the -w 0 option like follows:
$ base64 -w0 server.crt LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR6VENDQXJXZ0F3SUJBZ0lVYVF0NkhVV21GRCtNZFFsOG5XT3l4K2hsck1Fd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lERUxNQWtHQTFVRUJoTUNVa014RWpBUUJnTlZCQWdNQ1VKaGNtTmxiRzl1WVRFVE1CRUdBMVVFQ2d3SwpjR1YwTW1OaGRIUnNaVEVvTUNZR0ExVUVBd3dmY0dWME1tTmhkSFJzWlMxb2IyOXJMbmRsWW1odmIydGtaVzF2CkxuTjJZekFlRncweU1UQTRNVEV5TURBMk16SmFGdzB5TWpBNE1URXlNREEyTXpKYU1HQXhDekFKQmdOVkJBWVQKQWxKRE1SSXdFQVlEVlFRSURBbENZWEpqWld4dmJtRXhFekFSQmdOVkJBb01DbkJsZERKallYUjBiR1V4S0RBbQpCZ05WQkFNTUgzQmxkREpqWVhSMGJHVXRhRzl2YXk1M1pXSm9iMjlyWkdWdGJ5NXpkbU13Z2dFaU1BMEdDU3FHClNJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUMxbVl6M3RRZFljaVV6ZG0wa1kyMmxHZmhSTEZzNHkwRXQKdmdtMElja3R0MmsrcFdodUc1MzJsS0ZJanRtTDAvRytsbVBaVWxSbTdTN0RVSVR5MnlQMlJLMXpPNWdVY2UycgpJT3I0QWc1eldwWWxZQVVwS3pQeFEzSWdvaTlsOXRYYTc3THNmMEZYTUxPNDd2ZlVFQkRlYStla0pacWp2eHRpCmZ5RTFYYzdCTERFTnBSSXYzR3dlWHZWZ0JFdGlvMXJDUTIzMHZGdXhHYUhnUVUzUXQyOGR5RDhOMVAyUmJYUTQKTTVENzRhaEJJZmJDZXlCdk9teXY1Qm0wQlQ2WUpyRVZBWTYyWmV5S0NXNlNOTTVwc2Jmbm9CRzRiTzYyc2swegpLKzd6V1dRbTgvNFMrazFGZ0tYdDhadU5odFJvQnBBVEFLOG1jcDVjOEY5dy83eDFRRy9GQWdNQkFBR2pmekI5Ck1CMEdBMVVkRGdRV0JCVG5mQnU5d1BqMVhaZVpHWjd0bVE4WDdmc1U4REFmQmdOVkhTTUVHREFXZ0JUbmZCdTkKd1BqMVhaZVpHWjd0bVE4WDdmc1U4REFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQ29HQTFVZEVRUWpNQ0dDSDNCbApkREpqWVhSMGJHVXRhRzl2YXk1M1pXSm9iMjlyWkdWdGJ5NXpkbU13RFFZSktvWklodmNOQVFFTEJRQURnZ0VCCkFFZjVGZFRvaTlZVVJmNWJmN05mVkFVNmw3MFJiRmdmRld2NGR6TUZDcCtqc2dNT2VMM081N0lVODByUnhIQkkKRkVreElmSWhpVG5SK2FkRFUrZWlYbThPak00YlFVSWxpb01ob0RCZnVEa0Z2a3NXQ0RxUmZSKzNpQlg3eTdRZQpVRHZHcCtOQmlCbTcra0J1REc0eHVQNDFyK0dVdXc1OUxMek50MCtrUlFwNkh4dHpMK21KMVY4YmVXTXJYRnJVCk00WndHZG1UU2I0c3dQK2hDMndHUEk1RU9Pc0JrRGMyZXFYZ1JSMm42M2VTVFZDOEVvMFVXSlF3LzZSR3V6QlcKcTRJeUhVaDV4b1o3VHhlYTc4dHJ1VmdzNDEzRHVoZHZwMDlWbjhHZ1VnUllYZWZBMGhFTTZSUk85c0RMSUs5ZgpVVklUMkZ6YjJ4RVRJTTNJTU9qQjZzVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= This output needs to to to as a string on the caBundle key:
apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: pet2cattle-webhook webhooks: - name: pet2cattle-hook.webhookdemo.svc failurePolicy: Fail clientConfig: service: name: pet2cattle-hook namespace: webhookdemo path: "/mutate" caBundle: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR6VENDQXJXZ0F3SUJBZ0lVYVF0NkhVV21GRCtNZFFsOG5XT3l4K2hsck1Fd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lERUxNQWtHQTFVRUJoTUNVa014RWpBUUJnTlZCQWdNQ1VKaGNtTmxiRzl1WVRFVE1CRUdBMVVFQ2d3SwpjR1YwTW1OaGRIUnNaVEVvTUNZR0ExVUVBd3dmY0dWME1tTmhkSFJzWlMxb2IyOXJMbmRsWW1odmIydGtaVzF2CkxuTjJZekFlRncweU1UQTRNVEV5TURBMk16SmFGdzB5TWpBNE1URXlNREEyTXpKYU1HQXhDekFKQmdOVkJBWVQKQWxKRE1SSXdFQVlEVlFRSURBbENZWEpqWld4dmJtRXhFekFSQmdOVkJBb01DbkJsZERKallYUjBiR1V4S0RBbQpCZ05WQkFNTUgzQmxkREpqWVhSMGJHVXRhRzl2YXk1M1pXSm9iMjlyWkdWdGJ5NXpkbU13Z2dFaU1BMEdDU3FHClNJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUMxbVl6M3RRZFljaVV6ZG0wa1kyMmxHZmhSTEZzNHkwRXQKdmdtMElja3R0MmsrcFdodUc1MzJsS0ZJanRtTDAvRytsbVBaVWxSbTdTN0RVSVR5MnlQMlJLMXpPNWdVY2UycgpJT3I0QWc1eldwWWxZQVVwS3pQeFEzSWdvaTlsOXRYYTc3THNmMEZYTUxPNDd2ZlVFQkRlYStla0pacWp2eHRpCmZ5RTFYYzdCTERFTnBSSXYzR3dlWHZWZ0JFdGlvMXJDUTIzMHZGdXhHYUhnUVUzUXQyOGR5RDhOMVAyUmJYUTQKTTVENzRhaEJJZmJDZXlCdk9teXY1Qm0wQlQ2WUpyRVZBWTYyWmV5S0NXNlNOTTVwc2Jmbm9CRzRiTzYyc2swegpLKzd6V1dRbTgvNFMrazFGZ0tYdDhadU5odFJvQnBBVEFLOG1jcDVjOEY5dy83eDFRRy9GQWdNQkFBR2pmekI5Ck1CMEdBMVVkRGdRV0JCVG5mQnU5d1BqMVhaZVpHWjd0bVE4WDdmc1U4REFmQmdOVkhTTUVHREFXZ0JUbmZCdTkKd1BqMVhaZVpHWjd0bVE4WDdmc1U4REFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQ29HQTFVZEVRUWpNQ0dDSDNCbApkREpqWVhSMGJHVXRhRzl2YXk1M1pXSm9iMjlyWkdWdGJ5NXpkbU13RFFZSktvWklodmNOQVFFTEJRQURnZ0VCCkFFZjVGZFRvaTlZVVJmNWJmN05mVkFVNmw3MFJiRmdmRld2NGR6TUZDcCtqc2dNT2VMM081N0lVODByUnhIQkkKRkVreElmSWhpVG5SK2FkRFUrZWlYbThPak00YlFVSWxpb01ob0RCZnVEa0Z2a3NXQ0RxUmZSKzNpQlg3eTdRZQpVRHZHcCtOQmlCbTcra0J1REc0eHVQNDFyK0dVdXc1OUxMek50MCtrUlFwNkh4dHpMK21KMVY4YmVXTXJYRnJVCk00WndHZG1UU2I0c3dQK2hDMndHUEk1RU9Pc0JrRGMyZXFYZ1JSMm42M2VTVFZDOEVvMFVXSlF3LzZSR3V6QlcKcTRJeUhVaDV4b1o3VHhlYTc4dHJ1VmdzNDEzRHVoZHZwMDlWbjhHZ1VnUllYZWZBMGhFTTZSUk85c0RMSUs5ZgpVVklUMkZ6YjJ4RVRJTTNJTU9qQjZzVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=" rules: - operations: [ "CREATE" ] apiGroups: [""] apiVersions: ["v1"] resources: ["pods"] admissionReviewVersions: ["v1", "v1beta1"] sideEffects: None timeoutSeconds: 5 Finally, we are going to create the webhookdemo namespace as defined on the MutatingWebhookConfiguration:
$ kubectl create ns webhookdemo namespace/webhookdemo created $ kubectl config set-context --current --namespace webhookdemo Context "minikube" modified. Finally, I've added a yaml file with all the demo objects on the kubernetes-mutating-webhook repository to make it easier to create:
$ kubectl apply -f yaml/demo-webhook.yaml configmap/ssl-pet2cattle-webhook created deployment.apps/pet2cattle-hook created service/pet2cattle-hook created mutatingwebhookconfiguration.admissionregistration.k8s.io/pet2cattle-webhook created We will have to wait for the Pod to be in Running state:
$ kubectl get pods NAME READY STATUS RESTARTS AGE pet2cattle-hook-8cff54d84-gq487 0/1 ContainerCreating 0 6s $ kubectl get pods NAME READY STATUS RESTARTS AGE pet2cattle-hook-8cff54d84-gq487 1/1 Running 0 12s Once it is ready, any new pod will contain the powered-by label. On the kubernetes-mutating-webhook repository I have also included a test Pod:
$ kubectl apply -f tests/testpod.yaml pod/test-mutator-webhook created If we describe it we will be able to see how the label have been added without being present on the yaml configuration:
$ kubectl describe pod/test-mutator-webhook Name: test-mutator-webhook Namespace: webhookdemo Priority: 0 Node: minikube/192.168.49.2 Start Time: Wed, 11 Aug 2021 22:16:58 +0200 Labels: powered-by=pet2cattle.com test=test-mutator-webhook Annotations: <none> Status: Running IP: 172.17.0.5 IPs: IP: 172.17.0.5 Containers: demo: Container ID: docker://c25b464fca1392d23bda538517bdfffdfcd8a769dcf1049e7b9c037b7befeee9 Image: nginx Image ID: docker-pullable://nginx@sha256:8f335768880da6baf72b70c701002b45f4932acae8d574dedfddaf967fc3ac90 Port: <none> Host Port: <none> State: Running Started: Wed, 11 Aug 2021 22:17:20 +0200 Ready: True Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lf94n (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: kube-api-access-lf94n: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 63s default-scheduler Successfully assigned webhookdemo/test-mutator-webhook to minikube Normal Pulling 59s kubelet Pulling image "nginx" Normal Pulled 45s kubelet Successfully pulled image "nginx" in 14.13825962s Normal Created 41s kubelet Created container demo Normal Started 40s kubelet Started container demo Do we really need to go into all that trouble to modify Kubernetes objects as they are created? Not really, there are generic tools to do so like the KubeMod operator
Posted on 12/08/2021