Kubernetes: Enforcing policies using the OPA gatekeeper
6 min read | by Jordi Prats
We might call it best-practices or policies but most organizations have some rules about how their applications should run, for example: Do not use the latest tag. Some others might even be required to meet certain compliance requirements to reach some security standard, for example: Do not use NodePort services.
To be able to enforce these policies we can use a policy engine like OPA.
OPA gatekeeper is a validating webhook that enforces CRD-based policies executed by the Open Policy Agent.
To implement these policies as CRDs it uses two objects:
- A ConstraintTemplate that defines the rule
- The actual instance of the defined template, allowing you to control (using parameter) when and how it is actually applied the rule
The validating webhook, by default, is ignored when it is returning an error (webhook down or unreachable): During this time constraints will not be enforced, but the audit process is expected to highlight any invalid resources that made it into the cluster.
We can also set it to fail closed by changing the failurePolicy of the ValidatingWebhookConfiguration object to Fail (with the helm chart it is controlled using the validatingWebhookFailurePolicy option) Even though you can always edit or delete a ValidatingWebhookConfiguration since these operations are not subjected to admission webhooks, it can cause circular dependencies where the actions to fix the webhook itself depend on some other action that failed because the webhook is failing: This needs to be carefully considered
OPA gatekeeper installation
To install OPA gatekeeper we can either install the manifests or go for a helm based install, which can't be more straightforward:
helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts helm install gatekeeper/gatekeeper --name-template=gatekeeper --namespace gatekeeper-system --create-namespace Policy configuration
In order to start applying our rules we can start writing them from scratch or look at the gatekeeper-library for rules that somebody have already implemented.
They provide a way of installing them using kustomize as follows:
$ kustomize build github.com/open-policy-agent/gatekeeper-library/library | kubectl apply --filename - But we can choose hand pick which ConstraintTemplate we want to install. For example, to prevent the usage of NodePort we will have to install the following definition:
apiVersion: templates.gatekeeper.sh/v1 kind: ConstraintTemplate metadata: name: k8sblocknodeport annotations: description: >- Disallows all Services with type NodePort. https://kubernetes.io/docs/concepts/services-networking/service/#nodeport spec: crd: spec: names: kind: K8sBlockNodePort targets: - target: admission.k8s.gatekeeper.sh rego: | package k8sblocknodeport violation[{"msg": msg}] { input.review.kind.kind == "Service" input.review.object.spec.type == "NodePort" msg := "User is not allowed to create service of type NodePort" } Once this constraint is installed, we will be able to create instances of this rule by creating the K8sBlockNodePort object as follows:
apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sBlockNodePort metadata: name: block-node-port spec: match: kinds: - apiGroups: [""] kinds: ["Service"] With this in place we won't be able to create the following object:
apiVersion: v1 kind: Service metadata: name: demo-nodeport spec: type: NodePort ports: - port: 80 targetPort: 80 protocol: TCP name: http selector: app: pet2cattle If we try, we will get the following error message:
$ kubectl apply -f nodeportdemo.yaml Error from server ([block-node-port] User is not allowed to create service of type NodePort): error when creating "nodeportdemo.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [block-node-port] User is not allowed to create service of type NodePort We can use these objects to fine-tune what we want to block, for example, we can also apply this restriction to a specific namespace as follows:
apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sBlockNodePort metadata: name: block-node-port spec: match: kinds: - apiGroups: [""] kinds: ["Service"] namespaces: - "test" With this configuration we will be able to create NodePort services on the namespaces that are not included on the list:
$ kubectl apply -f demonodeport.yaml -n test Error from server ([block-node-port] User is not allowed to create service of type NodePort): error when creating "demonodeport.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [block-node-port] User is not allowed to create service of type NodePort $ kubectl apply -f demonodeport.yaml -n trial service/demo-nodeport created Audit existing resources
With the OPA gatekeeper installation, there is a Pod that performs periodic evaluations of existing resources against constraints, detecting pre-existing misconfigurations or any object that somehow managed to get past the validating webhook.
We have tree main ways of checking the audit results:
- Prometheus Metrics: provides an aggregated look at the number of audit violations
- Constraint Status: Violations are listed in the status field of the corresponding constraint
- Audit Logs: At each run, the audit pod emits JSON-formatted logs with information about the violations
For example, if we had an existing NodePort service before applying the K8sBlockNodePort constraint we will be able to see the violations on it's status:
$ kubectl describe K8sBlockNodePort block-node-port Name: block-node-port Namespace: Labels: <none> Annotations: helm.sh/hook: post-install,post-upgrade API Version: constraints.gatekeeper.sh/v1beta1 Kind: K8sBlockNodePort (...) Spec: Match: Kinds: API Groups: Kinds: Service Status: Audit Timestamp: 2022-03-28T21:34:21Z By Pod: Constraint UID: 3640d4af-dead-beef-a123-26bc96100f3e Enforced: true Id: gatekeeper-audit-b8af6ac78c-pf6pr Observed Generation: 3 Operations: audit mutation-status status Constraint UID: 3640d4af-dead-beef-a123-26bc96100f3e Enforced: true Id: gatekeeper-controller-manager-b8af6ac78c-9btw4 Observed Generation: 3 Operations: mutation-webhook webhook Constraint UID: 3640d4af-dead-beef-a123-26bc96100f3e Enforced: true Id: gatekeeper-controller-manager-b8af6ac78c-w9dw7 Observed Generation: 3 Operations: mutation-webhook webhook Constraint UID: 3640d4af-dead-beef-a123-26bc96100f3e Enforced: true Id: gatekeeper-controller-manager-b8af6ac78c-qq847 Observed Generation: 3 Operations: mutation-webhook webhook Total Violations: 1 Violations: Enforcement Action: deny Kind: Service Message: User is not allowed to create service of type NodePort Name: demo-nodeport Namespace: trial Events: <none> And on the audit pod as well:
$ kubectl get pods -n gatekeeper-system NAME READY STATUS RESTARTS AGE gatekeeper-audit-b8af6ac78c-pf6pr 1/1 Running 0 43h gatekeeper-controller-manager-b8af6ac78c-9btw4 1/1 Running 0 43h gatekeeper-controller-manager-b8af6ac78c-w9dw7 1/1 Running 0 43h gatekeeper-controller-manager-b8af6ac78c-qq847 1/1 Running 0 9h $ kubectl logs gatekeeper-audit-b8af6ac78c-pf6pr -n gatekeeper-system (...) {"level":"info","ts":1648496316.0117562,"logger":"controller","msg":"starting update constraints loop","process":"audit","audit_id":"2022-03-28T21:38:36Z","constraints to update":"map[{K8sBlockNodePort constraints.gatekeeper.sh/v1beta1 block-node-port}:{map[apiVersion:constraints.gatekeeper.sh/v1beta1 kind:K8sBlockNodePort metadata:map[annotations:map[helm.sh/hook:post-install,post-upgrade] creationTimestamp:2022-03-23T22:11:42Z generation:3 managedFields:[map[apiVersion:constraints.gatekeeper.sh/v1beta1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[]] manager:gatekeeper operation:Update time:2022-03-23T22:11:42Z] map[apiVersion:constraints.gatekeeper.sh/v1beta1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:helm.sh/hook:map[]]] f:spec:map[.:map[] f:match:map[.:map[] f:kinds:map[]]]] manager:terraform-provider-helm_v2.4.1_x5 operation:Update time:2022-03-23T22:11:42Z]] name:block-node-port resourceVersion:166046384 uid:0364a4df-d8e9-44fe-b131-0e602961f3bc] spec:map[match:map[kinds:[map[apiGroups:[] kinds:[Service]]]]] status:map[auditTimestamp:2022-03-28T13:32:21Z byPod:[map[constraintUID:0364a4df-d8e9-44fe-b131-0e602961f3bc enforced:true id:gatekeeper-audit-6c78cb88bf-9btw4 observedGeneration:3 operations:[audit mutation-status status]] map[constraintUID:0364a4df-d8e9-44fe-b131-0e602961f3bc enforced:true id:gatekeeper-controller-manager-b8af6ac78c-w9dw7 observedGeneration:3 operations:[mutation-webhook webhook]] map[constraintUID:0364a4df-d8e9-44fe-b131-0e602961f3bc enforced:true id:gatekeeper-controller-manager-b8af6ac78c-qq847 observedGeneration:3 operations:[mutation-webhook webhook]] map[constraintUID:0364a4df-d8e9-44fe-b131-0e602961f3bc enforced:true id:gatekeeper-controller-manager-b8af6ac78c-nr6pn observedGeneration:3 operations:[mutation-webhook webhook]]] totalViolations:1 violations:[map[enforcementAction:deny kind:Service message:User is not allowed to create service of type NodePort name:demo-nodeport namespace:trial]]]]}]"} Posted on 29/03/2022