Kubernetes: Managing tenants with Capsule

7 min read | by Jordi Prats

Once we have Capsule setup we'll need to start managing the tenants and their permissions. In this post, we'll see how to assign permissions to a user, cordoning a tenant, and enforcing resource quotas at the tenant level.

Assign tenant-admin permissions to a user
Assing view-only permissions to a user
Cordoning a tenant
Adding tenant-admin permissions to a service account
Enforce resource quotas at the tenant level

Assign tenant-admin permissions to a user

By default, when we add a user to the owners list, they are going to be able to act as the admin of all the tenant namespaces:

apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata:  name: demo spec:  owners:  - name: jordi  kind: User

Writing the the above manifest is equivalent to setting clusterRoles to admin and capsule-namespace-deleter:

apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata:  name: demo spec:  owners:  - name: jordi  kind: User  clusterRoles:  - admin  - capsule-namespace-deleter

The capsule controller is going to translate this definition into the following RoleBindings:

$ kubectl get rolebinding NAME ROLE AGE capsule-demo-0-admin ClusterRole/admin 2d20h capsule-demo-1-capsule-namespace-deleter ClusterRole/capsule-namespace-deleter 2d20h

Assing view-only permissions to a user

We can also create a user and scope it to view-only permissions:

apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata:  name: demo spec:  owners:  - name: jordi  kind: User  - name: view-only  kind: User  clusterRoles:  - view

In this case, the capsule controller is going to create a RoleBinding with the view ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata:  creationTimestamp: "2025-02-23T10:27:02Z"  labels:  capsule.clastix.io/role-binding: eaaa0637389ad533  capsule.clastix.io/tenant: demo  name: capsule-demo-2-view  namespace: jordi-1st-ns  ownerReferences:  - apiVersion: capsule.clastix.io/v1beta2  blockOwnerDeletion: true  controller: true  kind: Tenant  name: demo  uid: 90a1f6a1-2284-44dd-8f58-2b9f75b8616f  resourceVersion: "37840"  uid: 6eaec6f6-7718-44ba-bca5-310479d368ce roleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: view subjects: - apiGroup: rbac.authorization.k8s.io  kind: User  name: view-only

By automatically creating a RoleBinding to all the namespaces owned by the tenant, the user is going to be able to list resources in all the tenant namespaces but not in any other namespaces:

$ KUBECONFIG=view-only-demo.kubeconfig kubectl auth can-i get pod -n jordi-1st-ns yes $ KUBECONFIG=view-only-demo.kubeconfig kubectl auth can-i get pod -n kube-system no

We can also define RoleBindings using the tenant.spec.additionalRoleBindings field, it is intended to be used for much more generic use cases:

apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata:  name: demo spec:  owners:  - name: jordi  kind: User  additionalRoleBindings:  - clusterRoleName: 'view'  subjects:  - apiGroup: rbac.authorization.k8s.io  kind: User  name: additional-viewer

In this case, we are going to achieve the same result as before. Using the additionalRoleBindings field is a way to make sure we are not assigning unintended permissions to the user since the default list of clusterRoles include admin permissions to the tenant.

$ kubectl get rolebinding NAME ROLE AGE capsule-demo-0-admin ClusterRole/admin 2d20h capsule-demo-1-capsule-namespace-deleter ClusterRole/capsule-namespace-deleter 2d20h capsule-demo-2-view ClusterRole/view 16m capsule-demo-3-view ClusterRole/view 4s $ kubectl get rolebinding capsule-demo-3-view -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata:  creationTimestamp: "2025-02-23T10:43:47Z"  labels:  capsule.clastix.io/role-binding: 3bfe3bdd56040ecf  capsule.clastix.io/tenant: demo  name: capsule-demo-3-view  namespace: jordi-1st-ns  ownerReferences:  - apiVersion: capsule.clastix.io/v1beta2  blockOwnerDeletion: true  controller: true  kind: Tenant  name: demo  uid: 90a1f6a1-2284-44dd-8f58-2b9f75b8616f  resourceVersion: "39661"  uid: 6982812b-7cd1-4ea8-808b-7baf5a7e5403 roleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: view subjects: - apiGroup: rbac.authorization.k8s.io  kind: User  name: additional-viewer

Cordoning a tenant

In the same way we can cordon a node, we can cordon a tenant to prevent any workload to be updated. We might want to do this, for example, during cluster upgrades or production freezes:

apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata:  name: demo spec:  cordoned: true  owners:  - name: jordi  kind: User

As soon as the Tenant is cordoned, the admission controller is going to prevent any update to the tenant workloads:

$ kubectl get tenant NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE demo Cordoned 1 2d20h $ KUBECONFIG=jordi-demo.kubeconfig kubectl auth can-i update pod -n jordi-1st-ns yes $ KUBECONFIG=jordi-demo.kubeconfig kubectl edit pod demo Error from server (Forbidden): pods "demo" is forbidden: User "jordi" cannot get resource "pods" in API group "" in the namespace "default" $ KUBECONFIG=jordi-demo.kubeconfig kubectl create ns jordi-2nd-ns Error from server (Forbidden): admission webhook "namespaces.projectcapsule.dev" denied the request: the selected Tenant is freezed

Adding tenant-admin permissions to a service account

We can also have a service account acting as a tenant owner as follows:

apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata:  name: demo spec:  owners:  - name: jordi  kind: User  - name: system:serviceaccount:argocd:default  kind: ServiceAccount

The service account needs to be part of the Capsule group, so we'll have to update the CapsuleConfiguration object to include the service account group to the list of user groups:

apiVersion: capsule.clastix.io/v1beta2 kind: CapsuleConfiguration metadata:  annotations:  meta.helm.sh/release-name: capsule  meta.helm.sh/release-namespace: capsule-system  creationTimestamp: "2025-02-22T13:37:53Z"  generation: 1  labels:  app.kubernetes.io/instance: capsule  app.kubernetes.io/managed-by: Helm  app.kubernetes.io/name: capsule  app.kubernetes.io/version: 0.7.0  helm.sh/chart: capsule-0.7.0  name: default  resourceVersion: "554"  uid: e16fdd92-0aa1-4149-b95d-51ba3f08272d spec:  enableTLSReconciler: true  forceTenantPrefix: false  nodeMetadata:  forbiddenAnnotations:  denied: []  deniedRegex: ""  forbiddenLabels:  denied: []  deniedRegex: ""  overrides:  TLSSecretName: capsule-tls  mutatingWebhookConfigurationName: capsule-mutating-webhook-configuration  validatingWebhookConfigurationName: capsule-validating-webhook-configuration  protectedNamespaceRegex: ""  userGroups:  - projectcapsule.dev  - system:serviceaccount:argocd

Bear in mind that all the service accounts in the same namespace are included in this group.

Enforce resource quotas at the tenant level

One of the features of Capsule is the ability to enforce resource quotas at the tenant level instead of doing so just at the namespace level. We can achieve this by using the resourceQuotas field in the Tenant object:

apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata:  name: demo spec:  owners:  - name: jordi  kind: User  resourceQuotas:  scope: Tenant  items:  - hard:  pods: "2"

We can test this by maxing out the number of pods in the tenant in one namespace and trying to create a new pod on a different namespace belonging to the same tenant:

$ kubectl get ns -l capsule.clastix.io/tenant=demo NAME STATUS AGE jordi-1st-ns Active 2d21h jordi-2nd-ns Active 27s $ KUBECONFIG=jordi-demo.kubeconfig kubectl run demo-2 --image nginx -n jordi-1st-ns pod/demo-2 created $ KUBECONFIG=jordi-demo.kubeconfig kubectl run demo-3 --image nginx -n jordi-1st-ns Error from server (Forbidden): pods "demo-3" is forbidden: exceeded quota: capsule-demo-0, requested: pods=1, used: pods=2, limited: pods=2 $ KUBECONFIG=jordi-demo.kubeconfig kubectl run demo-3 --image nginx -n jordi-2ns-ns Error from server (Forbidden): pods is forbidden: User "jordi" cannot create resource "pods" in API group "" in the namespace "jordi-2ns-ns"

Capsule is achieving this by dynamically updating the ResourceQuota object in all the tenant namespaces, while maintaining a global count of the resources used by the tenant as an annotation:

$ kubectl get resourcequota capsule-demo-0 -n jordi-1st-ns -o yaml apiVersion: v1 kind: ResourceQuota metadata:  annotations:  quota.capsule.clastix.io/hard-pods: "2"  quota.capsule.clastix.io/used-pods: "2"  creationTimestamp: "2025-02-23T11:25:41Z"  labels:  capsule.clastix.io/resource-quota: "0"  capsule.clastix.io/tenant: demo  name: capsule-demo-0  namespace: jordi-1st-ns  ownerReferences:  - apiVersion: capsule.clastix.io/v1beta2  blockOwnerDeletion: true  controller: true  kind: Tenant  name: demo  uid: 90a1f6a1-2284-44dd-8f58-2b9f75b8616f  resourceVersion: "45001"  uid: 4b062e21-0643-4c72-8979-500bd03d374d spec:  hard:  pods: "2" status:  hard:  pods: "2"  used:  pods: "2" $ kubectl get resourcequota capsule-demo-0 -n jordi-2nd-ns -o yaml apiVersion: v1 kind: ResourceQuota metadata:  annotations:  quota.capsule.clastix.io/hard-pods: "2"  quota.capsule.clastix.io/used-pods: "2"  creationTimestamp: "2025-02-23T11:28:23Z"  labels:  capsule.clastix.io/resource-quota: "0"  capsule.clastix.io/tenant: demo  name: capsule-demo-0  namespace: jordi-2nd-ns  ownerReferences:  - apiVersion: capsule.clastix.io/v1beta2  blockOwnerDeletion: true  controller: true  kind: Tenant  name: demo  uid: 90a1f6a1-2284-44dd-8f58-2b9f75b8616f  resourceVersion: "45002"  uid: d865a8d4-8be0-44f6-b4a6-3757a562c83f spec:  hard:  pods: "0" status:  hard:  pods: "0"  used:  pods: "0"