Kubernetes multi-tenant with Capsule

Kubernetes Capsule multi-tenant

6 min read | by Jordi Prats

Capsule implements a multi-tenant and policy-based environment in your Kubernetes cluster, leveraging only on upstream Kubernetes. It allows you to create tenants, namespaces, and users, and define policies to control the resources and access within the cluster.

Installing Capsule

To install Capsule, you can use the Helm chart provided by the project:

helm repo add projectcapsule https://projectcapsule.github.io/charts helm install capsule projectcapsule/capsule -n capsule-system --create-namespace

Once installed you will only see one Deployment running in the capsule-system namespace:

$ kubectl get pods -n capsule-system NAME READY STATUS RESTARTS AGE capsule-controller-manager-685ddcc48b-7859r 1/1 Running 0 5s

Besides the controller manager, Capsule will a handful of Custom Resource Definitions (CRDs)`:

$ kubectl get crd NAME CREATED AT capsuleconfigurations.capsule.clastix.io 2025-02-22T13:37:38Z globalproxysettings.capsule.clastix.io 2025-02-22T14:24:40Z globaltenantresources.capsule.clastix.io 2025-02-22T13:37:38Z proxysettings.capsule.clastix.io 2025-02-22T14:24:40Z tenantresources.capsule.clastix.io 2025-02-22T13:37:38Z tenants.capsule.clastix.io 2025-02-22T13:37:39Z

Authentication

In a multi-tenant environment, authentication is crucial. Capsule does not care about the authentication strategy used in the cluster and all the Kubernetes methods of authentication are supported. The only requirement to use Capsule is to assign tenant users to the group defined by --capsule-user-group option, which defaults to capsule.clastix.io.

Assignment to a group depends on the authentication strategy in your cluster. We can use OIDC or AWS IAM for example.

Create tenant

To create a tenant, we'll have to use the Tenant object. Here is an example of a Tenant object that defines a tenant named demo with one owner named jordi:

apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata:  name: demo spec:  owners:  - name: jordi  kind: User

We can create the tenant using the kubectl apply just as any other Kubernetes object:

$ kubectl apply -f demo-tenant.yaml tenant.capsule.clastix.io/demo created $ kubectl get tenant NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE demo Active 0 10s

In the .spec section of the Tenant object, we can define the rules and limits for the tenant. We can control who owns the tenant, how many namespaces it can have, and what extra labels or settings they should get. We can also limit what container images are allowed, which storage types can be used, and how workloads are prioritized. This helps keep multi-tenant Kubernetes clusters organized, secure, and efficient. On the Tenant object documentation you can find all the available options.

Login as a tenant

Each tenant comes with a delegated user or group of users acting as the tenant admin (the tenant Owner). For this example, we'll use the hack/create-user.sh script provided by the Capsule project to create a kubeconfig for the user jordi within the demo tenant:

$ git clone https://github.com/projectcapsule/capsule.git $ file capsule/hack/create-user.sh capsule/hack/create-user.sh: Bourne-Again shell script text executable, ASCII text $ bash capsule/hack/create-user.sh jordi demo creating certs in TMPDIR /var/folders/hz/r62v__kj29jf__h_zmbbv0v40000gn/T/tmp.SkXrJvwxOZ merging groups /O=projectcapsule.dev certificatesigningrequest.certificates.k8s.io "jordi-demo" deleted certificatesigningrequest.certificates.k8s.io/jordi-demo created certificatesigningrequest.certificates.k8s.io/jordi-demo approved kubeconfig file is: jordi-demo.kubeconfig to use it as jordi export KUBECONFIG=jordi-demo.kubeconfig

If we using the generated kubeconfig file, we'll be able to create namespaces and deploy workloads within the demo tenant:

$ KUBECONFIG=jordi-demo.kubeconfig kubectl create ns jordi-1st-ns namespace/jordi-1st-ns created

Unlike other multi-tenant solutions, we are going to be able to create namespaces as usual. The only difference is that the namespace will be owned by the tenant:

$ kubectl get ns jordi-1st-ns -o yaml apiVersion: v1 kind: Namespace metadata:  creationTimestamp: "2025-02-22T14:20:05Z"  labels:  capsule.clastix.io/tenant: demo  kubernetes.io/metadata.name: jordi-1st-ns  name: jordi-1st-ns  ownerReferences:  - apiVersion: capsule.clastix.io/v1beta2  kind: Tenant  name: demo  uid: 90a1f6a1-2284-44dd-8f58-2b9f75b8616f  resourceVersion: "5268"  uid: 3118715b-02fe-4dc4-91f8-19ebdde7dbb7 spec:  finalizers:  - kubernetes status:  phase: Active

Going back to the Tenant object, we can see that the NAMESPACE COUNT has been updated to 1:

$ kubectl get tenant NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE demo Active 1 10m

Creating workfloads is going to be as usual as well:

$ KUBECONFIG=jordi-demo.kubeconfig kubectl run demo --image nginx -n jordi-1st-ns pod/demo created $ KUBECONFIG=jordi-demo.kubeconfig kubectl get pods -n jordi-1st-ns NAME READY STATUS RESTARTS AGE demo 0/1 ContainerCreating 0 3s

Listing namespaces

One of the problems we'll have as a tenant user is that we won't be able to list the cluster-scoped resources, such as namespaces:

$ KUBECONFIG=jordi-demo.kubeconfig kubectl get ns Error from server (Forbidden): namespaces is forbidden: User "jordi" cannot list resource "namespaces" in API group "" at the cluster scope

To overcome this issue, we can use the Capsule Proxy add-on. We can install it using the Helm chart provided by the capsule project:

helm install capsule-proxy projectcapsule/capsule-proxy -n capsule-system

Once installed, we'll need to use that service to access the cluster-scoped resources:

$ kubectl get service capsule-proxy -n capsule-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE capsule-proxy ClusterIP 10.96.179.132 <none> 9001/TCP 46s

On a production environment we would need to configure and expose the service properly, but for testing purposes we can use port-forward and ignore the certificate validation:

kubectl -n capsule-system port-forward service/capsule-proxy 9001:9001

Once the proxy is running, we can update the kubeconfig file. We need to:

  • Update the server to use localhost:9001
  • Remove certificate-authority-data from the cluster entry
  • Add insecure-skip-tls-verify: true to skip the certificate validation

The resulting kubeconfig file should look like this:

apiVersion: v1 clusters: - cluster:  server: https://127.0.0.1:9001  insecure-skip-tls-verify: true  name: kind-capsule contexts: - context:  cluster: kind-capsule  user: jordi  name: jordi-demo current-context: jordi-demo kind: Config preferences: {} users: - name: jordi  user:  client-certificate: jordi-demo.crt  client-key: jordi-demo.key

Once updated, we can use it to list the namespaces, in this case we'll be able to see the list of namespaces owned by the demo tenant:

$ KUBECONFIG=jordi-demo.kubeconfig kubectl get ns NAME STATUS AGE jordi-1st-ns Active 28m

Posted on 25/02/2025

Categories