How to Enable GitOps for Multi-Tenant Kubernetes Clusters with Capsule and Flux

Introduction

Managing multiple tenants in a Kubernetes cluster presents challenges around consistent deployment, resource isolation, and policy enforcement. In a multi-tenant environment, you need a solution that ensures each tenant can manage their resources independently while adhering to cluster-wide policies. This is where Capsule and Flux work together to provide a scalable and automated solution for Multi-Tenant Clusters.

• Capsule allows you to create and manage multi-tenant Kubernetes clusters, ensuring strict isolation and resource quotas at the tenant level.
• Flux enables GitOps by automatically syncing Kubernetes resources from a Git repository, ensuring that cluster state is consistent with the desired state defined in version control.

In this blog, we will explore how to enable GitOps for multi-tenant Kubernetes clusters using Capsule and Flux. We’ll walk through a practical example of setting up Capsule to manage tenants and configuring Flux to automate tenant deployments using a GitOps model.

---

Why Capsule and Flux Work Well Together

Capsule for Multi-Tenancy:

1. Logical separation of tenants within the same Kubernetes cluster.
2. Tenant-level resource quotas, network policies, and RBAC enforcement.
3. Secure multi-tenancy by isolating namespaces and workloads.

Flux for GitOps:

1. Declarative configuration using Kubernetes manifests stored in a Git repository.
2. Continuous reconciliation to keep the cluster state consistent with the desired state.
3. Rollback and version control for Kubernetes deployments.

When combined, Capsule and Flux enable you to automate and scale multi-tenant cluster management using Git as the single source of truth.

---

Step 1: Install Capsule and Flux

Install Capsule:

You can install Capsule using Helm:

helm repo add clastix https://clastix.github.io/charts
helm repo update
helm install capsule clastix/capsule

Verify that Capsule is running:

kubectl get pods -n capsule-system

Install Flux:

Install Flux using the Flux CLI:

1. Install the Flux CLI:

curl -s https://fluxcd.io/install.sh | sudo bash

2. Bootstrap Flux in your cluster using a Git repository:

flux bootstrap github \
  --owner=my-github-username \
  --repository=my-gitops-repo \
  --branch=main \
  --path=clusters/my-cluster \
  --personal

This command does the following:

• Installs Flux components into the cluster.
• Connects the cluster to a GitHub repository.
• Configures Flux to track the clusters/my-cluster directory in the repository.

---

Step 2: Create a Capsule Tenant

Create a tenant definition to manage multi-tenancy:

apiVersion: capsule.clastix.io/v1beta1
kind: Tenant
metadata:
  name: dev-team
spec:
  owner:
    kind: User
    name: dev-user
  namespaceOptions:
    quota:
      hard:
        pods: "10"
        requests.cpu: "4"
        requests.memory: "8Gi"
        limits.cpu: "8"
        limits.memory: "16Gi"

Explanation:

• owner – The user who manages the tenant.
• quota – Limits the number of resources (CPU, memory, pods) the tenant can consume.

Apply the tenant configuration:

kubectl apply -f tenant.yaml

Create a namespace for the tenant:

kubectl create namespace dev-team-app
kubectl label namespace dev-team-app capsule.clastix.io/tenant=dev-team

---

Step 3: Define GitOps Resources in Flux

Create a directory in your Git repository to store Kubernetes manifests for the tenant:

mkdir -p clusters/my-cluster/tenants/dev-team

Create a sample deployment manifest for the tenant:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  namespace: dev-team-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
        - name: sample-app
          image: nginx
          ports:
            - containerPort: 80

Add a service to expose the deployment:

apiVersion: v1
kind: Service
metadata:
  name: sample-app-service
  namespace: dev-team-app
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: sample-app

Push these files to the Git repository:

git add clusters/my-cluster/tenants/dev-team
git commit -m "Add sample app for dev-team"
git push origin main

---

Step 4: Configure Flux to Sync Tenant Resources

Create a Kustomization resource to tell Flux to apply the tenant resources:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: dev-team
  namespace: flux-system
spec:
  interval: 5m
  path: ./clusters/my-cluster/tenants/dev-team
  prune: true
  sourceRef:
    kind: GitRepository
    name: my-gitops-repo

Apply the Kustomization:

kubectl apply -f dev-team-kustomization.yaml

---

Step 5: Test and Monitor the Deployment

Verify the Deployment:

Check if the application is deployed:

kubectl get pods -n dev-team-app

Verify Flux Synchronization:

Check the Flux reconciliation status:

flux get kustomization

Sample output:

NAME       READY  MESSAGE
dev-team   True   Applied revision: main@sha1:abc123

Test the Application:

Expose the sample app using kubectl port-forward:

kubectl port-forward svc/sample-app-service -n dev-team-app 8080:80

Access the app in a browser:

http://localhost:8080

---

Step 6: Automate Future Updates Using GitOps

1. Make changes to the deployment in the Git repository:

spec:
  replicas: 2

2. Push the changes to Git:

git add .
git commit -m "Scale sample-app to 2 replicas"
git push origin main

Flux will detect the changes and automatically update the deployment in the cluster.

Monitor the update status:

flux get kustomization dev-team

---

Real-World Applications

1. Multi-Tenant SaaS Platform

• Each customer is treated as a tenant.
• Capsule manages customer-level isolation.
• Flux ensures that customer configurations are automatically updated from Git.

2. Internal Platform with Multiple Dev Teams

• Different teams manage their own Kubernetes namespaces.
• Capsule isolates resources between teams.
• Flux keeps application configurations consistent across environments.

3. Financial Services with Strict Quotas

• Different financial products operate under separate tenants.
• Capsule ensures fair resource distribution.
• Flux manages deployment consistency for regulatory compliance.

---

Best Practices

1. Define resource quotas carefully to prevent noisy neighbors.
2. Set up alerts to monitor Flux reconciliation status.
3. Use Git tags to enable rollback in case of failures.
4. Secure Git repository access to prevent unauthorized updates.
5. Monitor tenant usage and adjust quotas as needed.

---

Conclusion

By combining Capsule and Flux, you can enable a powerful GitOps-based approach to managing multi-tenant Kubernetes clusters. Capsule ensures strict tenant isolation and resource management, while Flux automates application deployment and keeps cluster state in sync with the Git repository.

This approach not only simplifies cluster management but also improves scalability and reliability. You can now confidently manage multiple tenants while ensuring consistent and predictable application behavior.

That’s it for now. I hope this article gave you some useful insights on the topic. Please feel free to drop a comment, question or suggestion.