How to Optimize Resource Allocation Across Tenants Using Capsule and KEDA

Introduction

Managing resource allocation across multiple tenants in a Kubernetes cluster can be challenging. Tenants often have different resource requirements and usage patterns, making it difficult to maintain optimal performance and avoid resource contention. Capsule, a multi-tenant operator for Kubernetes, simplifies tenant isolation and resource quotas. When combined with KEDA (Kubernetes Event-driven Autoscaling), you can dynamically adjust resource allocation based on real-time demand, ensuring efficient resource utilization without over-provisioning.

In this blog, we’ll explore how to use Capsule and KEDA together to optimize resource allocation across tenants in a Kubernetes environment. We’ll walk through a practical example with code snippets to demonstrate how to configure Capsule for multi-tenancy and KEDA for autoscaling based on workload metrics.

---

Why Capsule and KEDA Work Well Together

Capsule for Multi-Tenancy:

• Capsule allows you to create and manage Kubernetes tenants.
• It enables the enforcement of resource quotas and network policies at the tenant level.
• Capsule ensures that tenants are logically separated while still operating within the same cluster.

KEDA for Dynamic Scaling:

• KEDA enables scaling based on external metrics (e.g., Kafka lag, Prometheus, Azure Monitor, etc.).
• It works by deploying ScaledObjects and Triggers that define how and when pods should scale.
• KEDA provides fine-grained control over autoscaling, reducing resource wastage and improving performance.

When combined, Capsule and KEDA provide a robust solution for multi-tenant resource optimization. Capsule enforces tenant-level isolation and resource quotas, while KEDA ensures efficient resource usage by scaling workloads dynamically.

---

Step 1: Install Capsule and KEDA

Install Capsule:

You can install Capsule using Helm:

helm repo add clastix https://clastix.github.io/charts
helm repo update
helm install capsule clastix/capsule

Install KEDA:

Install KEDA using Helm:

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda --create-namespace

---

Step 2: Create a Capsule Tenant

Create a tenant using the following manifest:

apiVersion: capsule.clastix.io/v1beta1
kind: Tenant
metadata:
  name: dev-team
spec:
  owner:
    kind: User
    name: dev-user
  namespaceOptions:
    quota:
      hard:
        pods: "10"
        requests.cpu: "5"
        requests.memory: "10Gi"
        limits.cpu: "10"
        limits.memory: "20Gi"

Explanation:

• The owner field defines the user who will manage the tenant.
• The quota section defines resource limits for the tenant.
• This ensures that the tenant cannot exceed the defined CPU, memory, and pod limits.

Apply the tenant configuration:

kubectl apply -f tenant.yaml

---

Step 3: Deploy a Sample Application Within the Tenant

Create a namespace within the tenant and deploy an example application:

kubectl create ns dev-team-app
kubectl label namespace dev-team-app capsule.clastix.io/tenant=dev-team

Create a sample deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  namespace: dev-team-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
        - name: sample-app
          image: nginx
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: "0.1"
              memory: "128Mi"
            limits:
              cpu: "0.2"
              memory: "256Mi"

Apply the deployment:

kubectl apply -f sample-app.yaml

---

Step 4: Configure KEDA to Scale the Application

Create a ScaledObject that defines how KEDA should autoscale the application based on Prometheus metrics:

Step 4.1: Install Prometheus (if not installed)

Install Prometheus using Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus

Step 4.2: Create a ScaledObject for the Application:

Create a ScaledObject that scales based on CPU usage:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sample-app-scaler
  namespace: dev-team-app
spec:
  scaleTargetRef:
    name: sample-app
  minReplicaCount: 1
  maxReplicaCount: 10
  cooldownPeriod: 30
  pollingInterval: 5
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-server.dev-team-app.svc.cluster.local
        metricName: container_cpu_usage_seconds_total
        threshold: "0.2"

Explanation:

• minReplicaCount and maxReplicaCount define the minimum and maximum number of replicas.
• cooldownPeriod and pollingInterval determine how frequently KEDA checks for metrics.
• The trigger section defines that KEDA should scale based on the Prometheus metric container_cpu_usage_seconds_total.

Apply the ScaledObject:

kubectl apply -f scaled-object.yaml

---

Step 5: Test the Configuration

Generate CPU Load:

You can test the scaling behavior by generating load:

kubectl exec -it $(kubectl get pods -n dev-team-app -l app=sample-app -o jsonpath="{.items[0].metadata.name}") -- sh -c "stress --cpu 2 --timeout 60"

Monitor the Scaling Behavior:

Check the pod status and see if KEDA scales the deployment:

kubectl get pods -n dev-team-app -w

You should see that KEDA automatically adjusts the number of replicas based on the CPU load.

---

Step 6: Monitor Scaling Activity with KEDA Metrics

You can monitor KEDA’s scaling decisions using Prometheus:

1. Access the Prometheus dashboard:

kubectl port-forward svc/prometheus-server -n dev-team-app 9090

2. Open your browser and go to:

http://localhost:9090

3. Use the query:

keda_scaledobject_ready

This will show the current state of the ScaledObject and how KEDA is handling scaling events.

---

Real-World Application:

In a production scenario, you could:

• Use Capsule to create tenants for different teams, business units, or clients.
• Configure KEDA to scale based on real-time workload metrics such as Kafka lag, HTTP request count, or database connections.
• Ensure that tenants have isolated resources but still benefit from dynamic scaling.
• Use Prometheus and Grafana for monitoring and alerting to track the scaling activity and tenant-level resource consumption.

---

Best Practices

1. Set realistic quotas in Capsule to prevent noisy neighbors from affecting other tenants.
2. Use KEDA’s cooldown period to avoid aggressive scaling up and down.
3. Fine-tune Prometheus metrics to reflect realistic scaling thresholds.
4. Monitor KEDA logs to troubleshoot scaling issues.

---

Conclusion

By combining Capsule and KEDA, you can create a highly efficient multi-tenant Kubernetes environment. Capsule ensures tenant isolation and fair resource distribution, while KEDA dynamically scales workloads based on real-time demand. This approach not only improves resource utilization but also enhances application performance and reduces infrastructure costs.

That’s it for now. I hope this article gave you some useful insights on the topic. Please feel free to drop a comment, question or suggestion.