How to Handle Logging and Tracing for Multi-Tenant Kubernetes Clusters Using Capsule and Loki

Introduction

Managing logging and tracing in a multi-tenant Kubernetes cluster is complex due to the need to isolate logs and traces across different tenants while ensuring that logs remain accessible for debugging and performance analysis.

When multiple teams or customers share the same Kubernetes cluster, they should have access to their own logs without being able to access logs from other tenants.

• Capsule helps in creating and managing multi-tenant clusters, ensuring strict isolation of resources and access.
• Loki is a scalable log aggregation system designed specifically for Kubernetes, providing powerful querying and filtering options.

In this blog, we will explore how to set up Capsule for multi-tenancy and integrate Loki for centralized logging, ensuring that logs are properly isolated and accessible only to the relevant tenants.

---

Why Capsule and Loki Work Well Together

Capsule for Multi-Tenancy:

1. Isolates namespaces, resources, and policies at the tenant level.
2. Enforces Role-Based Access Control (RBAC) to control access to logs.
3. Allows tenants to manage their own workloads independently.

Loki for Logging:

1. Lightweight, horizontally scalable log aggregation.
2. Uses Kubernetes labels and metadata to filter and query logs.
3. Efficient log storage and retrieval using index-free architecture.

When combined, Capsule and Loki enable a multi-tenant logging architecture where logs are collected centrally but are securely accessible only to the appropriate tenants.

---

Step 1: Install Capsule and Loki

Install Capsule:

You can install Capsule using Helm:

helm repo add clastix https://clastix.github.io/charts
helm repo update
helm install capsule clastix/capsule

Verify that Capsule is running:

kubectl get pods -n capsule-system

Install Loki:

Install the Loki stack using Helm:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install loki grafana/loki-stack --set promtail.enabled=true

Verify that Loki and Promtail are running:

kubectl get pods -n default

Sample output:

NAME                                     READY   STATUS    RESTARTS   AGE
loki-0                                   1/1     Running   0          5m
promtail-xyz                             1/1     Running   0          5m

---

Step 2: Create a Capsule Tenant

Create a Capsule tenant definition to manage multi-tenancy:

apiVersion: capsule.clastix.io/v1beta1
kind: Tenant
metadata:
  name: dev-team
spec:
  owner:
    kind: User
    name: dev-user
  namespaceOptions:
    quota:
      hard:
        pods: "20"
        requests.cpu: "8"
        requests.memory: "16Gi"
        limits.cpu: "16"
        limits.memory: "32Gi"

Explanation:

• owner – The user who manages the tenant.
• quota – Limits the number of resources (CPU, memory, pods) the tenant can consume.

Apply the tenant configuration:

kubectl apply -f tenant.yaml

Create a namespace for the tenant:

kubectl create namespace dev-team-app
kubectl label namespace dev-team-app capsule.clastix.io/tenant=dev-team

---

Step 3: Configure Promtail to Collect Logs Per Tenant

Promtail is a lightweight log shipping agent that ships logs to Loki. To isolate logs by tenant, configure Promtail to filter logs based on namespace labels.

Create a ConfigMap for Promtail:

Create a Promtail configuration to label logs by tenant:

apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
  namespace: default
data:
  promtail.yaml: |
    server:
      http_listen_port: 3101
      grpc_listen_port: 0

    positions:
      filename: /var/log/positions.yaml

    clients:
      - url: http://loki.default.svc.cluster.local:3100/loki/api/v1/push

    scrape_configs:
      - job_name: kubernetes-pods
        pipeline_stages:
          - docker:
          - labels:
              tenant: {{ .namespace }}
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod
          - source_labels: [__meta_kubernetes_container_name]
            target_label: container

Apply the ConfigMap:

kubectl apply -f promtail-config.yaml

---

Step 4: Deploy Promtail with the New ConfigMap

Update the Promtail deployment to use the new configuration:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
  namespace: default
spec:
  selector:
    matchLabels:
      name: promtail
  template:
    metadata:
      labels:
        name: promtail
    spec:
      containers:
        - name: promtail
          image: grafana/promtail:latest
          args:
            - "-config.file=/etc/promtail/promtail.yaml"
          volumeMounts:
            - name: config
              mountPath: /etc/promtail/
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: promtail-config

Apply the deployment:

kubectl apply -f promtail-deployment.yaml

---

Step 5: Grant RBAC Permissions for Tenant Logs

Create a RoleBinding to allow the tenant owner to access logs for their namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-team-log-access
  namespace: dev-team-app
subjects:
  - kind: User
    name: dev-user
roleRef:
  kind: Role
  name: view
  apiGroup: rbac.authorization.k8s.io

Apply the RoleBinding:

kubectl apply -f rolebinding.yaml

---

Step 6: Query Logs Using Loki and Labels

1. Get tenant-specific logs:

Use the namespace label to query tenant logs in Loki:

kubectl port-forward svc/loki 3100:3100

Query logs using curl:

curl -G http://localhost:3100/loki/api/v1/query --data-urlencode 'query={namespace="dev-team-app"}'

2. View logs in Grafana:

• Add Loki as a data source in Grafana.
• Create a new dashboard using the following query:

{namespace="dev-team-app"}

---

Step 7: Enable Tracing with Grafana Tempo (Optional)

To enable tracing for tenant workloads:

1. Install Grafana Tempo using Helm:

helm install tempo grafana/tempo

2. Configure Promtail to send traces to Tempo:

Add the following to Promtail’s configuration:

tracing:
  tempo:
    endpoint: tempo.default.svc.cluster.local:4317

---

Real-World Applications

1. Multi-Tenant SaaS Platforms

• Capsule isolates logs by customer.
• Loki aggregates logs for all customers in a central location.
• RBAC ensures that customers can only see their own logs.

2. Enterprise DevOps Teams

• Different teams operate in separate namespaces.
• Capsule isolates logs between teams.
• Loki enables centralized log searching.

3. Compliance and Auditing

• Capsule provides tenant-level separation.
• Loki enables long-term log retention.
• Logs are available for audits and troubleshooting.

---

Best Practices

1. Use tenant-specific labels to filter logs.
2. Secure access to logs using RBAC.
3. Configure log retention policies based on tenant needs.
4. Monitor Loki performance using Grafana.
5. Enable trahttps://blog.nashtechglobal.com/understanding-distributed-tracing-with-jaeger-a-beginners-guide/cing for end-to-end visibility.

---

Conclusion

By combining Capsule and Loki, you can build a scalable and secure multi-tenant logging solution for Kubernetes. Capsule ensures tenant-level isolation, while Loki provides centralized logging and advanced querying. This setup improves visibility, simplifies troubleshooting, and enhances overall security in multi-tenant Kubernetes clusters.

That’s it for now. I hope this article gave you some useful insights on the topic. Please feel free to drop a comment, question or suggestion.