Secret Management in Kubernetes: The Right and Wrong Way

Secret management is a critical part of working with Kubernetes. When working with Kubernetes, you’ll often need to store and use sensitive information — like database passwords, API keys, and certificates. That’s where Kubernetes Secrets come in. But here’s the thing: while Kubernetes provides a way to store secrets, using them incorrectly can still expose your sensitive data.

This blog will walk you through:

• What Kubernetes Secrets are
• How they work
• Common mistakes (the wrong way)
• Best practices (the right way)
• Real-life examples and configurations

What Are Kubernetes Secrets?

A Secret in Kubernetes is an object that stores sensitive data, such as passwords, OAuth tokens, or SSH keys, in a Base64-encoded format.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: ZGJ1c2Vy
  password: c3VwZXJzZWNyZXQ=

Here, ZGJ1c2Vy is Base64 for dbuser and c3VwZXJzZWNyZXQ= is Base64 for supersecret.
⚠️ Important: Base64 encoding is not encryption — it’s just a way to store data in ASCII format. Therefore, anyone with access can decode it easily.

How Secrets Work in Kubernetes?

When you create a secret, Kubernetes stores it in etcd (its internal database). Pods can access it via environment variables and Mounted files in a volum.

                 ┌───────────────────────┐
                 │               Admin       │
                 │  Creates Secret (kubectl│
                 │  or manifest YAML)     │
                 └───────────┬────────────┘
                             │
                             ▼
               ┌──────────────────────────────┐
               │  Kubernetes API Server        │
               │  (Validates & stores secret)  │
               └───────────┬──────────────────┘
                             │
                             ▼
               ┌──────────────────────────────┐
               │  etcd (encrypted at rest)     │
               └───────────┬──────────────────┘
                             │
                             ▼
               ┌──────────────────────────────┐
               │  Pod requesting secret        │
               │  (via volume mount or env var)│
               └───────────┬──────────────────┘
                             │
                             ▼
               ┌──────────────────────────────┐
               │  Container in Pod uses secret │
               │  for DB auth, API keys, etc.  │
               └──────────────────────────────┘

The Wrong Way to Manage Secrets

• Hardcoding secrets in YAML files

env:
  - name: DB_PASSWORD
    value: mypassword123

• Committing secrets to Git: Once it’s in Git, it’s forever in the history.

• Using ConfigMaps instead of Secrets: ConfigMaps are not meant for sensitive data.

• By default, Kubernetes stores secrets unencrypted in etcd — so if someone compromises etcd, they can access all secrets.

The Right Way: Best Practices

• Enable Encryption at Rest for etcd: Configure Kubernetes to encrypt secrets before storing them in etcd.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-key>
      - identity: {}

• Restrict Access with RBAC: Only allow specific service accounts or namespaces to access certain secrets.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: prod
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["db-credentials"]
  verbs: ["get"]

• Avoid Passing Secrets as environment variables instead, mount them as files. This reduces the risk of secrets being exposed in process dumps or logs.

volumes:
  - name: secret-vol
    secret:
      secretName: db-credentials

• Use External Secret Managers: AWS Secrets Manager, HashiCorp Vault, Google Secret Manager.

Real-Life Example

Protecting Database Credentials in a Production E-Commerce Application
Imagine you’re running a production e-commerce platform on Kubernetes with microservices like frontend, orders-service, and payments-service.
Your orders-service needs to connect to a PostgreSQL database, which requires a username and password.

Instead of hardcoding credentials in Deployment YAMLs (which is risky because YAML files can be committed to Git), you create a Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: cG9zdGdyZXM=   # Base64 encoded value of "postgres"
  password: c2VjdXJlUGFzc3dvcmQ=  # Base64 encoded value of "securePassword"

You then mount this Secret into the orders-service as environment variables:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: orders-service
  template:
    metadata:
      labels:
        app: orders-service
    spec:
      containers:
      - name: orders-service
        image: myrepo/orders-service:latest
        env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

Why this matters:

• If an attacker gets access to your Git repository, they can’t see the credentials because they are stored securely in Kubernetes, not in code.
• You can rotate the credentials by simply updating the Secret, without redeploying the whole service.
• RBAC can limit which pods and users can read this Secret.

Conclusion

Managing sensitive information securely is not just a best practice in Kubernetes— in fact, it’s a necessity. Proper secret management in Kubernetes therefore ensures your sensitive data stays safe, your applications remain secure, and your deployments follow best practices without exposing credentials. To learn more, visit the official kubernetes documentation.