Kubernetes applications frequently need access to confidential information like API keys or database credentials. Storing this data directly in container images or environment variables is not advisable and can create severe security risks.
What Are Kubernetes Secrets?
Kubernetes secrets are a Kubernetes resource specifically designed for storing sensitive data, such as passwords, tokens, and keys. They offer a secure alternative to keeping this information in plain text within your application code or configuration files. By utilizing secrets, you can reduce the chance of accidentally exposing sensitive data or unauthorized access.
In this article:
- Are Kubernetes Secrets Really Secure?
- Quick Tutorial: Working with Kubernetes Secrets
- Types of Kubernetes Secrets
- Problems with the Built-in Secrets Mechanism in Kubernetes
- Third-Party Secret Management Tools
- Limitations of Secret Management in a Container Environment
Are Kubernetes Secrets Really Secure?
While Kubernetes secrets are a major improvement over storing sensitive data in plain text, their security depends on factors like access control policies, encryption mechanisms for data at rest and in transit, and cluster management practices.
Default Security Limitations
Kubernetes Secrets are stored in etcd by default, the primary datastore for Kubernetes objects. Data is encoded using base64 but not encrypted, so anyone with access to etcd can potentially decode secrets. Unauthorized access to the cluster’s API server or nodes running workloads using secrets poses a risk.
Data Encryption at Rest
To improve security, enable encryption at rest, ensure secret data is encrypted before being written to etcd. You can choose from encryption providers like aesgcm (AES-GCM), aescbc (AES-CBC), secretbox (XSalsa20-Poly1305), or cloud-based Key Management Services like AWS KMS or Google Cloud KMS.
Audit Logging & Monitoring
Enable audit logging to monitor access to resources within the cluster. Configuring Kubernetes audit logging tracks and analyzes API requests, including those related to secrets, helping detect unauthorized access or suspicious activities.
Role-Based Access Control (RBAC)
Implement Role-Based Access Control (RBAC) to limit access to Kubernetes Secrets based on user roles. RBAC specifies permissions for users or groups concerning resources and activities. Strict RBAC policies restrict access to sensitive information in secrets only to authorized users and workloads.
Namespaces and Network Policies
Employ namespaces as an isolation layer between different applications running within the cluster. Namespaces group resources, making them inaccessible to other namespaces without permission. Implementing network policies controls network traffic between pods, reducing potential attack surfaces.
Quick Tutorial: Working with Kubernetes Secrets
Step 1: Create a New Kubernetes Secret
To create a new Kubernetes secret, use the kubectl create secret command followed by the secret type and name:
kubectl create secret generic secret-name
Replace secret-name with a desired name for your secret, such as mycredentials. You can also create a secret by defining it in a YAML file:
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
username: YWRtaW4xMjM=
password: MTIzNDVhYmNkZWY=
In this YAML file, we define a secret named mysecret with two key-value pairs: username and password. The values are base64-encoded strings to ensure they are stored securely. You can encode a string using a command like echo -n ‘yourstring’ | base64.
Step 2: View Kubernetes Secrets
To view the secrets stored in your Kubernetes cluster, use the kubectl get secret command followed by the secret name:
kubectl get secret secret-name
Replace secret-name with the name of the secret you want to view, such as mycredentials. To decode a base64-encoded secret value, use the echo command followed by the encoded secret and the base64 command with the -d flag:
echo $encoded_secret | base64 -d
Replace $encoded_secret with the encoded secret value you want to decode. This will display the original, unencoded value of the secret.
Step 3: Edit a Secret
To edit an existing secret, use the kubectl edit secrets command followed by the secret name:
kubectl edit secrets mysecret
Replace mysecret with the name of the secret you want to edit. This command will open the secret’s YAML definition in your default text editor, allowing you to modify its content. You can update the data field to change the secret’s key-value pairs:
apiVersion: v1
data:
username: YWRtaW4xMjM=
password: MTIzNDVhYmNkZWY=
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: { ... }
creationTimestamp: 2022-01-01T12:34:56Z
name: mysecret
namespace: default
resourceVersion: "123456"
uid: 12345678-1234-1234-1234-1234567890ab
type: Opaque
Remember to re-encode any updated values with base64 before saving the file.
Step 4: Use Secrets as Environment Variables in a Pod
There are several ways to enable access to secrets from your containers. One way is to use secrets as environment variables in a pod. To do this, define the env field for the container in the pod’s YAML definition:
apiVersion: v1
kind: Pod
metadata:
name: secret-env-pod
spec:
containers:
- name: mycontainer
image: redis
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: mysecret
key: username
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: password
restartPolicy: Never
In this example, we create a pod with a single container running the Redis image. The container has two environment variables, SECRET_USERNAME and SECRET_PASSWORD, which are populated with the values from the mysecret secret’s username and password keys, respectively. This allows the sensitive information to be securely passed to the container without being exposed in plain text.
Types of Kubernetes Secrets
Kubernetes Secrets come in various types, each serving a specific purpose:
- Opaque Secrets: These Secrets store arbitrary user-defined data. By default, if you don’t specify a type when creating a Secret, it is considered an Opaque Secret.
- Service Account Token Secrets: These Secrets hold token credentials identifying a service account. When using this type of Secret, ensure the kubernetes.io/service-account.name annotation is set to a valid service account name.
- Basic Authentication Secrets: These Secrets store credentials required for basic authentication. When using this type of Secret, the data field must contain at least one of the following keys:
- username: the authentication user name
- password: the authentication password or token
- SSH Authentication Secrets: These Secrets store data for SSH authentication. When using this type of Secret, you must specify an ssh-privatekey key-value pair in the data (or stringData) field as the SSH credential.
- Docker Config Secrets: These Secrets store credentials for accessing a container image registry. Docker Config Secrets use one of the following type values:
- kubernetes.io/dockercfg
- kubernetes.io/dockerconfigjson
- TLS Secrets: These Secrets store a certificate and its associated key, typically used for TLS. When using this type of Secret, you must provide the tls.key and tls.crt keys in the configuration’s data (or stringData) field.
- Bootstrap Token Secrets: These Secrets store bootstrap token data during the node bootstrap process. You typically create a bootstrap token Secret in the kube-system namespace and name it in the format bootstrap-token-<token-id>.
Problems with the Built-in Secrets Mechanism in Kubernetes
The Kubernetes project documented several security risks affecting the built-in Kubernetes secrets mechanism, which users should pay attention to:
- Securing etcd—secret data is stored in etcd. By default, etcd data is not encrypted and neither are your secrets. You should enable encryption at rest, limit access to etcd to admin users only, and safely dispose of disks where etcd data was formerly stored.
- Use SSL/TLS—when running etcd in a cluster, you must use secure peer-to-peer communication.
- You can’t share the manifest file or check it into a repo—commonly, secrets are configured using JSON or YAML files, with the secret encoded in base64. If you share or check in these manifest files, the secret is compromised. This makes it difficult to manage secrets as part of your development workflow.
- Ensure applications don’t expose the secret—even if the secret is stored and transmitted securely, an application consuming the secret may store it unsecurely, may log it, or transmit it to a third party.
- Users who consume a secret can see its value—any user who creates a pod that uses a secret has access to that secret, even if the API Server policy does not allow the user to view the secret.
- Root exploit—anyone with root access on any node can read any secret, because they can impersonate the kubelet. Kubernetes does not currently send secrets on a “need to know” basis; it exposes secrets to anyone on any node with root access.
In addition, Kubernetes secrets have some important usability issues, pointed out in the excellent post by Omer Levi Hevroni:
- No visibility or change management—secrets are critical to the operation of many services and can break a service in production (e.g., if credentials are missing or wrong). In order to manage a development pipeline and troubleshoot production issues, you need to track changes to secrets. Kubernetes provides an audit mechanism but it’s not straightforward, and there is no way to track changes to secrets using version control.
- Secrets mounted as volumes are unwieldy—secrets can be stored as environment variables or mounted as a volume. The former technique is widely agreed to be less secure. If you opt for volumes, things quickly get complex when you have a large number of keys. Kubernetes creates one file per key, and you need to read all these files from within the application. There are workarounds, but they can be equally complex.
- Not a zero-trust system—once a user is allowed to receive a secret, that user receives the secret decrypted. It would be much better to define granular permissions that specify who can encrypt a secret and prevent anyone from directly decrypting a secret. Secrets should only be decrypted on demand, when they are actually needed. Unfortunately, Kubernetes does not allow this.
For these and other reasons, most practitioners are opting for third-party management tools to help them take control of Kubernetes secrets.
Third-Party Secret Management Tools
Here are some popular tools that can help you achieve better security and usability for secrets. Their primary value is that they provide a centralized, robust mechanism for encrypting secrets and granting access to secrets. All these tools support Kubernetes, but they are not purpose-built for container workloads.
Secret management solutions fall into two broad categories:
- Cloud provider tools—including AWS Secrets Manager, Google Cloud Platform KMS and Azure Key Vault—help encrypt secrets within each cloud environment, automatically rotate secret values, manage access to secrets with policies, and perform central auditing of secrets.
- Open source tools—Hashicorp Vault provides secrets management and data protection, with advanced features like dynamic secrets, namespaces, leases, and revocation for secrets data. Other options are CyberArk (with an open source version called Conjur) and Confidant.
Limitations of Secret Management in a Container Environment
There is no doubt that secrets management tools can provide value for Kubernetes projects. However, even when you are armed with a secret management solution, you will still be limited in your ability to manage and monitor secrets.
Secrets management systems have the following serious limitations in a container environment:
- Managing which containers have access to which secrets—secrets should only be accessible to containers that actually need them. Secrets management tools don’t provide a mechanism to map secrets to relevant containers, and don’t let you know which container is actually using which secret.
- Retrieving secrets from the vault only when actually needed—nobody wants a secret inside a container image, because this exposes the secret to many more users and processes than necessary. Secrets should be transmitted only when the appropriate container runs, and not before.
- Storing secrets in containers in a secure way—even when a container receives a secret, it should not be stored on disk or accessible at the host level. Secrets should only be stored in memory, should only be accessible to the specific container that needs them, and should disappear when the container shuts down. Secrets management tools cannot do this because they don’t have direct access to individual containers.
- Pushing updated secrets directly to containers—secrets management tools are great at rotating and modifying secrets to reduce the possibility of exposure. But they can’t automatically update the relevant containers with new values of secrets. Containers can hold out-of-date secrets, which may lead to production
- issues, or you may be able to update containers after a restart, which results in uptime issues.
Aqua Security: Secrets Management for Kubernetes and Cloud Native Environments
Aqua Security is a cloud native security platform that secures containerized and serverless applications, from the CI/CD pipeline to runtime production environments.
Aqua’s cloud native security platform provides a secrets management solution that lets you centrally control secrets and how containers access those secrets. Aqua can inject secrets into a running container, ensuring that secrets only run in a container’s memory, which is more secure.
Aqua integrates with secret management tools including Amazon KMS, HashiCorp Vault, Azure Key Vault, and CyberArk Enterprise Password Vault. If you don’t have an existing secret store, Aqua provides its own encrypted database for storing secrets.
Here are other ways Aqua can improve security and usability of secrets in Kubernetes:
- Encryption at Rest—secrets are encrypted at rest through third-party storage. Access to secrets is provided using access tokens or IAM roles of the current host.
- No Persistence—Aqua securely delivers secrets to containers, encrypted at rest, loading them in memory with no persistence on disk, where they are only visible to the container that needs them.
- Access Control—you can control user access to secrets (which may be integrated with Active Directory), and group together containers with similar security features.
- Write Only—once you create a secret, it cannot be seen via the web interface or the API. You can update the content of the secret or delete it, but not read it.
- Rotation—when you modify a secret, the new value is updated in real time on the running container. You do not have to restart the container for the changes in secrets to propagate.
- Revocation—when the vault revokes a secret, Aqua remove those secrets from the containers that use them, with no need to restart the containers.
- SSH Encryption—Aqua enforces mutually authenticated, SHH encrypted communications within a Kubernetes cluster. The Enforcer uses its public key to verify the Gateway and the Enforcer makes itself known via an authentication token.
- Administration and usage audit—Each administrative access event to a secret is logged, and you can see which containers had access to which secrets during runtime.
To see how a container security platform can upgrade your Kubernetes secrets management and help you create a secure development pipeline, learn more about the Aqua Cloud Native Security Platform.