Managing Docker Container Secrets with Hashicorp Vault

Docker Swarm has a built-in secret system, but storing secrets in Swarm Raft logs may not meet compliance needs. A third-party Vault secret driver can be used to route Swarm secret requests to Vault. This is less common but useful for organizations already invested in Swarm.

4. Kubernetes Integration with CSI Driver

For Kubernetes users, the Vault CSI Provider allows pods to mount Vault secrets as volumes. This uses the Container Storage Interface (CSI) to mount secrets without any application modifications. It integrates seamlessly with Kubernetes Secrets Store CSI Driver and supports automatic rotation.

Authentication Methods for Containers

Choosing the right authentication method is critical for security and automation. Common options include:

• AppRole: Ideal for automation. The container is given a RoleID and a SecretID (the latter can be a wrapped token or another secret). The container exchanges these for a Vault token.
• Kubernetes Auth: When running on Kubernetes, Vault can authenticate pods by verifying service account tokens. No explicit credentials need to be distributed.
• JWT/OIDC: Suitable for cloud-native environments where containers have JWT tokens from a trusted identity provider.
• Token: Simplest but least secure. Tokens can be pre-configured in CI/CD pipelines or injected via orchestration.

Dynamic Secrets: The Real Power

One of Vault’s strongest features for Docker workloads is dynamic secrets. Rather than storing static credentials in Vault’s key-value store, Vault can connect to a database (PostgreSQL, MySQL, MongoDB) and create a temporary user on the fly. The container receives this credential, uses it, and when the lease expires (or the container dies), Vault automatically deletes the user.

# Example: Enable PostgreSQL secrets engine
vault secrets enable database
vault write database/config/my-postgres-database \
    plugin_name=postgresql-database-plugin \
    allowed_roles="my-role" \
    connection_url="postgresql://{{username}}:{{password}}@postgres.example.com:5432/myapp" \
    username="vault_admin" \
    password="super_secret"

vault write database/roles/my-role \
    db_name=my-postgres-database \
    creation_statements="CREATE USER \"{{name}}\" WITH PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

The container then requests a credential: vault read database/creds/my-role. This returns a unique username/password valid for one hour. No static credentials are stored anywhere.

Best Practices for Docker + Vault

• Never hardcode secrets in images. Use runtime injection exclusively.
• Use least privilege Vault policies. Each container or service should only be able to read its own secrets and paths.
• Prefer Vault Agent or sidecars over embedding the Vault CLI in images. It simplifies lifecycle management and reduces image size.
• Rotate AppRole SecretIDs frequently. Use response-wrapping or periodic tokens to minimize exposure.
• Enable audit logging in Vault and ship logs to a central SIEM.
• Secure Vault itself: Use TLS for all communication, unseal via auto-unseal (KMS, cloud HSM), and restrict network access to Vault to only orchestrators and containers.
• Handle secret renewal gracefully. Applications should be able to refresh credentials without restarting. Vault Agent’s templates or environment watch mechanisms help.
• Test for failure scenarios: Simulate Vault downtime, network partitions, and token expiration to ensure applications degrade gracefully.
• Use short TTLs for dynamic secrets and tokens to limit exposure if a container is compromised.
• Consider a secrets caching layer (e.g., Vault Agent’s cache) to reduce load on Vault and improve performance for high-churn environments.

Comparison with Alternatives

While Vault is a leading solution, it’s worth understanding how it compares to other approaches:

• Docker Native Secrets (Swarm): Simple but limited to static secrets, no dynamic generation, audit trail, or fine-grained policies. Stored in Raft logs, which may not satisfy strict compliance.
• Kubernetes Secrets: Basic static secrets base64-encoded in etcd. Without encryption at rest (which requires extra configuration) they can be insecure. No central management across clusters.
• Cloud Provider Secret Managers (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager): Good integration with their ecosystems but lock-in. Typically lack dynamic secrets for databases or PKI as flexible as Vault.
• CyberArk Conjur: Enterprise-focused, strong on privileged access management, but more complex and costly than Vault for container use cases.

Vault strikes a balance between open-source flexibility, feature richness, and broad platform support, making it a popular choice for multi-cloud and hybrid Docker deployments.

Setting Up a Production-Grade Vault for Docker

1. Deploy a Highly Available Vault Cluster: Use the official Vault Helm chart on Kubernetes, or run Vault in manual HA mode with Raft storage. Ensure at least three nodes.
2. Configure Auto-Unseal: Use a cloud KMS (AWS KMS, Azure Key Vault, GCP Cloud KMS) or HSM. Never store unseal keys in the same cluster as Vault.
3. Enable Audit Devices: Send audit logs to stdout and a secure external store.
4. Set Up Policies and Roles: Create policies for each service (e.g., path "secret/myapp/*" { capabilities = ["read","list"] }). Bind these to AppRole roles or Kubernetes service accounts.
5. Integrate with Orchestration: In Kubernetes, install the Vault CSI Provider or Vault Injector (mutating webhook). For raw Docker, use Vault Agent sidecar containers via Docker Compose.
6. Test Dynamic Secrets: Enable the database secrets engine and create roles. Verify that containers can request and use temporary credentials.
7. Implement CI/CD Integration: In your pipeline, use Vault’s API to provision temporary tokens for each build stage, avoiding static credentials.

Security Considerations Beyond Secret Storage

Using Vault reduces the risk of secret exposure, but it doesn’t eliminate all attack vectors:

• Network security: Ensure Vault is not exposed to the public internet. Use internal DNS and TLS mutual authentication if possible.
• Image provenance: Verify that base images and pulled packages are from trusted registries. A compromised image could exfiltrate secrets before Vault injection.
• Runtime monitoring: Use container security tools (Falco, Tracee, AppArmor) to detect unexpected process executions or file accesses.
• Secret sprawl: Even with Vault, developers may still hardcode secrets in environment files for local testing. Enforce policies and use Vault dev proxies.
• Lease management: Applications that fail to renew leases may lose access at critical moments. Implement health checks and fallback mechanisms.

Real-World Use Case: Microservices with Database Credentials

Consider an e-commerce platform with 20 microservices, each connecting to a specific PostgreSQL database. Without Vault, each service has a hardcoded database user/password in its deployment manifest or image. Rotating passwords requires redeploying all services. With Vault:

• Each service authenticates via AppRole or Kubernetes auth.
• All services request dynamic database credentials at startup.
• Credentials are valid for 1 hour and automatically renewed by Vault Agent.
• If a service is compromised, the operator revokes all its active leases in one command.
• Database audit logs show temporary users created, reducing the blast radius of any stolen credential.

This pattern reduces operational overhead and improves security posture significantly.

Common Pitfalls and How to Avoid Them

• Forgetting to seal/unseal Vault: In production, Vault starts sealed. Automated unsealing (via auto-unseal) is essential.
• Exposing Vault tokens in logs: Use response-wrapping or Vault Agent to avoid tokens appearing in container logs.
• Mixing static and dynamic secrets: Avoid storing long-lived static secrets in Vault’s KV store for container workloads. Use dynamic secrets where possible.
• Not planning for Vault downtime: Cache secrets with TTL-aware caching, or use sidecars that can serve stale secrets temporarily.
• Overly permissive policies: A policy granting path "secret/*" to a frontend container could expose backend database passwords. Apply least privilege strictly.

Conclusion

Integrating HashiCorp Vault with Docker containers is a best practice for any organization serious about security, compliance, and operational efficiency. By moving from static, hardcoded secrets to a dynamic, centrally managed secret lifecycle, you reduce risk, simplify rotations, and gain full auditability. Whether you use agent sidecars, Kubernetes CSI, or custom entrypoint scripts, Vault provides a robust foundation for secret management in containerized environments. Start small with a single service, then expand across your fleet—your future self (and your security team) will thank you.

For further reading, consult the official Vault documentation, the Docker Secrets overview, and the Vault CSI provider guide.