How could Hashicorp Vault enhance Kubernetes Security

Please note: this article is written by Chen Xi

Deployment Pattern

Use Hashicorp Vault for Kubernetes Security

PKI cert for Kubernetes API server

When setup kubernetes cluster, Kubernetes requires PKI for bunch of certificate for kubelet and API server, scheduler, controller and etcd.

With popular tool like kubeadm, it can auto generates all the certifcates required for a cluster to run.

However, for enterprise or production cluster, there would be no doublt to use custom certficates. With Vault’s PKI secrets engine, User can coinfigure a CA certficate and private key. Then, signing intermediate CA for kubeadm to override the self-generated CA.

It could be fairly usefully for mutiple clusters launched by kubeadm. And, the process can be automated.

Note:
This required vault deployed as tier0 service.

Refer: Certs for k8s setup

Encrypt Data at ETCD Database.

Encryption at rest was introduced in Kubernetes v1.7.0 as an alpha feature, and became stable from v1.13.0. Once enabled, APIServer will encrypt data before putting it into etcd. And, cluster can use a KMS provider for data encryption. With Vault data encryption feature, hashicorp vault can configure to be a KMS provider for data encryption at Rest.

Note:
This required vault deployed as tier0 service.

Refer:

keep your Kubernetes secrets

Kubernetes KMS Plugin Provider for HashiCorp Vault

Vault PKI for K8S certs management

Kubernetes provides a certificates.k8s.io API, which lets you provision TLS certificates signed by a Certificate Authority (CA) that you control.

Vault can be used to sign certificates for your Public Key Infrastructure.

Check: https://cert-manager.io/docs/configuration/vault/

Secrets for POD at runtime

Kubernetes Secrets let you store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys. However, kubernetes secrets are stored with base64-encoded data at ETCD.

Hashicorp Vault is a tool that can help to enhance the security.

Note:
This required vault deployed as tier0 or tiuer1 service.

Usage Pattern1:

Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar

Pod with annotatin will directly talk to vault agent and fetch secrets with service account token.

Note: this would leverage the feature: vault kubernetes auth method

To help with process, there would be init container to fetch the secrets and sidercar container that starts alongside your application for keeping secrets fresh.

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}

Refer: https://banzaicloud.com/blog/inject-secrets-into-pods-vault-revisited/ https://itnext.io/dynamic-vault-secrets-agent-sidecar-on-kubernetes-cc0ce3e54a94

Usage Pattern2:

Deploy Vault-Agent as daemonSet, use https://github.com/uber-go/fx to leavage the code inside a pod that can talk to vault-agent with service account token.

Benefits: Vault-agent has cache and auto auth feature.

Usage Pattern3:

Deploy operator controller to fetch secrets from secrets store, aka, vault and upsert into PODs. kubernetes-external-secrets

Usage Pattern4 (Prefered):

Secrets Store CSI driver for Kubernetes secrets - Integrates secrets stores with Kubernetes via a Container Storage Interface (CSI) volume.

The Secrets Store CSI driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container’s file system.On pod start and restart, the driver will communicate with the provider using gRPC to retrieve the secret content from the external Secrets Store

Note: SecretProviderClass is a custom resource, very similiar like pattern3.

HashiCorp Vault provider for the Secrets Store CSI driver: https://github.com/hashicorp/secrets-store-csi-driver-provider-vault