Use Vault and Kubernetes to provide strong password protection

This article begins with: Chinese Community of Jenkins

Text Link Author: Johann Gyger

Translator: s1mple_zj

The integration of Vault and Kubernetes is illustrated with specific examples to make the use of passwords safer for applications running Kubernetes

introduce

Kubernetes has become the industry standard for container layouts, while Vault from HashiCorp is the industry standard for password management.The question arises: How do you combine these two techniques to allow you to use passwords from Vault central instances in Kubernetes applications?

One solution is to use AppRole Authentication. Boostport It provides perfect integration for AppRoles on Kubernetes.Another possible approach is to use Kubernetes Certification .This authentication mechanism creates a trusted connection between the Vault and Kubernetes clusters so that you can authenticate to Vault using a service account.Later you can use Vault node for Kubernetes Get and update authentication tokens.

In this practical article, I'll show you how to use some of the Go assistant tools to accomplish the same work, such as authenticating update tokens, and go one step further - synchronizing predefined subsets of passwords from Vault to Kubernetes.

Level: Advanced

Dead work

For simplicity, I have some options:

  • There are several different ways to start a Kubernetes cluster.Usually, minikube is used for testing or development.I'll use kubeadm because it's very simple to start a real cluster.

  • In Kubernetes, the default namespace is used.

  • Vault runs in development mode._Don't use it like you would in a production environment!_Make sure that VAULT_ADDR is set in the environment variable.

  • Ubuntu is used in the code sample.These have been tested on Ubuntu 18.10 VM with a GCE configuration of 2 vCPU and 7.5 GB.(Take a look at the free $300 package on the GCP s, that is, it's already done.)

  • Bash will be used unless otherwise specified.

Kubernetes

Let's start with a simple test cluster.The code below shows you what the installation steps are for a simple node.

# 1) Install Kubernetes on a Ubuntu machine
sudo -i
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo 'deb http://apt.kubernetes.io/ kubernetes-xenial main' >> /etc/apt/sources.list.d/kubernetes.list
apt update && apt install -y docker.io kubelet kubeadm kubectl
sudo systemctl enable docker.service
kubeadm init --pod-network-cidr=10.244.0.0/16  # Flannel pod network, see below
exit

# 2) Prepare kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
echo "source <(kubectl completion bash); alias k=kubectl; complete -F __start_kubectl k" >> .bashrc && exec $SHELL

# 3) Finalize K8s config 
kubectl cluster-info 
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl taint nodes --all node-role.kubernetes.io/master-
kubectl get nodes -o wide

# For details, see: 
# - https://kubernetes.io/docs/setup/independent/install-kubeadm/
# - https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

Vault

install

The installation of Vault is very straightforward: just download the unzipped binary package:

# Install Vault
sudo apt install -y zip
curl -OL https://releases.hashicorp.com/vault/1.1.1/vault_1.1.1_linux_amd64.zip
unzip vault_1.1.1_linux_amd64.zip
sudo mv vault /usr/local/bin/
vault -autocomplete-install && exec $SHELL

Run a Vault Server

We will run a Vault server in development mode.Again, it's very simple.Keep in mind that when you start a development server, a root token is written to $HOME/.vault-token, as is the case for root users.Using the &symbol causes the Vault process to run in the background so we can continue to use the same shell.

$ vault server -dev -dev-listen-address=0.0.0.0:8200 &
==> Vault server configuration:

             Api Address: http://0.0.0.0:8200
                     Cgo: disabled
         Cluster Address: https://0.0.0.0:8201
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
                 Storage: inmem
                 Version: Vault v1.1.1
             Version Sha: a3dcd63451cf6da1d04928b601bbe9748d53842e

Configure Kubernetes authentication

Now we must ensure that Kubernetes can communicate with Vault through the Kubernetes certification.This will establish a trusted relationship between Kubernetes and Vault.The pre-named vault-demo-role will map the policy and define a TTL.

Because we use the Kubernetes cluster started by kubeadm, it is very easy to find the value stored by the certification authority (CA) for the kubernetes_ca_cert parameter.This process is more difficult if you use a Kubernetes installed in the cloud.

# NOTE: You may need to set these addresses differently.
export INTERNAL_IP=$(dig +short `hostname -f`)
export VAULT_ADDR=http://${INTERNAL_IP}:8200

# Enable and configure the Kubernetes auth method.
# For details, see: 
# - https://www.vaultproject.io/docs/auth/kubernetes.html
# - https://www.vaultproject.io/api/auth/kubernetes/index.html
vault auth enable kubernetes
vault write auth/kubernetes/config \
    kubernetes_host=https://${INTERNAL_IP}:6443 \
    kubernetes_ca_cert=@/etc/kubernetes/pki/ca.crt 
vault write auth/kubernetes/role/vault-demo-role \
    bound_service_account_names=vault-serviceaccount \
    bound_service_account_namespaces=default \
    policies=vault-demo-policy \
    ttl=1h

# Create a policy for demo purposes
cat <<EOF | vault policy write vault-demo-policy -
path "sys/mounts" { capabilities = ["read"] }
path "secret/data/demo/*" { capabilities = ["read"] }
path "secret/metadata/demo/*" { capabilities = ["list"] }
EOF

# Write some demo secret
vault kv put secret/demo/most-used-password password=123456
vault kv put secret/demo/first one=1234567890 two=2345678901
vault kv put secret/demo/second green=lantern poison=ivy
vault kv put secret/demo/greek/alpha philosopher=plato
vault kv put secret/demo/greek/beta god=zeus
vault kv put secret/demo/greek/gamma mountain=olympus

Role-based access control (RBAC)

On the Kubernetes side, we now need to install RBAC with matching settings.First, we create a service account named vault-service account.We then add a cluster role binding called vault-closterrolebinding so that our newly created service account can be allowed to send authentication requests using the default cluster role system:auth-delegator.The role vault-secrectadmin-role and role binding vault-secretadmin-rolebinding are also bound to vault-service account so that we can synchronize passwords.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-serviceaccount

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: vault-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-serviceaccount
    namespace: default

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: vault-secretadmin-role
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["*"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: vault-secretadmin-rolebinding
subjects:
- kind: ServiceAccount
  name: vault-serviceaccount
roleRef:
  kind: Role
  name: vault-secretadmin-role
  apiGroup: rbac.authorization.k8s.io

Let's apply the following configuration:

$ k apply -f vault-rbac.yaml 
serviceaccount/vault-serviceaccount created
clusterrolebinding.rbac.authorization.k8s.io/vault-clusterrolebinding created
role.rbac.authorization.k8s.io/vault-secret-admin-role created
rolebinding.rbac.authorization.k8s.io/vault-demo-secret-admin-rolebinding created

Preparations are complete.Now we can move on to our use case.

use case

We will cover the following three use cases:

  • The first example demonstrates how to authenticate to Vault and then use an initialized container to obtain an authentication token.

  • The second example demonstrates how to update this token using the sidecar container.

  • The third example demonstrates how to synchronize passwords from Vault to Kubernetes.

All three use cases run on three Docker mirrors built by my colleagues at PostFinance.Special thanks to Marc Sauter, who is Seth Vargo The original implementation was compiled inspired by the work.These three mirrors are created - in Docker Hub Available on - Includes a few Go helper tools, code can be obtained from GitHub Find it on.

Authentication using the initial container

The first example shows the use of vault-kubernetes-authenticator mirrors.Auher runs in an initial container, authenticates Vault using the service account vault-service account, and writes the Vault's authentication token to/home/vault/.vault-token.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-kubernetes-authenticator-demo
  labels:
    appl: vault-kubernetes-authenticator-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      appl: vault-kubernetes-authenticator-demo
  template:
    metadata:
      labels:
        appl: vault-kubernetes-authenticator-demo
    spec:
      serviceAccountName: vault-serviceaccount
      volumes:
      - name: vault-token
        emptyDir:
          medium: Memory
      initContainers:
      - name: vault-kubernetes-authenticator
        image: postfinance/vault-kubernetes-authenticator
        imagePullPolicy: Always
        volumeMounts:
        - name: vault-token
          mountPath: /home/vault
        env:
        - name: VAULT_ADDR
          value: ${VAULT_ADDR}
        - name: VAULT_ROLE
          value: vault-demo-role
        - name: VAULT_TOKEN_PATH
          value: /home/vault/.vault-token
      containers:
      - name: kuard
        image: gcr.io/kuar-demo/kuard-amd64:blue
        volumeMounts:
        - name: vault-token
          mountPath: /home/vault

We apply these configurations and then perform some tests to verify that all configurations run successfully.

$ envsubst < vault-kubernetes-authenticator-demo.yaml | k apply -f -
deployment.apps/vault-kubernetes-authenticator-demo created

$ k get all
NAME                                                      READY   STATUS    RESTARTS   AGE
pod/vault-kubernetes-authenticator-demo-fc49b957c-b5bnx   1/1     Running   0          81s
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   20h
NAME                                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-kubernetes-authenticator-demo   1/1     1            1           81s
NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-kubernetes-authenticator-demo-fc49b957c   1         1         1       81s

$ k logs vault-kubernetes-authenticator-demo-fc49b957c-b5bnx -c vault-kubernetes-authenticator 
2019/04/16 04:45:23 successfully authenticated to vault
2019/04/16 04:45:23 successfully stored vault token at /home/vault/.vault-token

$ k exec vault-kubernetes-authenticator-demo-fc49b957c-b5bnx -- sh -c "VAULT_ADDR=${VAULT_ADDR} sh"
~ $ cat /home/vault/.vault-token; echo
s.xrrJoCARIC0Z84vcvcwuH5XG
~ $ wget --header="X-Vault-Token: $(cat /home/vault/.vault-token)" -q -O - ${VAULT_ADDR}/v1/secret/data/demo/most-used-password
{"request_id":"12660a6b-7ad0-85bc-8841-d21c7cc8248a","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":{"password":"123456"},"metadata":{"created_time":"2019-04-16T05:11:44.651116748Z","deletion_time":"","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}
~ $ wget --header="X-Vault-Token: $(cat /home/vault/.vault-token)" -q -O - ${VAULT_ADDR}/v1/secret/data/sensitive-password
wget: server returned error: HTTP/1.1 403 Forbidden

Update token with Sidecar

The second example will show you the use of vault-kubernetes-token-renewer mirrors.Renwer runs in a sidecar container, periodically checks the TTL and updates the authentication token as checked.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-kubernetes-token-renewer-demo
  labels:
    appl: vault-kubernetes-token-renewer-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      appl: vault-kubernetes-token-renewer-demo
  template:
    metadata:
      labels:
        appl: vault-kubernetes-token-renewer-demo
    spec:
      shareProcessNamespace: true
      serviceAccountName: vault-serviceaccount
      volumes:
        - name: vault-token
          emptyDir:
            medium: Memory
      initContainers:
        - name: vault-kubernetes-authenticator
          image: postfinance/vault-kubernetes-authenticator
          imagePullPolicy: Always
          volumeMounts:
            - name: vault-token
              mountPath: /home/vault
          env:
            - name: VAULT_ADDR
              value: ${VAULT_ADDR}
            - name: VAULT_ROLE
              value: vault-demo-role
            - name: VAULT_TOKEN_PATH
              value: /home/vault/.vault-token
      containers:
        - name: vault-kubernetes-token-renewer
          image: postfinance/vault-kubernetes-token-renewer
          imagePullPolicy: Always
          volumeMounts:
            - name: vault-token
              mountPath: /home/vault
          env:
            - name: VAULT_ADDR
              value: ${VAULT_ADDR}
            - name: VAULT_ROLE
              value: vault-demo-role
            - name: VAULT_TOKEN_PATH
              value: /home/vault/.vault-token
        - name: kuard
          image: gcr.io/kuar-demo/kuard-amd64:blue
          volumeMounts:
            - name: vault-token
              mountPath: /home/vault

We'll also apply these configurations and do some validation.(I deleted the previous deployment.)

$ envsubst < vault-kubernetes-token-renewer-demo.yaml | k apply -f -
deployment.apps/vault-kubernetes-token-renewer-demo created

$ k get all
NAME                                                       READY   STATUS    RESTARTS   AGE
pod/vault-kubernetes-token-renewer-demo-694cc7dbbd-rkbbs   2/2     Running   0          4s
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   31h
NAME                                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-kubernetes-token-renewer-demo   1/1     1            1           4s
NAME                                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-kubernetes-token-renewer-demo-694cc7dbbd   1         1         1       4s

$ k logs vault-kubernetes-token-renewer-demo-694cc7dbbd-rkbbs -c vault-kubernetes-authenticator 
2019/04/16 15:40:55 successfully authenticated to vault
2019/04/16 15:40:55 successfully stored vault token at /home/vault/.vault-token

$ k logs vault-kubernetes-token-renewer-demo-694cc7dbbd-rkbbs -c vault-kubernetes-token-renewer 
2019/04/16 15:40:56 start renewer loop
2019/04/16 15:40:56 token renewed

Synchronize passwords from Vault to Kubernetes

A third example shows you the use of vault-kubernetes-synchronizer (*syncer*).Syncer can be used in many ways.Inside this demo, a Kubernetes task will be used to synchronize the Vault password once from a predefined path.These Vault passwords will be written to the corresponding Kubernetes password.

---
apiVersion: batch/v1
kind: Job
metadata:
  name: vault-kubernetes-synchronizer-demo
spec:
  backoffLimit: 0
  template:
    spec:
      serviceAccountName: vault-serviceaccount
      restartPolicy: Never
      volumes:
        - name: vault-token
          emptyDir:
            medium: Memory
      initContainers:
        - name: vault-kubernetes-authenticator
          image: postfinance/vault-kubernetes-authenticator
          imagePullPolicy: Always
          volumeMounts:
            - name: vault-token
              mountPath: /home/vault
          env:
            - name: VAULT_ADDR
              value: ${VAULT_ADDR}
            - name: VAULT_ROLE
              value: vault-demo-role
            - name: VAULT_TOKEN_PATH
              value: /home/vault/.vault-token
      containers:
        - name: vault-kubernetes-synchronizer
          image: postfinance/vault-kubernetes-synchronizer
          imagePullPolicy: Always
          volumeMounts:
            - name: vault-token
              mountPath: /home/vault
          env:
            - name: VAULT_ADDR
              value: ${VAULT_ADDR}
            - name: VAULT_ROLE
              value: vault-demo-role
            - name: VAULT_TOKEN_PATH
              value: /home/vault/.vault-token
            - name: VAULT_SECRETS
              value: secret/demo/first,secret/demo/second,secret/demo/first:third,secret/demo/greek/

Similarly, we'll apply these configurations and see if all the configurations work as expected:

$ envsubst < vault-kubernetes-synchronizer-demo.yaml | k apply -f -
job.batch/vault-kubernetes-synchronizer-demo created

$ k get all
NAME                                           READY   STATUS    RESTARTS   AGE
pod/vault-kubernetes-synchronizer-demo-m2xnz   1/1     Running   0          4s
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   3d5h
NAME                                           COMPLETIONS   DURATION   AGE
job.batch/vault-kubernetes-synchronizer-demo   0/1           4s         4s

$ k logs pod/vault-kubernetes-synchronizer-demo-m2xnz -c vault-kubernetes-authenticator
2019/04/18 14:29:42 successfully authenticated to vault
2019/04/18 14:29:42 successfully stored vault token at /home/vault/.vault-token

$ k logs pod/vault-kubernetes-synchronizer-demo-m2xnz
2019/04/18 14:29:43 read secret/demo/first from vault
2019/04/18 14:29:43 update secret third from vault secret secret/demo/first
2019/04/18 14:29:43 read secret/demo/greek/alpha from vault
2019/04/18 14:29:43 update secret alpha from vault secret secret/demo/greek/alpha
2019/04/18 14:29:43 read secret/demo/greek/beta from vault
2019/04/18 14:29:43 update secret beta from vault secret secret/demo/greek/beta
2019/04/18 14:29:43 read secret/demo/greek/gamma from vault
2019/04/18 14:29:43 update secret gamma from vault secret secret/demo/greek/gamma
2019/04/18 14:29:43 read secret/demo/first from vault
2019/04/18 14:29:43 update secret first from vault secret secret/demo/first
2019/04/18 14:29:43 read secret/demo/second from vault
2019/04/18 14:29:43 update secret second from vault secret secret/demo/second
2019/04/18 14:29:44 secrets successfully synchronized

$ k get secrets
NAME                               TYPE                                  DATA   AGE
alpha                              Opaque                                1      2m43s
beta                               Opaque                                1      2m43s
default-token-ssd7f                kubernetes.io/service-account-token   3      3d5h
first                              Opaque                                2      2m43s
gamma                              Opaque                                1      2m43s
second                             Opaque                                2      2m43s
third                              Opaque                                2      2m43s
vault-serviceaccount-token-f6tnw   kubernetes.io/service-account-token   3      2d20h

$ k describe secret first
Name:         first
Namespace:    default
Labels:       <none>
Annotations:  vault-secret: secret/demo/first
Type:  Opaque
Data
====
one:  10 bytes
two:  10 bytes

$ k describe secret alpha
Name:         alpha
Namespace:    default
Labels:       <none>
Annotations:  vault-secret: secret/demo/greek/alpha
Type:  Opaque
Data
====
philosopher:  5 bytes

syncer can also be used in the cron task of Kubernetes to periodically synchronize passwords from Vault or to another initial container of Kubernetes deployment so that passwords remain up-to-date.

It is important to note that Kubernetes password protection is not very good.Seth Vargo in recent FOSDEM interview By default, they are only base64 encoded and stored, as in etcd.You should allow it Static Encryption of Data .Also make sure that you only synchronize those passwords used by your Kubernetes application, which are protected by the appropriate Vault policy and named roles.In addition, this method allows you to use passwords as cloud native behavior.Your application cannot access the Vault password directly and can be injected into environment variables.

conclusion

Kubernetes and Vault are great combinations when used together or when integrated.The integrated solution may not seem simple, but it is still possible.I have shown you how to integrate the two in this article, and I hope it will help you.

You might ask why you choose third-party mirrors when you know you can run a node with an official Vault image to accomplish the same thing.The reasons are as follows: The Vault node requires a configuration file instead of an environment variable, which also means you have to manage other configuration mappings.And the current node cannot synchronize passwords.In addition, third-party mirroring is lighter.The official Vault image is about 100 MB.The auther and renewer mirrors are about 10 MB syncer and about 40 MB.

Tags: Kubernetes sudo Docker Ubuntu

Posted on Wed, 15 Apr 2020 17:28:40 -0700 by Antnee