Integration Kuberentes with Vault - auth

in version 0.8.3 HashiCorp has announced that they support Kuberentes as the auth backend in Vault. This blogpost is intended to be a point of knowledge for those of you who is curious how to “glue” these two things together

Intro 1

Before we go further, please go and read this document first - Kubernetes Authentication no, really! Go and read it now, it’s important to understand these concepts.

Intro 2

Now when you understand how the Authentication works in Kubernetes I can tell you about how the Vault can make our life easier. There are no users in Vault, but roles. Kubernetes Pod authentication in Vault is based on the bound between the serviceAccount (in Kubernetes with its namespace) and the role (in Vault). That bound can be attached to every role in Vault Kubernetes-auth backend. To access secrets in Vault a client has to request the client token from Vault. To request it, the client has to identify itself. Every Pod in Kubernetes has its own JWT token which is signed by the Kubernetes CA. Every Pod’s JWT token contains the information such as: serviceAccount, namespace, uuid, secret name. Vault is able to verify the client’s JWT token as it has the Kubernete’s CA public key which can confirm the validity of the signature every JWT token contains. Once the client is validated itself, the Vault will issue a temporary client token which can be used for the secrets retrieval from Vault by Pod (based on the assigned policy to the role).

OK, to use it with Kubernetes we have to install and setup Vault first.

Vault Installation and Configuration

First, generate x509 CA cert:

$ cat openssl.cnf
[ req ]
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_ca ]
basicConstraints = critical, CA:TRUE
keyUsage = critical, digitalSignature, keyEncipherment, keyCertSign
[ v3_req_server ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
[ v3_req_client ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
[ v3_req_vaultserver ]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names_cluster
[ alt_names_cluster ]
DNS.1 = localhost
$ openssl ecparam -name secp521r1 -genkey -noout -out ca.key
$ chmod 0600 ca.key
$ openssl req -x509 -new -sha256 -nodes -key ca.key -days 3650 -out ca.pem \
-subj "/CN=localhost" -extensions v3_ca -config openssl.cnf
$ openssl ecparam -name secp521r1 -genkey -noout -out vault.key
$ chmod 0600 vault.key
$ openssl req -new -sha256 -key vault.key -subj "/CN=vault" \
     | openssl x509 -req -sha256 -CA ca.pem -CAkey ca.key -CAcreateserial \
     -out vault.pem -days 365 \
     -extensions v3_req_vaultserver -extfile openssl.cnf

Clone git repo with vault demo example

$ git clone https://github.com/Evalle/vault-demo.git

Copy keys and certificates into vault/pki directory

$ cp -p vault.key vault.pem ca.pem vault-demo/vault/pki/

Run docker-compose (you should install docker-compose package if you don’t have it)

$ cd vault-demo
$ docker-compose up -d

Setup Environment

$ export VAULT_ADDR=https://localhost:8200
$ export VAULT_CACERT=$(pwd)/vault-demo/vault/pki/ca.pem

Install Vault binary

Check the link here and install Vault binary for your OS

Check Vault status

$ vault status
Error checking seal status: Error making API request.

URL: GET https://localhost:8200/v1/sys/seal-status
Code: 400. Errors:

* server is not yet initialized

Initialize Vault

$ vault init
Unseal Key 1: UsA/zFEt9LjE9MrQRACH6UszTekMUKRUFDh+ORcBJSc6
Unseal Key 2: FazE0ImcbOjsTIir23OmrkKXvMjNiEcLeYsg9OXQe4Ex
Unseal Key 3: LIPat5sCeweVrHFFekbr0ePZ1BNqccCQPCkKmC9TYKBK
Unseal Key 4: swMTxXVCczTVGih7NSQ+GomquHL/O63vN4uG/k8LCQVo
Unseal Key 5: dsJQf+BP5Y+s1NGeo/xwwJCDeW4yrPJ5rU7ILvcGGwCN
Initial Root Token: 9f4bbad0-6099-361f-348c-b12c2f54f088

Vault initialized with 5 keys and a key threshold of 3. Please
securely distribute the above keys. When the vault is re-sealed,
restarted, or stopped, you must provide at least 3 of these keys
to unseal it again.

Vault does not store the master key. Without at least 3 keys,
your vault will remain permanently sealed.

Export Initial Root Token:

$ export VAULT_TOKEN=9f4bbad0-6099-361f-348c-b12c2f54f088

Unseal Vault

When a Vault server is started, it starts in a sealed state. In this state, Vault is configured to know where and how to access the physical storage but doesn’t know how to decrypt any of it. Unsealing is the process of constructing the master key necessary to read the decryption key to decrypt the data, allowing access to the Vault.

$ vault unseal
Key (will be hidden): UsA/zFEt9LjE9MrQRACH6UszTekMUKRUFDh+ORcBJSc6
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 1
Unseal Nonce: dbb11dee-9f57-91e2-1565-dc19181f1c6d

Repeat above command two more times with different unseal keys, and check vault status again

$ vault status
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce:
Version: 0.8.3
Cluster Name: vault-cluster-bc8e1a43
Cluster ID: 7603a8e3-8323-de4e-7374-ef4e40f0b298

High-Availability Enabled: false

Now Vault is unsealed.

Kubernetes auth backend

Vault provides a production-ready interface for Kubernetes that allows a pod to authenticate with Vault via a JWT token from a pod’s service account. Full documentation is here API documentation is here

Configuration

First, you must enable the Kubernetes auth backend:

$ vault auth-enable kubernetes
kubernetes Successfully enabled 'kubernetes' at 'kubernetes'!

Check that Kubernetes backend is available

$ vault auth -methods
Path         Type        Accessor                 ...   Description
kubernetes/  kubernetes  auth_kubernetes_f057a5ca····
token/       token       auth_token_f9d6d837      ...   token based credentials

Prior to using the Kubernetes auth backend, it must be configured. To configure it, use the /config endpoint.

$ vault write auth/kubernetes/config \
    kubernetes_host=<server_name> \
    kubernetes_ca_cert=@/etc/kubernetes/pki/ca.crt

kubernetes_host - must be a host string, a host:port pair, or a URL to the base of the Kubernetes API server (you can usually find it in /etc/kubernetes/admin.conf)

kubernetes_ca_crt - PEM encoded CA cert for use by the TLS client used to talk with the Kubernetes API.

Check that config was successfully written:

$ vault read auth/kubernetes/config
Key                     Value
---                     -----
kubernetes_ca_cert  -----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTE3MTAwNTA4MjExMloXDTI3MTAwMzA4MjExMlowFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMif
rGVyJI4LtDsAFQfuHjEi8KRnS+E4wypSsVrSLnibIXWuP8h2TsEwv9io+SKLrpSC
AUwyUaaZ2vZSPWTc5+Q7OZqTZ2+cYZu2O5mi+42kFQky57XRLC9yPTxqZ8nyagJy
ReH+Ot87nyjWDdNH878wjgfeyR4MBrTIr0hoA3Qy0qr8FR0IgMvxdnFr7+9Msooh
b1KEf47xlrMOCV+/HnYudAL67u3E1PnelsYNgXF3G7O0pfV0Vb/mNzHQLtQCVwBJ
H0Ene1xV3B1YFzIGzsOi829mi5FRAgJdwie5fIOgF9zdpl9i2dSeAnzm+O/E2bzK
NXEDq+XQ+GxMtOYeVeECAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAJuWKuog2HlJ0cDjsaKayKBiGdoh
1lhhApRJW5X+aTz6mMJ+0Rh06zi/puWfb9ZxU0eNTXn1ukklxJ+pNU3HP3hjHDB6
AFis3GNBeLZLTXhLdRNvYQWURa6Dl7Ey8d9uRKPY6dewhIhHVhtTpAMxUsukNwro
JC+8DkqjGZnw/4drMElVJwEveieqUY/MsSq9KZjr6coXsYO5xgovoQKhLgQlMgEY
gqvKmwJK5Fd1HVZ+0HkWaTRYE54TPOvyroWGY9KZBLm4/cWZye5j7bKdS8NQGtyg
1yjONzXWrqzvmpx0tmPX3PFs8ySpZanvjz5kwZHZ/0Lht3jxtlydnepnOeQ=
-----END CERTIFICATE-----
kubernetes_host          https://172.26.79.220:6443
pem_keys                 []

Authentication with this backend is role based. Before a token can be used to log in it first must be configured in a role.

$ vault write auth/kubernetes/role/demo \
    bound_service_account_names=vault-auth \
    bound_service_account_namespaces=default \
    policies=default \
    ttl=1h

This role Authorizes the vault-auth service account in the default namespace and it gives it the default policy. Service Accounts used in this backend will need to have access to the TokenReview API. If Kubernetes is configured to use Role Based Access Control the Service Account should be granted permissions to access this API. The following example ClusterRoleBinding could be used to grant these permissions:

$ kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: default
< press ctrl+D >

Create Kubernetes ServiceAccount:

$ kubectl create sa vault-auth
serviceaccount "vault-auth" created

, this creates a service account in the current namespace and an associated secret. Make sure that ServiceAccount was created successfully, and that it has some secret:

$ kubectl get sa vault-auth -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: 2017-10-04T13:31:09Z
name: vault-auth
namespace: default
resourceVersion: "1120675"
selfLink: /api/v1/namespaces/default/serviceaccounts/vault-auth
uid: 4848b1dc-a908-11e7-9a55-fa163ef25625
secrets:
- name: vault-auth-token-vv84m

Check Kubernetes secrets:

$ kubectl get secrets
NAME                   TYPE                                  DATA  AGE
default-token-09k5l    kubernetes.io/service-account-token   3     8d
vault-auth-token-vv84m kubernetes.io/service-account-token   3     1h

Obtain service account JWT token and decode it:

$ kubectl get secrets vault-auth-token-vv84m -o yaml | grep token
token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJblpoZFd4MExXRjFkR2d0ZEc5clpXNHRkblk0TkcwaUxDSnJkV0psY201bGRHVnpMbWx2TDNObGNuWnBZMlZoWTJOdmRXNTBMM05sY25acFkyVXRZV05qYjNWdWRDNXVZVzFsSWpvaWRtRjFiSFF0WVhWMGFDSXNJbXQxWW1WeWJtVjBaWE11YVc4dmMyVnlkbWxqWldGalkyOTFiblF2YzJWeWRtbGpaUzFoWTJOdmRXNTBMblZwWkNJNklqUTRORGhpTVdSakxXRTVNRGd0TVRGbE55MDVZVFUxTFdaaE1UWXpaV1l5TlRZeU5TSXNJbk4xWWlJNkluTjVjM1JsYlRwelpYSjJhV05sWVdOamIzVnVkRHBrWldaaGRXeDBPblpoZFd4MExXRjFkR2dpZlEuaVl1TlAzMWc0M2ZfRU9ZV0FaWWhqMGlwZGVHbUJtUWxfTURwS3F3a1FKNktOQUZGYWJZX2JyclFRUXNlS2hEd2tJZkhLMUtvRkc0UjROcHUxaWw3c2xUbExVZU9pYnFZTUpSUHNTY2ZWN1gwaDc1cUJ1OXZ5VnlfN3dWcF92YUFtWUZmRjBYTGdOOVFlUVVUVXdqemdqWXFNRS1SRmthWlBLWHU2M3h1MWM0ZnQydFRzQWFLcGlYWDRrRDZrbXBSbWpEYzR4a1p2NGJ3RkR0T3Z5dndBSDNqTTRiZXZHY3lRcVFwSVktakRtMjNJcVV3eUFHMUcwdmZnbmVVdzhTbHlkUVFzSks1Y1N2X19LendNWm5mQmd2MzVxaVpvMjltdC1zRTZXaTZwSjlBdFBVanc1YTI5NEJvNkhDcGYwUlFCaXFLcWxwcC1EaWRRV1NTZkN5UjNn
ame: vault-auth-token-vv84m
selfLink: /api/v1/namespaces/default/secrets/vault-auth-token-vv84m
type: kubernetes.io/service-account-token

$ echo "ZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJblpoZFd4MExXRjFkR2d0ZEc5clpXNHRkblk0TkcwaUxDSnJkV0psY201bGRHVnpMbWx2TDNObGNuWnBZMlZoWTJOdmRXNTBMM05sY25acFkyVXRZV05qYjNWdWRDNXVZVzFsSWpvaWRtRjFiSFF0WVhWMGFDSXNJbXQxWW1WeWJtVjBaWE11YVc4dmMyVnlkbWxqWldGalkyOTFiblF2YzJWeWRtbGpaUzFoWTJOdmRXNTBMblZwWkNJNklqUTRORGhpTVdSakxXRTVNRGd0TVRGbE55MDVZVFUxTFdaaE1UWXpaV1l5TlRZeU5TSXNJbk4xWWlJNkluTjVjM1JsYlRwelpYSjJhV05sWVdOamIzVnVkRHBrWldaaGRXeDBPblpoZFd4MExXRjFkR2dpZlEuaVl1TlAzMWc0M2ZfRU9ZV0FaWWhqMGlwZGVHbUJtUWxfTURwS3F3a1FKNktOQUZGYWJZX2JyclFRUXNlS2hEd2tJZkhLMUtvRkc0UjROcHUxaWw3c2xUbExVZU9pYnFZTUpSUHNTY2ZWN1gwaDc1cUJ1OXZ5VnlfN3dWcF92YUFtWUZmRjBYTGdOOVFlUVVUVXdqemdqWXFNRS1SRmthWlBLWHU2M3h1MWM0ZnQydFRzQWFLcGlYWDRrRDZrbXBSbWpEYzR4a1p2NGJ3RkR0T3Z5dndBSDNqTTRiZXZHY3lRcVFwSVktakRtMjNJcVV3eUFHMUcwdmZnbmVVdzhTbHlkUVFzSks1Y1N2X19LendNWm5mQmd2MzVxaVpvMjltdC1zRTZXaTZwSjlBdFBVanc1YTI5NEJvNkhDcGYwUlFCaXFLcWxwcC1EaWRRV1NTZkN5UjNn" > vault-auth.encoded.jwt

$ base64 -d vault-auth.encoded.jwt
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LWF1dGgtdG9rZW4tdnY4NG0iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtYXV0aCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjQ4NDhiMWRjLWE5MDgtMTFlNy05YTU1LWZhMTYzZWYyNTYyNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnZhdWx0LWF1dGgifQ.iYuNP31g43f_EOYWAZYhj0ipdeGmBmQl_MDpKqwkQJ6KNAFFabY_brrQQQseKhDwkIfHK1KoFG4R4Npu1il7slTlLUeOibqYMJRPsScfV7X0h75qBu9vyVy_7wVp_vaAmYFfF0XLgN9QeQUTUwjzgjYqME-RFkaZPKXu63xu1c4ft2tTsAaKpiXX4kD6kmpRmjDc4xkZv4bwFDtOvyvwAH3jM4bevGcyQqQpIY-jDm23IqUwyAG1G0vfgneUw8SlydQQsJK5cSv__KzwMZnfBgv35qiZo29mt-sE6Wi6pJ9AtPUjw5a294Bo6HCpf0RQBiqKqlpp-DidQWSSfCyR3g

Now you can authenticate with CLI or via API and obtain necessary token

$ vault write auth/kubernetes/login role=demo jwt=$jwt
Key                                     Value
---                                     -----
token                                   bc2dc7d8-7fec-c749-cc7b-87a01a486980
token_accessor                          d8d524b4-66fc-b53e-2a37-6ef53528bc2a
token_duration                          1h0m0s
token_renewable                         true
token_policies                          [default]
token_meta_role                         "demo"
token_meta_service_account_name         "vault-auth"
token_meta_service_account_namespace    "default"
token_meta_service_account_secret_name  "vault-auth-token-vv84m"
token_meta_service_account_uid          "4848b1dc-a908-11e7-9a55-fa163ef25625

Use client JWT token for pod identification in Vault

Create a Pod:

$ kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: alpine-sleep
spec:
  serviceAccount: vault-auth
  containers:
  - name: alpine
    image: alpine
    args:
    - sleep
    - "1000000"
< press ctrl+D >

Check that the Pod was created successfully

$ kubectl get po
NAME                                    READY     STATUS    RESTARTS   AGE
alpine-sleep                            1/1       Running   0          6

Obtain JWT token from the pod

$ kubectl exec alpine-sleep -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LWF1dGgtdG9rZW4tcHg1OHYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtYXV0aCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImMwZGE2Mzc3LWE5YWQtMTFlNy1hZTYyLWZhMTYzZTA1YTAyNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnZhdWx0LWF1dGgifQ.gTH-oEoASbfQjr0nwyrL2u6sWE969Wtw4LaCOvziumVXf0arzpSHGAjKiifAAxjAY9YIdRyebu8XGfweioW0OGQO9P-S0T6Kv4srS-0h8KqxAsYabmQFVECmLrjbOnJpMiYbzaughV2C7BMPJtR-l3r_EbAxhU42jGK2xY9xxdsmk_iimHUc2Ao8rgccTw81f73qYcZNYFqKzZDuUnZxG5Q8SgYLFKEQGpkKf3xU70TPdUKUi1V0SESSJp-gnT7tcpDRyBnlj_ygS1-lbzbfvdT_MzK1sNoxnpetS-cWTCF9jLSsM7Z8rWWCoExV_quWM2iEwr7BedxmlJswPcs6UQ

Get client token

$ curl  --cacert ${VAULT_CACERT} -sSL ${VAULT_ADDR}/v1/auth/kubernetes/login -d '{ "jwt": "PASTE-JWT-HERE", "role": "demo" }' | jq .
{
  "request_id": "1e7d31d5-5f82-d552-ae0c-424da90049dd",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": null,
  "wrap_info": null,
  "warnings": null,
  "auth": {
    "client_token": "6c2ce31b-b9d9-45eb-12e4-b30164478feb",
    "accessor": "af942cd6-e409-d2d1-0456-c87464371e8f",
    "policies": [
      "default"
    ],
    "metadata": {
      "role": "demo",
      "service_account_name": "vault-auth",
      "service_account_namespace": "default",
      "service_account_secret_name": "vault-auth-token-px58v",
      "service_account_uid": "c0da6377-a9ad-11e7-ae62-fa163e05a027"
    },
    "lease_duration": 3600,
    "renewable": true
  }
}

Use client JWT token for self-lookup

$ curl -sSL --cacert ${VAULT_CACERT} --header "X-Vault-Token: 6c2ce31b-b9d9-45eb-12e4-b30164478feb" ${VAULT_ADDR}/v1/auth/token/lookup-self | jq .
{
  "request_id": "36f2bb97-e1c4-a8c1-0fb1-095f1b0bf104",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "accessor": "af942cd6-e409-d2d1-0456-c87464371e8f",
    "creation_time": 1507204651,
    "creation_ttl": 3600,
    "display_name": "kubernetes-vault-auth",
    "expire_time": "2017-10-05T12:57:31.130858038Z",
    "explicit_max_ttl": 0,
    "id": "6c2ce31b-b9d9-45eb-12e4-b30164478feb",
    "issue_time": "2017-10-05T11:57:31.130857402Z",
    "meta": {
      "role": "demo",
      "service_account_name": "vault-auth",
      "service_account_namespace": "default",
      "service_account_secret_name": "vault-auth-token-px58v",
      "service_account_uid": "c0da6377-a9ad-11e7-ae62-fa163e05a027"
    },
    "num_uses": 0,
    "orphan": true,
    "path": "auth/kubernetes/login",
    "policies": [
      "default"
    ],
    "renewable": true,
    "ttl": 3537
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

Outro

So there you have it and here are some additional notes: