May 27, 2019

Kubernetes Authentication with GitLab and Guard

Recently, I had written a blogpost about Kubernetes Authentication with GitHub. Then I realized that someone could prefer GitLab as his main managing system for git repositories. If it’s your case then this blogpost is what you were looking for :)

Intro

Guard from AppsCode is a Kubernetes Authentication WebHook Server. Using guard, you can log into your Kubernetes cluster using GitLab accounts, which is the easiest solution in case you don’t have LDAP infrastructure in your company but you still want to give users the possibility to authenticate against your Kubernetes clusters and to allow cluster administrators to setup RBAC rules based on membership in GitLab teams.

I hope you will enjoy this guide. Feel free to ask any questions in the comment section below (just press Load Comments).

So enough lyrics, let’s get started!

Table of Contents

  1. Preparations
  2. Guard Installation
  3. Configure Kubernetes API Server
  4. Issue Token
  5. Configure Kubectl
  6. Testing
  7. Useful Links

Preparations

Kubernetes Cluster

First of all, you need cluster admin access to a Kubernetes cluster. I recommend using minikube to test Guard and if you will like how it works you can install it on your real kubernetes clusters.

In general, minikube is the way to go for me if I want to test something new in the Kubernetes world. In most cases, it’s enough to install minikube binary and run

minikiube start

If you’ve faced any problems with minikube take a look at supported hypervisors section and/or ask your question on minikube’s slack channel.

GitLab Account

You will need a GitLab account. It’s pretty easy to get one, just follow this link.

GitLab Group

We need to create a GitLab Group as well, follow this link to do it. Note that, group names are not stable and editable in Gitlab!

Pre-Flight Checks

Ok, now when we have the access to a kubernetes cluster:

$ minikube status
host: Running
kubelet: Running
apiserver: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.64.65

$ kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health": "true"}

and we’ve created our GitLab account and a GitLab Group, so it’s time to install Guard itself!

Guard Installation

The general Guard installation guide can be found here.

Install Guard as CLI

Guard binary works as a CLI and a server. In CLI mode, you can use Guard to generate various configuration to easily deploy Guard server.

Download pre-built binaries from appscode/guard Github releases and put the binary to some directory in your PATH. To install on Linux 64-bit and MacOS 64-bit you can run the following commands:

# Linux amd 64-bit:
wget -O guard https://github.com/appscode/guard/releases/download/0.4.0/guard-linux-amd64 \
  && chmod +x guard \
  && sudo mv guard /usr/local/bin/

# Mac 64-bit
wget -O guard https://github.com/appscode/guard/releases/download/0.4.0/guard-darwin-amd64 \
  && chmod +x guard \
  && sudo mv guard /usr/local/bin/

If you’re familiar with Golang and prefer to install Guard “the go-way” feel free to run this command:

go get github.com/appscode/guard

Initialize PKI

Guard uses TLS client certs to secure the communication between guard server and Kubernetes api server. Guard also uses the CommonName and Organization in client certificate to identify which auth provider to use.

We run Guard server using a predefined Service ClusterIP 10.96.10.96 and port 443. This ClusterIP is chosen so that it falls in the default –service-cidr range for Kubeadm (and minikube uses kubeadm under the hood by default). If the service CIDR range for your cluster is different, please pick an appropriate ClusterIP.

Follow the steps below to initialize a self-signed ca and to generate a pair of server and client certificates.

# initialize self-signed ca
$ guard init ca
Wrote ca certificates in  $HOME/.guard/pki

# generate server certificate pair
$ guard init server --ips=10.96.10.96
Wrote server certificates in  $HOME/.guard/pki

# generate client certificate pair for GitLab group
# <gitlab_group_name> is the name of your GitLab group
$ guard init client <gitlab_group_name> -o Gitlab
Wrote client certificates in  $HOME/.guard/pki

$ ls -l $HOME/.guard/pki
total 48
-rw-r--r--  1 user  staff   1.0K May 22 06:38 ca.crt
-rw-------  1 user  staff   1.6K May 22 06:38 ca.key
-rw-r--r--  1 user  staff   1.1K May 22 06:40 some-group@gitlab.crt
-rw-------  1 user  staff   1.6K May 22 06:40 some-group@gitlab.key
-rw-r--r--  1 user  staff   1.0K May 22 06:38 server.crt
-rw-------  1 user  staff   1.6K May 22 06:38 server.key

Deploy Guard Server

Now, let’s deploy a guard server.

Use the following command to generate YAMLs for your particular setup and apply the created file to your Kubernetes cluster.

Keep in mind that if you’re using your own GitLab instance, you will need to provide the GitLab base-url with the path to the API, e.g. https://<base-url>/api/v4

# Base url for GitLab. Keep empty to use default gitlab base url
--gitlab.base-url=<base_url>
# generate Kubernetes YAMLs
$ guard get installer \
    --auth-providers="gitlab" \
    > installer.yaml

installer.yaml should look like this

apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: null
  labels:
    app: guard
  name: guard
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  creationTimestamp: null
  labels:
    app: guard
  name: guard
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: null
  labels:
    app: guard
  name: guard
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: guard
subjects:
- kind: ServiceAccount
  name: guard
  namespace: kube-system
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: guard
  name: guard
  namespace: kube-system
spec:
  replicas: 1
  strategy: {}
  template:
    metadata:
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ""
      creationTimestamp: null
      labels:
        app: guard
    spec:
      containers:
      - args:
        - run
        - --v=3
        - --tls-ca-file=/etc/guard/pki/ca.crt
        - --tls-cert-file=/etc/guard/pki/tls.crt
        - --tls-private-key-file=/etc/guard/pki/tls.key
        - --auth-providers=gitlab
        - --gitlab.use-group-id=false
        image: appscode/guard:0.4.0
        name: guard
        ports:
        - containerPort: 8443
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8443
            scheme: HTTPS
          initialDelaySeconds: 30
        resources: {}
        volumeMounts:
        - mountPath: /etc/guard/pki
          name: guard-pki
      nodeSelector:
        node-role.kubernetes.io/master: ""
      serviceAccountName: guard
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      volumes:
      - name: guard-pki
        secret:
          defaultMode: 365
          secretName: guard-pki
status: {}
---
apiVersion: v1
data:
  ca.crt: <some_data>
  tls.crt: <some_data>
  tls.key: <some_data>
kind: Secret
metadata:
  creationTimestamp: null
  labels:
    app: guard
  name: guard-pki
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: guard
  name: guard
  namespace: kube-system
spec:
  clusterIP: 10.96.10.96
  ports:
  - name: api
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    app: guard
  type: ClusterIP
status:
  loadBalancer: {}

The good idea is to add a liveness probe there (add this liveness probe just after the readiness probe):

...
livenessProbe:
          httpGet:
            path: /healthz
            port: 8443
            scheme: HTTPS
          initialDelaySeconds: 30
...
# deploy guard
kubectl apply -f installer.yaml

Configure Kubernetes API Server

To use webhook authentication, you need to set --authentication-token-webhook-config-file flag of your Kube-API server to a kubeconfig file describing how to access the Guard webhook service. You can use the following command to generate a sample kubeconfig file.

guard get webhook-config <gitlab_group_name> -o gitlab --addr=10.96.10.96:443

If you don’t know how to configure Kubernetes apiserver in minikube, take a look at my blogpost

Issue Token

To use Gitlab authentication, you can use your personal access token with scope api. You can use the following command to issue a token:

guard get token -o gitlab

Configure Kubectl

Now, configure kubectl for our user, token here is the one that we generated in the previous step.

kubectl config set-credentials <user_name> --token=<token>

Testing

Let’s create RBAC for our user, let’s imagine that we want to give him permission to view pods in the default namespace. In that case, our RBAC file should look like:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User
  name: evalle # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role #this must be Role or ClusterRole
  name: pod-reader # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io

Save this file as rbac.yaml and let’s apply it:

kubectl apply -f rbac.yaml

Finally, let’s test everything:

# trying to get pods in the kube-system namespace
$ kubectl get po -n kube-system --user evalle
Error from server (Forbidden): pods is forbidden: User "evalle" cannot list resource "pods" in API group "" at the cluster scope

# trying to get pods in the default namespace
$ kubectl get po --user evalle
NAME                   READY     STATUS    RESTARTS   AGE
nginx-8586cf59-5bm9g   1/1       Running   0          1m
nginx-8586cf59-7lcxg   1/1       Running   0          1m
nginx-8586cf59-t252h   1/1       Running   0          1m
nginx-8586cf59-w7m2f   1/1       Running   0          1m
nginx-8586cf59-xffhn   1/1       Running   0          1m

Awesome, it works! Now you can issue the tokens for additional users in your group and start to use your Kubernetes cluster with GitLab authentication!

Creative Commons - Attribution, Noncommercial, No Derivative Works 3.0 License