April 1, 2019

Setting up a Kubernetes 1.14 Raspberry Pi Cluster using kubeadm

Raspberry PI is the most popular small computer for home projects - almost everyone has the one (or ten) in his basement and you probably don’t know what to do with it. So here is the idea: let’s create a Kubernetes cluster of those dusty Raspberry Pi’s for our brave experiments!

Table of Contents

My Setup

Note: if you have RPi with wi-fi module onboard you probably don’t need to buy a switch and all those cables.

  • 3 Raspberry Pi (I have Model 3B)
  • 3 MicroSDHC cards (see this article of how to choose the right one for your RPi)
  • 1 Switch (you can choose the choose one, like D-Link GO-SW-5E)
  • 4 Ethernet cables
  • USB hub (like this one)
  • 3 mini-USB power cables (one for each RPi )

Networking

I prefer to assign IP addresses statically, e.g. on the router level. My IP-addresses are: 192.168.1.100, 192.168.1.101 and 192.168.1.102. You can configure your network the same way or choose another path. Just make sure to replace my IP-addresses in the code snippets below to yours.

Preparations

Note: You need to repeat this step for each RPi you have!

First thing first - you need to install an OS to your RPi’s. I recommend raspbian. Download it and follow this guide to install the OS to your miscoSD card. Don’t forget to activate SSH access:

touch /<volume_name>/boot/ssh

Now, unmount microSD card and put it into RPi slot. Wait for 1 minute and try to ssh to your RPi.

Post Installation Steps

SSH to first RPi (the default password is raspberry, don’t forget to change it afterward)

ssh pi@192.168.1.100
sudo hostnamectl set-hostname master01

Add hostnames into /etc/hosts:

192.168.1.100 master01
192.168.1.101 worker01
192.168.1.102 worker02

Turn off swap

sudo dphys-swapfile swapoff && sudo dphys-swapfile uninstall && sudo update-rc.d dphys-swapfile remove

Enable cgroups, e.g. add these two key-values into /boot/cmdline.txt

cgroup_enable=cpuset cgroup_enable=memory

I’ve faced some locale-related issues during the setup, if you will have the same issue, just add the following line into your .bashrc

add export LC_ALL=C to ~/.bashrc

Installing Packages

Install Docker packages

curl -s https://download.docker.com/linux/raspbian/gpg | sudo apt-key add -
echo "deb [arch=armhf] https://download.docker.com/linux/raspbian stretch edge" | sudo tee /etc/apt/sources.list.d/socker.list
sudo apt-get update -q
sudo apt-get install -y docker-ce=18.06.0~ce~3-0~raspbian --allow-downgrades
echo "docker-ce hold" | sudo dpkg --set-selections
sudo usermod pi -aG docker

Install Kubernetes packages

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update -q && sudo apt-get install -y kubeadm kubectl kubelet
sudo reboot

Repeat these steps for worker01 (IP-address: 192.168.1.101) and worker02 (IP-address: 192.168.1.102) respectively.

Master Initialization

Note: if you’re planning to add multiple master nodes later consider to use this new awesome feature --experimental-upload-certs, more about it here.

sudo kubeadm init --experimental-upload-certs --token-ttl=0 --apiserver-advertise-address=192.168.1.100

Don’t forget to copy kubeadm join command from the kubeadm init ... output, we will need it later:

kubeadm join 192.168.1.100:6443 --token <some_token> \
    --discovery-token-ca-cert-hash sha256:<some_hash>

Follow the instructions from the output and configure the regular kubectl user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Setup kubernetes overlay network

kubectl apply -f https://git.io/weave-kube-1.6
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Make sure everything is up and running

kubectl get all -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-fb8b8dccf-chnzj            1/1     Running   0          9m56s
kube-system   pod/coredns-fb8b8dccf-p8dp9            1/1     Running   0          9m56s
kube-system   pod/etcd-master01                      1/1     Running   0          9m43s
kube-system   pod/kube-apiserver-master01            1/1     Running   0          9m43s
kube-system   pod/kube-controller-manager-master01   1/1     Running   0          9m39s
kube-system   pod/kube-proxy-xtw87                   1/1     Running   0          9m56s
kube-system   pod/kube-scheduler-master01            1/1     Running   0          9m41s
kube-system   pod/weave-net-kzfzv                    2/2     Running   0          2m28s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  10m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   10m

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           <none>          10m
kube-system   daemonset.apps/weave-net    1         1         1       1            1           <none>          3m42s

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   2/2     2            2           10m

NAMESPACE     NAME                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-fb8b8dccf   2         2         2       9m56s

Workers Setup

Note: repeat these steps for worker01 and worker02 respectively

kubeadm join 192.168.1.100:6443 --token <some_token> \
    --discovery-token-ca-cert-hash sha256:<some_hash>

Checks

Wait for 2 minutes and run the following command on the master01:

kubectl get no

NAME       STATUS   ROLES    AGE     VERSION
master01   Ready    master   28m     v1.14.0
worker01   Ready    <none>   3m44s   v1.14.0
worker02   Ready    <none>   3m7s    v1.14.0

As you can see all nodes are in the Ready state, so we can start our home experiments now!

Bonus

You may want to deploy your kubernetes through the webui, if so follow this instruction to deploy Kubernetes dashboard.

On your master01 node run

echo -n 'apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system' | kubectl apply -f -

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/alternative/kubernetes-dashboard-arm.yaml

kubectl proxy --address 0.0.0.0 --accept-hosts '.*' 

Now, you can check that your dashboard is accessible, just open this link in your browser:

http://192.168.2.31:8001/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/

Creative Commons - Attribution, Noncommercial, No Derivative Works 3.0 License