Kubernetes

https://kubernetes.io

Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management. Google originally designed Kubernetes, but the Cloud Native Computing Foundation now maintains the project.

Goal is to install High Available Cluster with High Available Control plane. Virtual machines can be deployed at any on-premise or cloud virtualization providers, but this guide requires OS of the virtual machine to be Rocky Linux 8.

Also you can build huge cluster on-premise or in cloud with https://github.com/cybozu-go/neco

Installation

List of servers

Hostname	IP Address	Cluster Role
k8s-bastion.home.lichnak.cz	10.0.0.28	Bastion host
k8s-master-01.home.lichnak.cz	10.0.0.29	Master Node
k8s-master-02.home.lichnak.cz	10.0.0.30	Master Node
k8s-master-03.home.lichnak.cz	10.0.0.31	Master Node
k8s-worker-01.home.lichnak.cz	10.0.0.32	Worker Node
k8s-worker-02.home.lichnak.cz	10.0.0.33	Worker Node
k8s-worker-03.home.lichnak.cz	10.0.0.34	Worker Node
k8s-worker-04.home.lichnak.cz	10.0.0.35	Worker Node

Virtual Machines Specs

Memory

$ free -h

Disk space

$ df -hT /

CPU Cores

$ egrep ^processor /proc/cpuinfo | wc -l

DNS Settings

; Create entries for the master nodes
k8s-master-01  IN A 10.0.0.29
k8s-master-02  IN A 10.0.0.30
k8s-master-03  IN A 10.0.0.31
; Create entries for the worker nodes
k8s-worker-01  IN A 10.0.0.32
k8s-worker-02  IN A 10.0.0.33
k8s-worker-03  IN A 10.0.0.34
;
; The Kubernetes cluster ControlPlaneEndpoint point these to the IP of the masters
k8s-endpoint IN A 10.0.0.29
k8s-endpoint IN A 10.0.0.30
k8s-endpoint IN A 10.0.0.31

Step 1: Prepare Bastion Server for Kubernetes installation

Install basic tools

sudo yum -y install git wget curl vim bash-completion

Install Ansible configuration management

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

python3 get-pip.py --user

Installing Ansible using pip

python3 -m pip install ansible --user`

Check Ansible version

$ ansible --version
ansible [core 2.11.5]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.7 (default, Aug 30 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]
  jinja version = 3.0.1
  libyaml = True

Update /etc/hosts file in your Bastion machine

$ sudo vim /etc/hosts
10.0.0.28  k8s-bastion.home.lichnak.cz k8s-bastion

10.0.0.29 k8s-master-01.home.lichnak.cz k8s-master-01
10.0.0.30 k8s-master-02.home.lichnak.cz k8s-master-02
10.0.0.31 k8s-master-03.home.lichnak.cz k8s-master-03

10.0.0.32 k8s-worker-01.home.lichnak.cz k8s-worker-01
10.0.0.33 k8s-worker-02.home.lichnak.cz k8s-worker-02
10.0.0.34 k8s-worker-03.home.lichnak.cz k8s-worker-03
10.0.0.35 k8s-worker-04.home.lichnak.cz k8s-worker-04

Generate SSH keys

$ ssh-keygen -t rsa -b 4096 -N ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:LwgX4oCWENqWAyW9oywAv9jTK+BEk4+XShgX0galBqE root@k8s-master-01.home.lichnak.cz
The key's randomart image is:
+---[RSA 4096]----+
|OOo              |
|B**.             |
|EBBo. .          |
|===+ . .         |
|=*+++ . S        |
|*=++.o . .       |
|=.o. .. . .      |
| o. .    .       |
|   .             |
+----[SHA256]-----+

Create SSH client configuration file

$ vim ~/.ssh/config
Host *
    UserKnownHostsFile /dev/null
    StrictHostKeyChecking no
    IdentitiesOnly yes
    ConnectTimeout 0
    ServerAliveInterval 30

Copy SSH keys to all Kubernetes cluster nodes

# Master Nodes
for host in k8s-master-0{1..3}; do
  ssh-copy-id root@$host
done

# Worker Nodes
for host in k8s-worker-0{1..4}; do
  ssh-copy-id root@$host
done

Step 2: Set correct hostname on all nodes

# Examples
# Master Node 01
sudo hostnamectl set-hostname k8s-master-01.home.lichnak.cz

# Worker Node 01
sudo hostnamectl set-hostname k8s-worker-01.home.lichnak.cz

Logout then back in to confirm the hostname is set correctly

$ hostnamectl
   Static hostname: k8s-master-01.home.lichnak.cz
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 7a7841970fc6fab913a02ca8ae57fe18
           Boot ID: 4388978e190a4be69eb640b31e12a63e
    Virtualization: kvm
  Operating System: Rocky Linux 8.4 (Green Obsidian)
       CPE OS Name: cpe:/o:rocky:rocky:8.4:GA
            Kernel: Linux 4.18.0-305.19.1.el8_4.x86_64
      Architecture: x86-64

Step 3: Prepare Rocky Linux 8 servers for Kubernetes (Pre-reqs setup)

Ansible role for doing the standard Kubernetes node preparation. The role contain the tasks to:

Install standard packages required to manage nodes
Setup standard system requirements – Disable Swap, Modify sysctl, Disable - SELinux
Install and configure a container runtime of your Choice – cri-o, Docker, Containerd
Install the Kubernetes packages – kubelet, kubeadm and kubectl
Configure Firewalld on Kubernetes Master and Worker nodes – open all ports required
Clone my Ansible role that we’ll use to setup Kubernetes Requirements before kubeadm init to your Bastion machine:

git clone https://github.com/jmutai/k8s-pre-bootstrap.git
Switch to k8s-pre-bootstrap directory created from the clone process:

cd k8s-pre-bootstrap

Set hosts inventory correctly with your Kubernetes nodes. Here is the inventory list:

# vim hosts
[k8snodes]
k8s-master-01
k8s-master-02
k8s-master-03
k8s-worker-01
k8s-worker-02
k8s-worker-03
k8s-worker-04

You should also update the variables in playbook file. The most important being:

Kubernetes version: k8s_version
Your timezone: timezone
Kubernetes CNI to use: k8s_cni
Container runtime: container_runtime

# vim  k8s-prep.yml
---
- name: Setup Proxy
  hosts: k8snodes
  remote_user: root
  become: yes
  become_method: sudo
  #gather_facts: no
  vars:
    k8s_version: "1.21"                                  # Kubernetes version to be installed
    selinux_state: permissive                            # SELinux state to be set on k8s nodes
    timezone: "Africa/Nairobi"                           # Timezone to set on all nodes
    k8s_cni: calico                                      # calico, flannel
    container_runtime: cri-o                             # docker, cri-o, containerd
    configure_firewalld: true                            # true / false
    # Docker proxy support
    setup_proxy: false                                   # Set to true to configure proxy
    proxy_server: "proxy.home.lichnak.cz:8080"               # Proxy server address and port
    docker_proxy_exclude: "localhost,127.0.0.1"          # Addresses to exclude from proxy
  roles:
    - kubernetes-bootstrap

Check Playbook syntax:

$ ansible-playbook  --syntax-check -i hosts k8s-prep.yml

playbook: k8s-prep.yml

If your SSH private key has a passphrase then save it to prevent prompts at the time of executing playbook:

eval `ssh-agent -s` && ssh-add

Run the playbook to prepare your nodes:

ansible-playbook -i hosts k8s-prep.yml

If the servers are accessible execution should start immediately:

PLAY [Setup Proxy] ***********************************************************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************************************************************
ok: [k8s-worker-02]
ok: [k8s-worker-01]
ok: [k8s-master-03]
ok: [k8s-master-01]
ok: [k8s-worker-03]
ok: [k8s-master-02]
ok: [k8s-worker-04]

TASK [kubernetes-bootstrap : Add the OS specific variables] ******************************************************************************************************************************************
ok: [k8s-master-01] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-master-02] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-master-03] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-worker-01] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-worker-02] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-worker-03] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)
ok: [k8s-worker-04] => (item=/root/k8s-pre-bootstrap/roles/kubernetes-bootstrap/vars/RedHat8.yml)

TASK [kubernetes-bootstrap : Put SELinux in permissive mode] *****************************************************************************************************************************************
changed: [k8s-master-01]
changed: [k8s-worker-01]
changed: [k8s-master-03]
changed: [k8s-master-02]
changed: [k8s-worker-02]
changed: [k8s-worker-03]
changed: [k8s-worker-04]

TASK [kubernetes-bootstrap : Update system packages] *************************************************************************************************************************************************

After the setup confirm there are no errors in the output:

...output omitted...
PLAY RECAP *******************************************************************************************************************************************************************************************
k8s-master-01              : ok=28   changed=20   unreachable=0    failed=0    skipped=12   rescued=0    ignored=0
k8s-master-02              : ok=28   changed=20   unreachable=0    failed=0    skipped=12   rescued=0    ignored=0
k8s-master-03              : ok=28   changed=20   unreachable=0    failed=0    skipped=12   rescued=0    ignored=0
k8s-worker-01              : ok=27   changed=19   unreachable=0    failed=0    skipped=13   rescued=0    ignored=0
k8s-worker-02              : ok=27   changed=19   unreachable=0    failed=0    skipped=13   rescued=0    ignored=0
k8s-worker-03              : ok=27   changed=19   unreachable=0    failed=0    skipped=13   rescued=0    ignored=0
k8s-worker-04              : ok=27   changed=19   unreachable=0    failed=0    skipped=13   rescued=0    ignored=0

Configured /etc/hosts file contents:

[root@k8s-master-01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.0.0.29 k8s-master-01.home.lichnak.cz k8s-master-01
10.0.0.30 k8s-master-02.home.lichnak.cz k8s-master-02
10.0.0.31 k8s-master-03.home.lichnak.cz k8s-master-03
10.0.0.32 k8s-worker-01.home.lichnak.cz k8s-worker-01
10.0.0.33 k8s-worker-02.home.lichnak.cz k8s-worker-02
10.0.0.34 k8s-worker-03.home.lichnak.cz k8s-worker-03
10.0.0.35 k8s-worker-04.home.lichnak.cz k8s-worker-04

Status of cri-o service:

[root@k8s-master-01 ~]# systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
   Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-24 18:06:53 EAT; 11min ago
     Docs: https://github.com/cri-o/cri-o
 Main PID: 13445 (crio)
    Tasks: 10
   Memory: 41.9M
   CGroup: /system.slice/crio.service
           └─13445 /usr/bin/crio

Sep 24 18:06:53 k8s-master-01.home.lichnak.cz crio[13445]: time="2021-09-24 18:06:53.052576977+03:00" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_>
Sep 24 18:06:53 k8s-master-01.home.lichnak.cz crio[13445]: time="2021-09-24 18:06:53.111352936+03:00" level=info msg="Conmon does support the --sync option"
Sep 24 18:06:53 k8s-master-01.home.lichnak.cz crio[13445]: time="2021-09-24 18:06:53.111623836+03:00" level=info msg="No seccomp profile specified, using the internal default"
Sep 24 18:06:53 k8s-master-01.home.lichnak.cz crio[13445]: time="2021-09-24 18:06:53.111638473+03:00" level=info msg="AppArmor is disabled by the system or at CRI-O build-time"
Sep 24 18:06:53 k8s-master-01.home.lichnak.cz crio[13445]: time="2021-09-24 18:06:53.117006450+03:00" level=info msg="Found CNI network crio (type=bridge) at /etc/cni/net.d/100-crio-bridge.co>
Sep 24 18:06:53 k8s-master-01.home.lichnak.cz crio[13445]: time="2021-09-24 18:06:53.120722070+03:00" level=info msg="Found CNI network 200-loopback.conf (type=loopback) at /etc/cni/net.d/200>
Sep 24 18:06:53 k8s-master-01.home.lichnak.cz crio[13445]: time="2021-09-24 18:06:53.120752984+03:00" level=info msg="Updated default CNI network name to crio"
Sep 24 18:06:53 k8s-master-01.home.lichnak.cz crio[13445]: W0924 18:06:53.126936   13445 hostport_manager.go:71] The binary conntrack is not installed, this can cause failures in network conn>
Sep 24 18:06:53 k8s-master-01.home.lichnak.cz crio[13445]: W0924 18:06:53.130986   13445 hostport_manager.go:71] The binary conntrack is not installed, this can cause failures in network conn>
Sep 24 18:06:53 k8s-master-01.home.lichnak.cz systemd[1]: Started Container Runtime Interface for OCI (CRI-O).

Configured sysctl kernel parameters

[root@k8s-master-01 ~]# sysctl -p
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

Firewalld opened ports:

[root@k8s-master-01 ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp1s0
  sources:
  services: cockpit dhcpv6-client ssh
  ports: 22/tcp 80/tcp 443/tcp 6443/tcp 2379-2380/tcp 10250/tcp 10251/tcp 10252/tcp 30000-32767/tcp 4789/udp 5473/tcp 179/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Step 4: Bootstrap Kubernetes Control Plane (Single / Multi-node)

We’ll use kubeadm init command to initialize a Kubernetes control-plane node. It will start by running a series of pre-flight checks to validate the system state before making changes

Below are the key options you should be aware of:

–apiserver-advertise-address: The IP address the API Server will advertise it’s listening on. If not set the default network interface will be used.
–apiserver-bind-port: Port for the API Server to bind to; default is 6443
–control-plane-endpoint: Specify a stable IP address or DNS name for the control plane.
–cri-socket: Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket.
–dry-run: Don’t apply any changes; just output what would be done
–image-repository: Choose a container registry to pull control plane images from; Default: “k8s.gcr.io“
–kubernetes-version: Choose a specific Kubernetes version for the control plane.
–pod-network-cidr: Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
–service-cidr: Use alternative range of IP address for service VIPs. Default: “10.96.0.0/12“

Option 1: Bootstrapping single node Control Plane Kubernetes Cluster

If you have plans to upgrade a single control-plane kubeadm cluster to high availability you should specify the --control-plane-endpoint to set the shared endpoint for all control-plane nodes.

But if this is meant for test environment with single node control plane then you can ignore –control-plane-endpoint option.

[root@k8s-bastion ~]# ssh k8s-master-01
Warning: Permanently added 'k8s-master-01' (ED25519) to the list of known hosts.
Last login: Fri Sep 24 18:07:55 2021 from 10.0.0.28
[root@k8s-master-01 ~]#

Then initialize the Control Plane:

sudo kubeadm init --pod-network-cidr=192.168.0.0/16

Deploy Calico Pod network to the cluster fist master

Use the commands below to deploy the Calico Pod network add-on:

kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml

Check pods status with the following command:

[root@k8s-master-01 ~]# kubectl get pods -n calico-system
NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-bdd5f97c5-6856t   1/1     Running   0          60s
calico-node-vnlkf                         1/1     Running   0          60s
calico-typha-5f857549c5-hkbwq             1/1     Running   0          60s

Option 2: Bootstrapping Multi-node Control Plane Kubernetes Cluster

The --control-plane-endpoint option can be used to set the shared endpoint for all control-plane nodes. This option allows both IP addresses and DNS names that can map to IP addresses

Example of A records in Bind DNS server:

; Create entries for the master nodes
k8s-master-01  IN A 10.0.0.29
k8s-master-02  IN A 10.0.0.30
k8s-master-03  IN A 10.0.0.31

;
; The Kubernetes cluster ControlPlaneEndpoint point these to the IP of the masters
k8s-endpoint IN A 10.0.0.29
k8s-endpoint IN A 10.0.0.30
k8s-endpoint IN A 10.0.0.31

Example of A records /etc/hosts file:

$ sudo vim /etc/hosts

10.0.0.29 k8s-master-01.home.lichnak.cz k8s-master-01
10.0.0.30 k8s-master-02.home.lichnak.cz k8s-master-02
10.0.0.31 k8s-master-03.home.lichnak.cz k8s-master-03

####  Kubernetes cluster ControlPlaneEndpoint Entries ###
10.0.0.29 k8s-endpoint.home.lichnak.cz  k8s-endpoint
#10.0.0.30 k8s-endpoint.home.lichnak.cz  k8s-endpoint
#10.0.0.31 k8s-endpoint.home.lichnak.cz  k8s-endpoint

Using Load Balancer IP for ControlPlaneEndpoint

The most ideal approach for HA setups is mapping ControlPlane Endpoint to a Load balancer IP. The LB will then point to the Control Plane nodes with some form of health checks.

# Entry in Bind DNS Server
k8s-endpoint IN A 192.168.200.8

# Entry in /etc/hosts file
192.168.200.8 k8s-endpoint.home.lichnak.cz  k8s-endpoint

Bootstrap Multi-node Control Plane Kubernetes Cluster

[root@k8s-bastion ~]$ ssh k8s-master-01
Warning: Permanently added 'k8s-master-01' (ED25519) to the list of known hosts.
Last login: Fri Sep 24 18:07:55 2021 from 10.0.0.28

Update /etc/hosts file with this node IP address and a custom DNS name that maps to this IP:

[root@k8s-master-01 ~]$ vim /etc/hosts
10.0.0.29 k8s-endpoint.home.lichnak.cz  k8s-endpoint

To initialize the control-plane node run:

[root@k8s-master-01 ~]$ kubeadm init \
  --pod-network-cidr=192.168.0.0/16 \
  --control-plane-endpoint=k8s-endpoint.home.lichnak.cz \
  --cri-socket=/var/run/crio/crio.sock \
  --upload-certs

Where:

k8s-endpoint.home.lichnak.cz is a valid DNS name configured for ControlPlane Endpoint
/var/run/crio/crio.sock is Cri-o runtime socket file
192.168.0.0/16 is your Pod network to be used in Kubernetes
–upload-certs Flag used ti upload the certificates that should be shared across all the control-plane instances to the cluster

If successful you’ll get an output with contents similar to this:

...output omitted...
[mark-control-plane] Marking the node k8s-master-01.home.lichnak.cz as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: p11op9.eq9vr8gq9te195b9
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join k8s-endpoint.home.lichnak.cz:6443 --token 78oyk4.ds1hpo2vnwg3yykt \
 --discovery-token-ca-cert-hash sha256:4fbb0d45a1989cf63624736a005dc00ce6068eb7543ca4ae720c7b99a0e86aca \
 --control-plane --certificate-key 999110f4a07d3c430d19ca0019242f392e160216f3b91f421da1a91f1a863bba

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-endpoint.home.lichnak.cz:6443 --token 78oyk4.ds1hpo2vnwg3yykt \
 --discovery-token-ca-cert-hash sha256:4fbb0d45a1989cf63624736a005dc00ce6068eb7543ca4ae720c7b99a0e86aca

Configure Kubectl as shown in the output:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Test by checking active Nodes:

[root@k8s-master-01 ~]# kubectl get nodes
NAME                                  STATUS   ROLES                  AGE    VERSION
k8s-master-01.home.lichnak.cz             Ready    control-plane,master   4m3s   v1.22.2

Deploy Calico Pod network to the cluster

Download the Calico networking manifest for the Kubernetes API datastore.

curl https://docs.projectcalico.org/manifests/calico.yaml -O

Change CALICO_IPV4POOL_CIDR in calico.yaml to 10.1.0.0/16

vi calico.yml

Configure the kubernetes network

kubectl apply -f calico.yaml
su - ${KUBE_USER} -c "mkdir -p /home/${KUBE_USER}/.kube"
cp -au /etc/kubernetes/admin.conf /home/${KUBE_USER}/.kube/config
chown ${KUBE_USER}:${KUBE_USER} /home/${KUBE_USER}/.kube/config

Check pods status with the following command:

[root@k8s-master-01 ~]# kubectl get pods -n calico-system -w
NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-bdd5f97c5-6856t   1/1     Running   0          60s
calico-node-vnlkf                         1/1     Running   0          60s
calico-typha-5f857549c5-hkbwq             1/1     Running   0          60s

Add other control plane nodes

Add Master Node 02

[root@k8s-bastion ~]# ssh k8s-master-02
Warning: Permanently added 'k8s-master-02' (ED25519) to the list of known hosts.
Last login: Sat Sep 25 01:49:15 2021 from 10.0.0.28
[root@k8s-master-02 ~]#

Update /etc/hostsfile by setting the ControlPlaneEndpoint to first control node from where bootstrap process was initiated:

[root@k8s-master-02 ~]# vim /etc/hosts
10.0.0.29 k8s-endpoint.home.lichnak.cz   k8s-endpoint
#10.0.0.30 k8s-endpoint.home.lichnak.cz  k8s-endpoint
#10.0.0.31 k8s-endpoint.home.lichnak.cz  k8s-endpoint

I’ll use the command printed after a successful initialization:

kubeadm join k8s-endpoint.home.lichnak.cz:6443 --token 78oyk4.ds1hpo2vnwg3yykt \
  --discovery-token-ca-cert-hash sha256:4fbb0d45a1989cf63624736a005dc00ce6068eb7543ca4ae720c7b99a0e86aca \
  --control-plane --certificate-key 999110f4a07d3c430d19ca0019242f392e160216f3b91f421da1a91f1a863bba

Add Master Node 03

[root@k8s-bastion ~]# ssh k8s-master-03
Warning: Permanently added 'k8s-master-02' (ED25519) to the list of known hosts.
Last login: Sat Sep 25 01:55:11 2021 from 10.0.0.28
[root@k8s-master-02 ~]#

Update /etc/hostsfile by setting the ControlPlaneEndpoint to first control node from where bootstrap process was initiated:

[root@k8s-master-02 ~]# vim /etc/hosts
10.0.0.29 k8s-endpoint.home.lichnak.cz  k8s-endpoint
#10.0.0.30 k8s-endpoint.home.lichnak.cz  k8s-endpoint
#10.0.0.31 k8s-endpoint.home.lichnak.cz  k8s-endpoint

I’ll use the command printed after a successful initialization:

kubeadm join k8s-endpoint.home.lichnak.cz:6443 --token 78oyk4.ds1hpo2vnwg3yykt \
  --discovery-token-ca-cert-hash sha256:4fbb0d45a1989cf63624736a005dc00ce6068eb7543ca4ae720c7b99a0e86aca \
  --control-plane --certificate-key 999110f4a07d3c430d19ca0019242f392e160216f3b91f421da1a91f1a863bba

Check Control Plane Nodes List

From one of the master nodes with Kubectl configured check the list of nodes:

[root@k8s-master-03 ~]# kubectl get nodes
NAME                                  STATUS   ROLES                  AGE   VERSION
k8s-master-01.home.lichnak.cz             Ready    control-plane,master   11m   v1.22.2
k8s-master-02.home.lichnak.cz             Ready    control-plane,master   5m    v1.22.2
k8s-master-03.home.lichnak.cz             Ready    control-plane,master   32s   v1.22.2

You can now remove uncomment other lines in /etc/hosts file on each control node if not using Load Balancer IP:

# Perform on all control plane nodes
[root@k8s-master-03 ~]# vim /etc/hosts
####  Kubernetes cluster ControlPlaneEndpoint Entries ###
10.0.0.29 k8s-endpoint.home.lichnak.cz  k8s-endpoint
10.0.0.30 k8s-endpoint.home.lichnak.cz  k8s-endpoint
10.0.0.31 k8s-endpoint.home.lichnak.cz  k8s-endpoint

Step 5: Adding Worker Nodes to Kubernetes Cluster

#### Example ###
[root@k8s-bastion ~]# ssh root@k8s-worker-01
Warning: Permanently added 'k8s-worker-01' (ED25519) to the list of known hosts.
Enter passphrase for key '/root/.ssh/id_rsa':
Last login: Sat Sep 25 04:27:42 2021 from 192.168.200.9
[root@k8s-worker-01 ~]#

Update /etc/hosts file on each node with master and worker nodes hostnames/ip address if no DNS in place:

[root@k8s-worker-01 ~]# sudo vim /etc/hosts
#### Also add Kubernetes cluster ControlPlaneEndpoint Entries for multiple control plane nodes(masters) ###
10.0.0.29 k8s-endpoint.home.lichnak.cz  k8s-endpoint
10.0.0.30 k8s-endpoint.home.lichnak.cz  k8s-endpoint
10.0.0.31 k8s-endpoint.home.lichnak.cz  k8s-endpoint

Join your worker machines to the cluster using the commands given earlier:

kubeadm join k8s-endpoint.home.lichnak.cz:6443 \
  --token 78oyk4.ds1hpo2vnwg3yykt \
  --discovery-token-ca-cert-hash sha256:4fbb0d45a1989cf63624736a005dc00ce6068eb7543ca4ae720c7b99a0e86aca

Once done run kubectl get nodes on the control-plane to see the nodes join the cluster:

[root@k8s-master-02 ~]# kubectl get nodes
NAME                                  STATUS   ROLES                  AGE     VERSION
k8s-master-01.home.lichnak.cz             Ready    control-plane,master   23m     v1.22.2
k8s-master-02.home.lichnak.cz             Ready    control-plane,master   16m     v1.22.2
k8s-master-03.home.lichnak.cz             Ready    control-plane,master   12m     v1.22.2
k8s-worker-01.home.lichnak.cz             Ready    <none>                 3m25s   v1.22.2
k8s-worker-02.home.lichnak.cz             Ready    <none>                 2m53s   v1.22.2
k8s-worker-03.home.lichnak.cz             Ready    <none>                 2m31s   v1.22.2
k8s-worker-04.home.lichnak.cz             Ready    <none>                 2m12s   v1.22.2

Configuration

# Configuration
# matallb
# ingress traefik or nginx
# certmanager
# istio
# Hint: mate 3 mastery a 5 workeru. calico a k8s rozdeluje ve vychozim stavu subnetu 10.1.0.0/16 na pod cidr subnety po metrice /24, cili nodum prideli pro pody /24 blok site z 10.1.0.0/16 je prvni 10.1.0.0/24, dalsi 10.1.1.0/24, ard. cili delka pod subnety musi obsahovat minimalne 8 /24 bloku site. Blok 10.1.0.0/16 je dost velky pro 8 nodu.
# ref. https://www.tigera.io/blog/designing-on-prem-kubernetes-networks-for-high-availability/

sudo rpm -Uvh http://repo.zabbix.com/zabbix/6.0/rhel/8/x86_64/zabbix-release-6.0-1.el8.noarch.rpm
sudo dnf install -y zabbix-agent
sudo cp /etc/zabbix/zabbix_agentd.conf /etc/zabbix/zabbix_agentd.conf.orig
sudo cat > /etc/zabbix/zabbix_agentd.conf <<"EOF"
PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=0
# SourceIP=0.0.0.0
Server=10.0.0.20
# ListenPort=10050
# ListenIP=0.0.0.0
ServerActive=10.0.0.20
Hostname=k8s-worker-01.home.lichnak.cz
# HostnameItem=system.hostname
Include=/etc/zabbix/zabbix_agentd.d/*.conf
# Include=/usr/local/etc/zabbix_agentd.userparams.conf
# Include=/usr/local/etc/zabbix_agentd.conf.d/
# Include=/usr/local/etc/zabbix_agentd.conf.d/*.conf
# LoadModulePath=${libdir}/modules
# LoadModule=
EOF
sudo systemctl enable zabbix-agent
sudo systemctl start zabbix-agent
sudo systemctl status zabbix-agent
sudo firewall-cmd --permanent --add-port=10050/tcp --zone=public
sudo firewall-cmd --reload

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py --user
python3 -m pip install ansible awscli boto3 --user

git clone git@git.lichnak.cz:infra/k8s-pre-bootstrap.git
cd k8s-pre-bootstrap/
ansible-playbook -u lichnak -i hosts k8s-prep.yml --ask-become -ask-become--pass


ssh 10.0.0.28
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py --user
python3 -m pip install ansible awscli boto3 --user
git clone git@git.lichnak.cz:infra/k8s-pre-bootstrap.git
cd k8s-pre-bootstrap/
ansible-playbook -u lichnak -i hosts k8s-prep.yml --ask-become -ask-become--pass

ssh 10.0.0.29
export KUBE_USER=kube
sudo useradd -m -s /bin/bash ${KUBE_USER} || echo "ERR: Adding user failed, already exists?"
sudo usermod -aG wheel ${KUBE_USER} || echo "ERR: Adding user to sudo group failed, already added?"

sudo su - ${KUBE_USER} -c "echo 'source <(kubectl completion bash)' >>~/.bashrc"
sudo su - ${KUBE_USER} -c "echo 'source <(helm completion bash)' >>~/.bashrc"
sudo su - ${KUBE_USER} -c "echo 'alias k=\"kubectl --namespace default\"' >>~/.bashrc"
sudo su - ${KUBE_USER} -c "echo 'complete -F __start_kubectl k' >>~/.bashrc"

sudo kubeadm init \
  --pod-network-cidr=10.1.0.0/16 \
  --service-cidr=10.2.0.0/16 \
  --service-dns-domain="k8s.home.lichnak.cz" \
  --apiserver-cert-extra-sans="localhost,127.0.0.1,k8s-endpoint,10.0.0.29,10.0.0.30,10.0.0.31,*.k8s.home.lichnak.cz,*.f13cybertech.net,*.home.lichnak.cz" \
  --apiserver-advertise-address=0.0.0.0 \
  --control-plane-endpoint="k8s-endpoint.home.lichnak.cz" \
  --cri-socket=/var/run/crio/crio.sock \
  --dry-run
  
# ~~~
# note: --upload-certs plati pouze 2h
# ~~~

sudo kubeadm init \
  --pod-network-cidr=10.1.0.0/16 \
  --service-cidr=10.2.0.0/16 \
  --service-dns-domain="k8s.home.lichnak.cz" \
  --apiserver-cert-extra-sans="localhost,127.0.0.1,k8s-endpoint,10.0.0.29,10.0.0.30,10.0.0.31,*.k8s.home.lichnak.cz,*.f13cybertech.net,*.home.lichnak.cz" \
  --apiserver-advertise-address=0.0.0.0 \
  --control-plane-endpoint="k8s-endpoint.home.lichnak.cz" \
  --cri-socket=/var/run/crio/crio.sock \
  --upload-certs

sudo systemctl enable kubelet

sudo su - ${KUBE_USER} -c "mkdir -p /home/${KUBE_USER}/.kube"
sudo cp -au /etc/kubernetes/admin.conf /home/${KUBE_USER}/.kube/config
sudo chown ${KUBE_USER}:${KUBE_USER} /home/${KUBE_USER}/.kube/config


#master
sudo kubeadm join k8s-endpoint.home.lichnak.cz:6443 --token r2pz2g.dlz7f4q1wa3hung3 \
--discovery-token-ca-cert-hash sha256:b30b2e6973dffc0ccc6bb84430949c63e49e8af116819b8817dac3756faa4563 \
--control-plane --certificate-key 155126b2097dccebd21bcfb7bc2262e302ef1b64ff3e2ed28116fdccf401484f

#worker
sudo kubeadm join k8s-endpoint.home.lichnak.cz:6443 --token r2pz2g.dlz7f4q1wa3hung3 \
--discovery-token-ca-cert-hash sha256:b30b2e6973dffc0ccc6bb84430949c63e49e8af116819b8817dac3756faa4563

curl https://docs.projectcalico.org/manifests/calico.yaml -O
# edit CALICO_IPV4POOL_CIDR https://docs.projectcalico.org/manifests/calico.yaml

vi calico.yml
# ~~~
#             # The default IPv4 pool to create on startup if none exists. Pod IPs will be
#             # chosen from this range. Changing this value after installation will have
#             # no effect. This should fall within `--cluster-cidr`.
#              - name: CALICO_IPV4POOL_CIDR
#                value: "10.1.0.0/16"
# ~~~

kubectl apply -f calico.yaml

ssh 10.0.0.30
kubeadm join k8s-endpoint.home.lichnak.cz:6443 --token 78 \
  --discovery-token-ca-cert-hash sha256:4f \
  --control-plane --certificate-key 99

# k8s-wokers
ssh 10.0.0.32-35
kubeadm join k8s-endpoint.home.lichnak.cz:6443 --token 78 \
  --discovery-token-ca-cert-hash sha256:4f

sysctl net.ipv4.conf.eth0.rp_filter=0
sysctl net.ipv4.conf.eth1.rp_filter=0
sysctl net.ipv4.conf.gre0.rp_filter=0
sysctl net.ipv4.conf.gre1.rp_filter=0

wget https://get.helm.sh/helm-v3.6.0-linux-amd64.tar.gz
tar xvf helm-v3.6.0-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin
rm helm-v3.6.0-linux-amd64.tar.gz 
rm -rf linux-amd64
helm version

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

export KUBE_USER=kube
sudo useradd -m -s /bin/bash ${KUBE_USER} || echo "ERR: Adding user failed, already exists?"
sudo usermod -aG wheel ${KUBE_USER} || echo "ERR: Adding user to sudo group failed, already added?"

sudo su - ${KUBE_USER} -c "echo 'source <(kubectl completion bash)' >>~/.bashrc"
sudo su - ${KUBE_USER} -c "echo 'source <(helm completion bash)' >>~/.bashrc"
sudo su - ${KUBE_USER} -c "echo 'alias k=\"kubectl --namespace default\"' >>~/.bashrc"
sudo su - ${KUBE_USER} -c "echo 'complete -F __start_kubectl k' >>~/.bashrc"

kubeadm init \
  --pod-network-cidr=10.1.0.0/16 \
  --service-cidr=10.2.0.0/16 \
  --service-dns-domain="k8s.home.lichnak.cz" \
  --apiserver-cert-extra-sans="localhost,127.0.0.1,k8s-endpoint,10.0.0.29,10.0.0.30,10.0.0.31,*.k8s.home.lichnak.cz,*.f13cybertech.net,*.home.lichnak.cz" \
  --apiserver-advertise-address=0.0.0.0 \
  --control-plane-endpoint="k8s-endpoint.home.lichnak.cz" \
  --cri-socket=/var/run/crio/crio.sock \
  --dry-run
  
# ~~~
# note: --upload-certs plati pouze 2h
# ~~~

kubeadm init \
  --pod-network-cidr=10.1.0.0/16 \
  --service-cidr=10.2.0.0/16 \
  --service-dns-domain="k8s.home.lichnak.cz" \
  --apiserver-cert-extra-sans="localhost,127.0.0.1,k8s-endpoint,10.0.0.29,10.0.0.30,10.0.0.31,*.k8s.home.lichnak.cz,*.f13cybertech.net,*.home.lichnak.cz" \
  --apiserver-advertise-address=0.0.0.0 \
  --control-plane-endpoint="k8s-endpoint.home.lichnak.cz" \
  --cri-socket=/var/run/crio/crio.sock \
  --upload-certs

curl https://docs.projectcalico.org/manifests/calico.yaml -O
# edit CALICO_IPV4POOL_CIDR https://docs.projectcalico.org/manifests/calico.yaml

vi calico.yaml
# ~~~
#             # The default IPv4 pool to create on startup if none exists. Pod IPs will be
#             # chosen from this range. Changing this value after installation will have
#             # no effect. This should fall within `--cluster-cidr`.
#              - name: CALICO_IPV4POOL_CIDR
#                value: "10.1.0.0/16"
# ~~~

kubectl apply -f calico.yaml

# https://kubernetes.github.io/ingress-nginx/deploy/baremetal/

kubectl edit configmap -n kube-system kube-proxy
~~~
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
  strictARP: true
~~~
# see what changes would be made, returns nonzero returncode if different
kubectl get configmap kube-proxy -n kube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
kubectl diff -f - -n kube-system

# actually apply the changes, returns nonzero returncode on errors only
kubectl get configmap kube-proxy -n kube-system -o yaml | \
  sed -e "s/strictARP: false/strictARP: true/" | \
  kubectl apply -f - -n kube-system
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml
cat > matallb-configmp.yml <<"EOF"
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.0.0.51-10.0.0.62
EOF
kubectl apply -f matallb-configmap.yml
kubectl apply -f https://github.com/kubernetes/ingress-nginx/blob/main/deploy/static/provider/baremetal/deploy.yaml


kubectl get pods --namespace=ingress-nginx
kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=120s
kubectl create deployment demo --image=httpd --port=80
kubectl expose deployment demo
cat > demo-ingress.yaml <<"EOF"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
    - host: demo.home.lichnak.cz
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: demo
                port:
                  number: 80
EOF
kubectl apply -f demo-ingress.yaml

kubectl create ingress demo-localhost --class=nginx \
  --rule=demo.home.lichnak.cz/*=demo:80
kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8080:80

Backup and restore

# Backups

# Restore

From Zero to Hero

My name is Adam Lichonvsky and I'm proud father and researcher.

Kubernetes

Installation

List of servers

Virtual Machines Specs

Memory

Disk space

CPU Cores

DNS Settings

Step 1: Prepare Bastion Server for Kubernetes installation

Install basic tools

Install Ansible configuration management

Installing Ansible using pip

Check Ansible version

Update /etc/hosts file in your Bastion machine

Generate SSH keys

Create SSH client configuration file

Copy SSH keys to all Kubernetes cluster nodes

Step 2: Set correct hostname on all nodes

Login to each node in the cluster and configure correct hostname

Logout then back in to confirm the hostname is set correctly

Step 3: Prepare Rocky Linux 8 servers for Kubernetes (Pre-reqs setup)

Step 4: Bootstrap Kubernetes Control Plane (Single / Multi-node)

Option 1: Bootstrapping single node Control Plane Kubernetes Cluster

Deploy Calico Pod network to the cluster fist master

Option 2: Bootstrapping Multi-node Control Plane Kubernetes Cluster

Using Load Balancer IP for ControlPlaneEndpoint

Bootstrap Multi-node Control Plane Kubernetes Cluster

Deploy Calico Pod network to the cluster

Add other control plane nodes

Add Master Node 02

Add Master Node 03

Check Control Plane Nodes List

Step 5: Adding Worker Nodes to Kubernetes Cluster

Configuration

Backup and restore

From Zero to Hero

Recent Posts