You may find my language uncharacteristically terse in this post, this is due to the large scope of topics we'll touch on. I consider any Kubernetes work: implementation, development or administration complex resulting from the number of technical competencies required, as outlined below.

Prerequisites:

• At least two nodes (computers, SBCs or similar)
  • Each with network connectivity (wired/wireless)
• Fundamental Linux OS knowledge
• Understanding of basic networking concepts (IPs, subnets, CIDR, etc...)
• Shell scripting proficiency
• 2GB+ Micro SD Card

This process has been adapted to my specific Raspberry-Pi clone (Le Potato) model, and while many of the steps are device/OS agnostic, a sizeable number are not.

Since the actions we are going to perform must be applied to each device intended for PI-cluster assimilation, I created the following shell script to automate that effort. Script linked below.

Config Kubernetes (SBCs)

Let's begin by dissecting this script. It contains all configurations required to implement a Kubernetes cluster.

CGROUP_DATA='GRUB_CMDLINE_LINUX="cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1"'

if grep -Fxq "$CGROUP_DATA" /etc/default/grub
then
    # aggressively disable swap file, it's a persistent little fucker
    sudo apt install -y dphys-swapfile
    sudo dphys-swapfile swapoff
    sudo swapoff /swapfile
    sudo dphys-swapfile uninstall
    update-rc.d dphys-swapfile remove
    sudo rm -f /etc/init.d/dphys-swapfile
    sudo service dphys-swapfile stop
    sudo systemctl disable dphys-swapfile.service
    sudo sed -i 's@/swapfile    none    swap    defaults        0       0@#/swapfile    none    swap    defaults        0       0@g' /etc/fstab


    #verify swap is disable successfully
    FREE_SWAP=$(free | grep Swap | awk '{ print $2$3$4}')
    if [[ "$FREE_SWAP" != "000" ]]
    then
        echo "Swap was not successfully disabled. Investigate why. Exiting script."
        exit 1
    fi

While missing the other conditions if the if statement it is used to ensure that certain steps are performed on the first execution of the script (enabling systems resources for use inside of cgroups). That is because distro updates performed may require a reboot, specifically kernel updates. When re-running the script (2nd pass) it will recognize modifications were made to a config file validating that first time script execution was already successfully performed.

The block contained in the conditional statement shown below, is a brutish attempt to remove any trace of a swapfile. Exact commands will vary across distro releases but for any recent Ubuntu Server image swap should be erradicated appropriately.

Lastly in the bottom-most section is a small validation confirming whether swap was disabled successfully.

Now, section two handles the installation and enablement of essential Kubernetes dependencies via K8 configs files.

    # Installs containerd (Container Runtime) & additional networking features
    sudo apt install -y containerd containernetworking-plugins

    if test -d /etc/containerd/
        then
            echo "containerd directory exists."
        else
            mkdir /etc/containerd
    fi
        cat <<-EOF | sudo tee /etc/containerd/config.toml
        version = 2
        [plugins]
        [plugins."io.containerd.grpc.v1.cri"]
            [plugins."io.containerd.grpc.v1.cri".containerd]
            [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
                [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
                runtime_type = "io.containerd.runc.v2"
                [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
                    SystemdCgroup = true
EOF

    cat <<-EOF | sudo tee /etc/modules-load.d/k8s.conf
    overlay
    br_netfilter
EOF


    cat <<-EOF | sudo tee /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-iptables  = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    net.ipv4.ip_forward                 = 1
EOF

    cat <<-EOF | sudo tee /etc/modules-load.d/k8s.conf
    overlay
    br_netfilter
EOF

Package installation commands (via Apt) are simple enough. We then ensure a proper directory structure exists and output our heredoc of settings/values to a file. We expect Containerd (Container Daemon) to handle start/stop-ing & management of our containers, so we want to explicitly define that selection. We also allow usage of cgroups. cgroups.

The remaining config definitions relate to packet routing as summarized below.

Kernel IP forwarding

IP forwarding is a kernel setting that allows forwarding of the traffic coming from one interface to be routed to another interface. This setting is necessary for Linux kernel to route traffic from containers to the outside world."

net.bridge.bridge-nf-call values control whether or not packets traversing the bridge are sent to iptables for processing. In the case of using bridges to connect virtual machines to the network, generally such processing is not desired, as it results in guest traffic being blocked due to host iptables rules that only account for the host itself, and not for the guests."

Our modprobe lines simply add the specified module to the kernel and then enables their functionality with sysctl.

Hurray, we've now reached our actual Kubernetes installation proceedings.

    sudo apt install -y apt-transport-https ca-certificates curl
    sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
    echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
    curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
    sudo apt update && sudo apt install -y kubelet kubeadm kubectl && sudo apt-mark hold kubelet kubeadm kubectl 
    # apt-mark hold prevents this package from being auto-updated accidentally.


    # How flannel network implementation works - https://blog.laputa.io/kubernetes-flannel-networking-6a1cb1f8ec7c
    cd ~
    wget https://github.com/flannel-io/flannel/releases/download/v0.19.2/flanneld-arm64
    sudo chmod +x flanneld-arm64
    sudo cp flanneld-arm64 /usr/local/bin/flanneld
    sudo mkdir -p /var/lib/k8s/flannel/networks

    echo "All pre-configuration have been completed. Proceed by initializing your Kubernetes Cluster 

All code prior to K8 package installs are performed to securely add Google's package repos. Additional information regarding gpg-based securement of repo lists can be found here.

Using gpg to Add External Repositories

In relation to the Flannel section, we're only downloading it for now, but I want to review at a high level what services it provides and later we'll review our exact usage of it.

A couple things to note:

• Kubenetes does not come pre-packaged with any networking modules/packages! Instead of providing a built-in networking solution, Kubernetes defines a set of networking requirements and interfaces, allowing us to choose and integrate a networking solution that best fits our needs
• Four kubernetes networking requirements/tenets
  1. Pods need to be uniquely addressable to intercommunicate within a cluster, this is solved with IP address allocation (pretty typical way to solve)
  2. Pods within the same node (meaning host/server) should be able to communicate directly without (NAT) -- would otherwise be a LOT of additional compute resource overhead and complexity.
  3. Pod to service communication is needed. This is meant to abstract away targeting single pods by their IP and instead have a single stable endpoint for addressing a set of pods (load-balancing)
  4. cluster to public communication. Pods should be able to communicate with outside resources, like an external API or database. This is typically handled by through a network gateway or similar to pass traffic outward to external networks.

Here is how Flannel accomplishes these requirements specifically.

More details on this in my next update of this post.

Continued... WIP