I recently embarked down the rabbit hole of using kubeadm to bootstrap a Kubernetes cluster on GCP, and found that the documentation wasn’t quite complete, so I ended up pulling documentation from several different sources to get a fully functional cluster (and also ended up with a revelation at the end).
One big caveat of the official kubernetes documentation is that it doesn’t include a cloud controller, which means there is no interconnect between your cloud provider of choice and the cluster. I’ll go more into detail about that as we go along in this post, but I will say this: there’s a TLDR at the end.
So firstly, as I mentioned, let’s document what’s needed to get the cluster going with kubeadm. This assumes you have a controlplane node and a worker node. This documentation is aimed at Debian/Ubuntu only, although I will say that I used the official documentation on a Redhat based cluster and it actually worked as is.
Step 1
The official documentation says you should install a container runtime, set up cgroup drivers, and and set up ip4 forwarding before installing kubeadm/kubelet. It doesn’t really show you much that’s useful as far as commands go for setting up the cgroup drivers, and you would eventually end up with your kube-proxy in CrashLoopBackoff state if you followed those instructions. Let’s document the commands concisely:
Set up overlay kernel modules which you will need for your network addon:
sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
Set up ip4/ip6 forwarding:
sudo tee /etc/sysctl.d/kubernetes.conf <<EOT
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOT
sudo sysctl --system
Install containerd runtime, configure containerd, and set up the cgroup drivers to use systemd (last two commands were missing from official documentation):
sudo apt update
sudo apt install -y containerd.io
sudo systemctl enable containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
Step 2
These next commands will update the system and kernel, install necessary programs, update apt sources.list with the official Kubernetes repo, and finally install kubeadm, kubelet, kubectl, and kubeadm:
sudo apt-get -y dist-upgrade
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl enable --now kubelet
# apt-get dist-upgrade would have most likely upgraded your kernel, let's go ahead and reboot here, if needed
[ -f /var/run/reboot-required ] && sudo reboot
Step 3
Okay, now that you’ve done all that, you should have a controlplane and worker node ready for the kudeadm init process. From here on out, the official documentation can be used.
kubeadm init --pod-network-cidr 192.168.0.0/16
# Set up your kubeconfig
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# Set up kubectl completion only for your user
echo 'source <(kubectl completion bash)' >>~/.bashrc
echo 'alias k=kubectl' >>~/.bashrc
echo 'complete -o default -F __start_kubectl k' >>~/.bashrc
source ~/.bashrc
On your worker node, you can now enter in the kubeadm join command that was printed:
kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>
We’re getting close now. The last thing we need to have a working cluster is to install a network addon. You can pick a network addon here, but I’ve used Flannel, Weave, Calico, and Cilium without issue. Let’s go with Cilium. Run this on your control plane:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.10/install/kubernetes/quick-install.yaml
One thing I want to document here, by default, Cilium replaces kube-proxy, but its behavior is different from kube-proxy in regards to routing to ClusterIPs. The issue is detailed here, and the fix is here
TLDR #1:
apiVersion: cilium.io/v2alpha1
kind: CiliumNodeConfig
metadata:
namespace: kube-system
name: bpf-lb-sock-hostns-only
spec:
nodeSelector:
matchLabels:
kubernetes.io/hostname: abc.def.xyz
defaults:
bpf-lb-sock-hostns-only: "true"
End
So, at this point, you should have a fully functioning cluster, with one caveat: By default, Kubernetes ships with an internal controller that doesn’t interface with your cloud provider. That means if you’re running on AWS, GCP, Azure (and others), you’ll need to either provision storage, IPs, and load balancers manually, or install a cloud controller. Since I was running on GCP, I began looking in to what was required to do this. It quickly became apparent that development and support of their cloud controller has been sunsetted in order to get people to lean towards GKE. I tore my new cluster down several times and rebuilt it in efforts to get the Google Cloud Controller Manager working.
Here’s where the revelation came in. Programs like kOps and kubespray do all of the above for you, (plus much more), while still offering the same flexiblity that you were looking to achieve by manually installing Kubernetes in the first place.
So, there’s where TLDR #2 comes in. Don’t do any of the above. Use kOps.