Since I’ve gone through the rather step-filled process of connecting a Kubernetes cluster to a CI/CD pipeline a number of times in the last few months, I wanted to document it here and also talk a bit about options to potentially save time in this process.
Discussion
In general in the Kubernetes world, to achieve continuous deployment from a CI/CD pipeline such as GitHub Actions or GitLab Pipelines, there is a very specific and repetitive process that you need to go through to set everything up. This entails setting up Service Accounts on your cluster, setting up credentials to pull your images, then encoding and adding the key for the SA to your repository secrets/variables. My purpose in this blog post is to firstly document that process using an example blog repository called example-blog. After the nitty-gritty of that is complete, I’ll talk a bit about other options to automate this process, which admittedly, is the least fun of the entire development process.
Here’s the hypothetical setup:
example-blog.com, namespace example-blog. The developer makes a commit to the main branch of the repo example-blog, and a pipeline builds a Docker image called example-blog:latest, uploads it to Google Artifact Registry, which we must then set as the current image for a deployment that we’ve previously set up called example-blog. We need an imagePullSecrets as well, so I’ll document how to set that up.
So all in all, here’s a list of every piece that we need:
- A Kubernetes Service Account with proper permissions to update the deployment, and a token (This is actually 4 separate pieces, ServiceAccount, Role, RoleBinding, and Secret). Needs to be base64 encoded to be able to be used as a Repository Secret.
- A GCP Service Account with permissions to read/write to Artifact Registry. This also needs to be base64 encoded.
- A pipeline that references the correct repository secrets that we set up previously
Step 1. Create Kubernetes Service Account
cat << EOF | kubectl apply -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: github-actions-deployer
namespace: example-blog
---
apiVersion: v1
kind: Secret
metadata:
name: github-actions-deployer-secret
namespace: example-blog
annotations:
kubernetes.io/service-account.name: github-actions-deployer
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: example-blog
name: deployment-updater
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: github-actions-deployer-binding
namespace: example-blog
subjects:
- kind: ServiceAccount
name: github-actions-deployer
namespace: example-blog
roleRef:
kind: Role
name: deployment-updater
apiGroup: rbac.authorization.k8s.io
EOF
Step 2. Combine the token, CA certs, and API endpoint into a base64 encoded kubeconfig
SERVICE_ACCOUNT_NAME=github-actions-deployer
NAMESPACE=example-blog
SECRET_NAME=$(kubectl get sa $SERVICE_ACCOUNT_NAME -n $NAMESPACE -o jsonpath="{.secrets[0].name}")
# Extract the token, CA cert, and server endpoint
TOKEN=$(kubectl get secret $SECRET_NAME -n $NAMESPACE -o jsonpath="{.data.token}" | base64 --decode)
CA_CERT=$(kubectl get secret $SECRET_NAME -n $NAMESPACE -o jsonpath="{.data['ca\.crt']}" | base64 --decode)
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
# Create the kubeconfig file
cat <<EOF > kubeconfig
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority-data: $(echo $CA_CERT | base64 | tr -d '\n')
server: $SERVER
name: kubernetes
contexts:
- context:
cluster: kubernetes
namespace: $NAMESPACE
user: $SERVICE_ACCOUNT_NAME
name: $SERVICE_ACCOUNT_NAME
current-context: $SERVICE_ACCOUNT_NAME
users:
- name: $SERVICE_ACCOUNT_NAME
user:
token: $TOKEN
EOF
base64 kubeconfig > kubeconfig.base64
You can now take the contents of kubeconfig.base64 and add it as a repository secret called KUBE_CONFIG
Step 3. Create Service Account with Artifact Registry read/write in Cloud Provider
Out of scope for this doc but pretty simple. Once you’ve created the Service Account with your Cloud Provider, you need to create a Kubernetes secret from the JSON file (this is sensitive data, ensure you take precautions) and also add it as a repository secret.
Step 4. Create Kubernetes Secret from Artifact Registry SA you created
kubectl create secret generic example-blog --from-file=.dockerconfigjson=docker-registry.json --type=kubernetes.io/dockerconfigjson
And reference the ImagePullSecret in your deployment:
imagePullSecrets:
- name: example-blog
Now you can base64encode the JSON and add it as a repository secret called GAR_SA.
End of manual process
# Here's a VLE (very low effort) ASCII CAT for a segue
.----------------------------------------------.
| |
| |
| \ \ F Cisco |
| /./././. | | i Advanced |
| / `/. | | r Terminal |
| / __ `/'/' e |
| /\\__/\\ /' `\\ / w |
| | 00 | `.,.| a |
| \\Vvvv/ |||| l |
| |||| |||| l |
| |||| |||| |
| `'`' `'`' |
.----------------------------------------------.
Automation
So now that we’ve got the manual process out of the way, let’s explore some options to automate all of that with Continuous Deployment specific tooling. OTOH, here are the options available today, but I’m sure there are many more and I’m sure many more are to come. In no particular order, they are:
- GitLab Agent: Great for GitLab users, uses GitOps workflow, built in monitoring/observability
- Flux: Helm integration, GitOps, extensible
- ArgoCD: Declarative GitOps from a repo, nice UI, multi-cluster, cool name
Keep in mind that all 3 of these options require setting up deployments on your cluster, which will consume additional resources.