Kubernetes on dev@iostream.pl

Jenkins, Part Two: Disposable Agents and Builder Images

Sun, 12 Jul 2026 17:20:00 +0200

Once again: why all these containers?

The Kubernetes approach — short-lived, disposable things — works beautifully for Jenkins and similar CI/CD systems.

Here is a list of tasks to execute in a place described by a specification, not by a physical location. Whether that place existed five minutes ago or will disappear five minutes from now does not matter all that much.

Look at how the approach to running CI/CD jobs has evolved over the last twenty years:

run the job in a blocking shell script,
run the job directly on the Jenkins master,
run the job on a dedicated Jenkins node,
run the job on one of several dedicated Jenkins nodes.

And now?

run the job inside an ephemeral Pod, created just before the work starts and removed when the work is done.

Hard not to admire that, is it?

And there is more to admire. Each container can provide a different set of tools. Instead of installing everything manually on physical machines, you can point Jenkins at a ready-made image, built by someone else — and if you do not like that one, you can use another.

All right, that is enough admiration for now.

What do we already have?

Quite a lot:

Jenkins running under Kubernetes,
a working container registry,
a PVC for Jenkins data,
a Kubernetes cloud definition in Jenkins,
two already built images that we are about to use.

What else do we need?

Eventually, Jenkins will run jobs such as:

building Python applications,
building Go applications,
building Hugo sites.

So we need three images.

Python builder

The Python image can be built from the following Dockerfile:

FROM registry.lab.local/base-builder:0.1

USER root

RUN apt-get update && apt-get install -y --no-install-recommends \
 python3 \
 python3-pip \
 python3-venv \
 python3-build \
 && rm -rf /var/lib/apt/lists/*

RUN python3 -m pip install --break-system-packages \
 build \
 twine \
 wheel \
 setuptools \
 pytest \
 ruff

USER builder
WORKDIR /workspace

CMD ["sleep", "infinity"]

Wait a moment. Do not reach for docker build yet.

Actually, do not reach for it at all.

Put this Dockerfile into a source repository, preferably in a dedicated subdirectory. Later it will be much easier to tell what kind of creatures are being produced from it.

We are about to enable a tiny bit of magic. If you are not ready, perhaps take a short walk.

Jenkins can wait. That is, after all, usually its main job.

Create a Jenkinsfile with the following contents:

pipeline {
 agent {
 kubernetes {
 yaml '''
apiVersion: v1
kind: Pod
spec:
 hostAliases:
 - ip: "10.10.10.24"
 hostnames:
 - "registry.lab.local"
 containers:
 - name: base
 image: registry.lab.local/base-builder:0.1
 command: ["sleep"]
 args: ["infinity"]
 - name: kaniko
 image: registry.lab.local/kaniko-builder:latest
 command: ["sleep"]
 args: ["infinity"]
 volumeMounts:
 - name: docker-config
 mountPath: /kaniko/.docker
 volumes:
 - name: docker-config
 configMap:
 name: kaniko-docker-config
'''
 }
 }

 environment {
 IMAGE_NAME = 'registry.lab.local/python-builder'
 IMAGE_TAG = '3.11.1'
 }

 stages {
 stage('Prepare') {
 steps {
 container('base') {
 script {
 env.GIT_SHA = sh(
 script: 'git rev-parse --short HEAD',
 returnStdout: true
 ).trim()

 currentBuild.displayName = "#${env.BUILD_NUMBER} python-3"
 currentBuild.description = "python-builder:3, commit ${env.GIT_SHA}"
 }

 sh '''
 echo "IMAGE=${IMAGE_NAME}:${IMAGE_TAG}"
 ls -la
 '''
 }
 }
 }

 stage('Build and push image') {
 steps {
 container('kaniko') {
 sh '''
 /kaniko/executor \
 --context "${WORKSPACE}" \
 --dockerfile "${WORKSPACE}/python-builder/Dockerfile" \
 --destination "${IMAGE_NAME}:${IMAGE_TAG}" \
 --insecure \
 --skip-tls-verify \
 --insecure-registry registry.lab.local
 '''
 }
 }
 }
 }

 post {
 success {
 echo "Built and pushed ${IMAGE_NAME}:${IMAGE_TAG}"
 }
 }
}

Commit it to the same repository, in the same subdirectory as the Dockerfile.

The cluster needs one additional ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
 name: kaniko-docker-config
 namespace: jenkins
data:
 config.json: |
 {
 "auths": {}
 }

The Jenkinsfile refers to this ConfigMap in the agent section.

Now create a new Multibranch Pipeline job. Give it a sensible name, for example python-builder, and configure it:

enter the repository URL,
if credentials are required, select them, or add them first,
under Build configuration → Mode, make sure by Jenkinsfile is selected,
in Script Path, enter the path to the file, for example python-builder/Jenkinsfile.

Save the job and wait a moment.

Jenkins will start scanning branches in your repository, looking for the specified Jenkinsfile in each of them. If it finds one, it will try to run the job for that branch.

Once the Stage View starts rendering, you can observe what is happening in the jenkins namespace:

kubectl get pods -n jenkins -w

A new Pod will appear, do its job, and leave, as in the example below:

NAME READY STATUS RESTARTS AGE
jenkins-0 1/1 Running 12 (57m ago) 19d
python-builder-main-1-30xvv-l1cp7-vcw9m 0/3 Pending 0 0s
python-builder-main-1-30xvv-l1cp7-vcw9m 0/3 ContainerCreating 0 0s
python-builder-main-1-30xvv-l1cp7-vcw9m 3/3 Running 0 1s
python-builder-main-1-30xvv-l1cp7-vcw9m 3/3 Terminating 0 63s
python-builder-main-1-30xvv-l1cp7-vcw9m 0/3 Error 0 93s

The job completed successfully and the python-builder image landed in the registry.

You can verify that either through registry-ui or directly:

curl http://registry.lab.local/v2/python-builder/tags/list

The response should look like this:

{"name":"python-builder","tags":["3.11.1"]}

The image is now ready to use.

What just happened?

The contents of the Jenkinsfile definitely deserve an explanation, so let us start from the beginning.

`agent`

The agent section is without a doubt the most interesting part of this file.

Besides the type name, kubernetes, it contains a full Kubernetes Pod definition. In the specification we can see that:

hostAliases are used, because we do not have proper DNS here and still need to smuggle the registry address into the Pod,
all important containers are listed:
- base, which provides Git and basic shell tools,
- kaniko, which provides… well, kaniko.

Keeping the names consistent is important.

If you name the containers something like the_one_with_git and the_one_with_kaniko, you will have to use exactly those names later in the actual pipeline steps:

stages {
 stage('Prepare') {
 steps {
 container('the_one_with_git') {

In short: Jenkins creates an agent described in the agent section, and pipeline steps are executed inside specific containers from that Pod — the ones that provide the required tools.

This is the important part.

We are not installing build tools inside the main Jenkins server. The server only orchestrates work. The tools live in disposable containers, created only for the job that needs them.

That keeps the Jenkins controller cleaner, makes builds easier to reproduce, and avoids the traditional CI server problem where one machine slowly turns into a museum of forgotten compilers, package managers, SDKs, symlinks and emotional damage.

`kaniko`

Remember that kaniko is not Docker.

Its job is simply to build a container image and push it to a registry.

The important parameters are:

--context — the working directory, usually the job workspace,
--dockerfile — the location of the Dockerfile, in this case inside a subdirectory,
--destination — the registry location, image name and tag to push after the build.

Simple? Simple.

“But wait!” you might shout. “What about that mysterious kaniko-docker-config ConfigMap?”

Here it is.

kaniko likes to look inside /kaniko/.docker, mainly for a config.json file containing credentials, if they are required.

Our registry does not require authentication, but we still mount a ConfigMap there and provide an empty auths object:

{
 "auths": {}
}

This allows kaniko to try pushing the image anonymously right away.

Go builder

Not enough examples? Fine. Let us create a go-builder.

The Dockerfile could look like this:

FROM golang:1.26-bookworm

RUN apt-get update && apt-get install -y --no-install-recommends \
 git \
 openssh-client \
 ca-certificates \
 curl \
 wget \
 jq \
 make \
 rsync \
 && rm -rf /var/lib/apt/lists/*

RUN update-ca-certificates

USER root

WORKDIR /workspace

RUN go version

CMD ["sleep", "infinity"]

And the Jenkinsfile could look like this:

pipeline {
 agent {
 kubernetes {
 yaml '''
apiVersion: v1
kind: Pod
spec:
 hostAliases:
 - ip: "10.10.10.24"
 hostnames:
 - "registry.lab.local"
 containers:
 - name: base
 image: registry.lab.local/base-builder:0.1
 command: ["sleep"]
 args: ["infinity"]
 - name: kaniko
 image: registry.lab.local/kaniko-builder:latest
 command: ["sleep"]
 args: ["infinity"]
 volumeMounts:
 - name: docker-config
 mountPath: /kaniko/.docker
 volumes:
 - name: docker-config
 configMap:
 name: kaniko-docker-config
'''
 }
 }

 environment {
 IMAGE_NAME = 'registry.lab.local/go-builder'
 IMAGE_TAG = '1.26.1'
 }

 stages {
 stage('Prepare') {
 steps {
 container('base') {
 script {
 env.GIT_SHA = sh(
 script: 'git rev-parse --short HEAD',
 returnStdout: true
 ).trim()

 currentBuild.displayName = "#${env.BUILD_NUMBER} go-${env.IMAGE_TAG}"
 currentBuild.description = "go-builder:${env.IMAGE_TAG}, commit ${env.GIT_SHA}"
 }

 sh '''
 echo "GO_VERSION=${IMAGE_TAG}"
 echo "IMAGE=${IMAGE_NAME}:${IMAGE_TAG}"
 ls -la
 '''
 }
 }
 }

 stage('Build and push image') {
 steps {
 container('kaniko') {
 sh '''
 /kaniko/executor \
 --context "${WORKSPACE}" \
 --dockerfile "${WORKSPACE}/go-builder/Dockerfile" \
 --destination "${IMAGE_NAME}:${IMAGE_TAG}" \
 --insecure \
 --skip-tls-verify \
 --insecure-registry registry.lab.local
 '''
 }
 }
 }
 }

 post {
 success {
 echo "Built and pushed ${IMAGE_NAME}:${IMAGE_TAG}"
 }
 }
}

Yes, you do not need to rub your eyes in disbelief.

This Jenkinsfile looks almost identical to the previous one. The agent definition is the same, the kaniko invocation is the same, and the base image usage is the same. In practice, only names, version numbers and file paths change.

Simple? Simple.

Example use case

If you have a Go application lying around, you can use a Jenkinsfile like this one.

In this example, the test application is called rme:

pipeline {
 agent {
 kubernetes {
 yaml '''
apiVersion: v1
kind: Pod
spec:
 hostAliases:
 - ip: "10.10.10.24"
 hostnames:
 - "registry.lab.local"
 containers:
 - name: rme-go-builder
 image: registry.lab.local/go-builder:1.26.1
 command: ["sleep"]
 args: ["infinity"]
 - name: rme-kaniko
 image: registry.lab.local/kaniko-builder:latest
 command: ["sleep"]
 args: ["infinity"]
 volumeMounts:
 - name: docker-config
 mountPath: /kaniko/.docker
 volumes:
 - name: docker-config
 configMap:
 name: kaniko-docker-config
'''
 }
 }

 environment {
 IMAGE_NAME = 'registry.lab.local/rme'
 }

 stages {
 stage('Prepare') {
 steps {
 container('rme-go-builder') {
 script {
 env.VERSION = env.GIT_COMMIT.take(7)

 currentBuild.displayName = "#${env.BUILD_NUMBER} ${env.VERSION}"
 currentBuild.description = "${env.IMAGE_NAME}:${env.VERSION}"
 }
 }
 }
 }

 stage('Build binary') {
 steps {
 container('rme-go-builder') {
 sh '''
 git config --global --add safe.directory "$WORKSPACE"
 CGO_ENABLED=0 GOOS=linux go build -o rme .
 '''
 }
 }
 }

 stage('Build and push image') {
 steps {
 container('rme-kaniko') {
 sh '''
 /kaniko/executor \
 --context "${WORKSPACE}" \
 --dockerfile "${WORKSPACE}/Dockerfile" \
 --destination "${IMAGE_NAME}:${VERSION}" \
 --destination "${IMAGE_NAME}:latest" \
 --insecure \
 --skip-tls-verify \
 --insecure-registry registry.lab.local
 '''
 }
 }
 }
 }
}

The application Dockerfile can be very small:

FROM scratch

COPY rme /rme

EXPOSE 9101

ENTRYPOINT ["/rme"]

This is a more realistic example.

It uses a dedicated builder image, go-builder, to compile the Go application, and kaniko only for building and pushing the final container image.

A few details are worth noticing, because the devil usually lives exactly there:

container names matter, because they are later used in container(...) blocks,
the second --destination in the kaniko invocation:

--destination "${IMAGE_NAME}:latest" \

is intentional.

The latest tag will not magically appear in our registry by itself. If we want the most recently pushed image to also be available as latest, we need to tag and push it that way.

Summary and notes

As you can see, there is less and less that we strictly must do inside the cluster, and more and more that we can do in a controlled and repeatable way.

Versioned configuration, job history, artifact history, example applications, builds running inside containers — quite a lot is happening here.

Why use a Multibranch Pipeline?

Good question, but the answers are fairly obvious:

we do not have to clone the repository manually or hard-code credentials in the Jenkinsfile,
test branches do not require separate Jenkins jobs,
Jenkins detects new branches automatically,
deleted branches can disappear from Jenkins automatically as well,
each branch can have its own version of the Jenkinsfile,
pull requests are handled in a much cleaner way.

And now a suspicious thought appears.

What if we put all those YAML files under version control and let some clever mechanism take care of them automatically, instead of calling kubectl by hand every few minutes?

Could such a thing exist?

Who knows.

Jenkins in Kubernetes: Bring a Bucket

Sat, 11 Jul 2026 11:10:00 +0200

Warning: working with Jenkins requires a lot of determination. It is also wise to prepare a bucket in advance — for blood, sweat, tears, and all the failed pipelines.

There. I warned you.

Why Jenkins?

Jenkins has accumulated enough myths and urban legends to deserve several dedicated episodes of a good television show about mysterious phenomena and conspiracy theories.

For now, let us focus on a few practical facts:

Jenkins is a true Swiss Army knife in the CI/CD world.
Jenkins is worth knowing, because it may be lurking just around the corner in the organization you are about to join.
Jenkins is easy to run both locally and centrally.
Thanks to its many plugins, Jenkins can probably be used for almost anything.
You do not have to spend your life clicking around the UI. Jenkins Pipelines do their job.
It is worth getting familiar with Jenkins if only to form your own opinion about what you actually need from a CI/CD ecosystem.

A pinch of history

Jenkins originally started as a tool for building Java code, wrapped in a convenient web service available to everyone.

At first it was called Hudson, and perhaps it would still be called that today, but Sun Microsystems — where Hudson was created — was acquired by the large and not particularly cuddly company known for large and not particularly agile databases.

Then came a dispute over the name and project ownership. A fork appeared, someone proposed a new name, and off it went.

If this reminds you of MariaDB, congratulations: your pattern recognition module is working correctly.

Jenkins in a Kubernetes cluster

We will use Jenkins not only to build containers, but also to build regular applications and run ordinary tasks using purpose-built containers.

Requirements

Jenkins stores data on disk, so we will need a PersistentVolumeClaim.

From the cluster’s point of view, the main Jenkins server will run as a StatefulSet. An Ingress controller will provide HTTP(S) access through the address jenkins.lab.local.

As usual in this lab, add the following line to /etc/hosts on every machine that needs to access Jenkins, including your workstation:

10.10.10.24 jenkins.lab.local

The image

You can use an existing Jenkins image, for example the latest LTS image.

I do not recommend doing that here.

The bucket for blood, sweat and tears would need to be emptied far too often, with very little benefit. It is better to commit a custom image yourself — or in pairs, because at some point the basement nerds need someone to hold the light bulb.

Dockerfile

The Dockerfile looks like this:

FROM jenkins/jenkins:lts-jdk21 AS jenkins-tools

FROM eclipse-temurin:21-jre-jammy

ARG JENKINS_VERSION=2.555.3

USER root

RUN apt-get update && apt-get install -y --no-install-recommends \
 curl \
 git \
 openssh-client \
 ca-certificates \
 tini \
 fontconfig \
 jq \
 unzip \
 zip \
 && rm -rf /var/lib/apt/lists/*

RUN useradd -m -d /var/jenkins_home -u 1000 jenkins

RUN mkdir -p /usr/share/jenkins /var/jenkins_home /usr/share/jenkins/ref/plugins

RUN curl -fsSL \
 -o /usr/share/jenkins/jenkins.war \
 "https://get.jenkins.io/war-stable/${JENKINS_VERSION}/jenkins.war"

COPY --from=jenkins-tools /usr/bin/jenkins-plugin-cli /usr/bin/jenkins-plugin-cli
COPY --from=jenkins-tools /opt/jenkins-plugin-manager.jar /opt/jenkins-plugin-manager.jar

COPY plugins.txt /tmp/plugins.txt

RUN JENKINS_WAR=/usr/share/jenkins/jenkins.war \
 JENKINS_HOME=/usr/share/jenkins/ref \
 jenkins-plugin-cli \
 --war /usr/share/jenkins/jenkins.war \
 --plugin-file /tmp/plugins.txt

RUN chown -R jenkins:jenkins /usr/share/jenkins /var/jenkins_home

USER jenkins

ENV JENKINS_HOME=/var/jenkins_home

VOLUME /var/jenkins_home

EXPOSE 8080 50000

ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["java", "-Djenkins.install.runSetupWizard=false", "-jar", "/usr/share/jenkins/jenkins.war", "--httpPort=8080"]

You can put plugins in plugins.txt and install them directly into the image.

You can also install them later, because Jenkins will have persistent storage. Whether this is a convenience or the beginning of a future archaeology project depends on your level of optimism.

Building the image

The image is built for the arm64 architecture:

docker buildx build \
 --platform linux/arm64 \
 -t registry.lab.local/jenkins-lab:0.0.1 \
 --push .

The most likely result at this point is something like this:

> [stage-1 2/11] RUN apt-get update && apt-get install -y --no-install-recommends curl git openssh-client ca-certificates tini fontconfig jq unzip zip && rm -rf /var/lib/apt/lists/*:
0.126 exec /bin/sh: exec format error

Which means we need to help Docker understand how to run arm64 binaries:

docker run --privileged --rm tonistiigi/binfmt --install arm64

A successful installation should produce something like this:

Installing: arm64 OK
{
 "supported": [
 "linux/amd64",
 "linux/amd64/v2",
 "linux/amd64/v3",
 "linux/arm64",
 "linux/386"
 ],
 "emulators": [
 "python3.14",
 "qemu-aarch64"
 ]
}

Now we can build the image again:

docker buildx build \
 --platform linux/arm64 \
 -t registry.lab.local/jenkins-lab:0.0.1 \
 --push .

Cluster objects

For Jenkins to work properly, we need a few Kubernetes objects.

Namespace

The simplest way to create the namespace is:

kubectl create namespace jenkins

If you keep your configuration under version control — and you should, because future you has enough problems already — use a manifest instead:

---
apiVersion: v1
kind: Namespace
metadata:
 name: jenkins

PersistentVolumeClaim

The PVC definition looks like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: jenkins-home
 namespace: jenkins
spec:
 accessModes:
 - ReadWriteOnce

 resources:
 requests:
 storage: 20Gi

 storageClassName: local-path

ServiceAccount and Role

Jenkins needs permission to create and manage agent Pods in its namespace.

The following definitions are enough for this lab:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: jenkins
 namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: jenkins-agent-manager
 namespace: jenkins
rules:
 - apiGroups: [""]
 resources: ["pods", "pods/exec", "pods/log", "persistentvolumeclaims"]
 verbs: ["get", "list", "watch", "create", "delete", "patch", "update"]
 - apiGroups: [""]
 resources: ["secrets", "configmaps"]
 verbs: ["get", "list", "watch"]
 - apiGroups: [""]
 resources: ["events"]
 verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: jenkins-agent-manager
 namespace: jenkins
subjects:
 - kind: ServiceAccount
 name: jenkins
 namespace: jenkins
roleRef:
 kind: Role
 name: jenkins-agent-manager
 apiGroup: rbac.authorization.k8s.io

Service and Ingress

Now we need a Service and an Ingress:

apiVersion: v1
kind: Service
metadata:
 name: jenkins
 namespace: jenkins
spec:
 selector:
 app: jenkins
 ports:
 - name: http
 port: 8080
 targetPort: http
 - name: agent
 port: 50000
 targetPort: agent
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: jenkins
 namespace: jenkins
spec:
 ingressClassName: nginx
 rules:
 - host: jenkins.lab.local
 http:
 paths:
 - path: /
 pathType: Prefix
 backend:
 service:
 name: jenkins
 port:
 name: http

StatefulSet

Finally, the StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: jenkins
 namespace: jenkins
spec:
 serviceName: jenkins
 replicas: 1

 selector:
 matchLabels:
 app: jenkins

 template:
 metadata:
 labels:
 app: jenkins

 spec:
 serviceAccountName: jenkins
 nodeSelector:
 przeznaczenie: ci-cd

 volumes:
 - name: jenkins-home
 persistentVolumeClaim:
 claimName: jenkins-home

 containers:
 - name: jenkins
 image: registry.lab.local/jenkins-lab:0.0.1
 imagePullPolicy: Always

 ports:
 - name: http
 containerPort: 8080
 - name: agent
 containerPort: 50000

 env:
 - name: JENKINS_HOME
 value: /var/jenkins_home
 - name: JAVA_OPTS
 value: "-Djenkins.install.runSetupWizard=false"

 volumeMounts:
 - name: jenkins-home
 mountPath: /var/jenkins_home

 startupProbe:
 httpGet:
 path: /login
 port: http
 periodSeconds: 10
 failureThreshold: 60

 readinessProbe:
 httpGet:
 path: /login
 port: http
 periodSeconds: 10
 failureThreshold: 12

 livenessProbe:
 httpGet:
 path: /login
 port: http
 periodSeconds: 30
 failureThreshold: 10

Save all of the above definitions as YAML files and apply them with kubectl.

First start

Jenkins should start and greet us by asking for the initial token.

That token can be found in the depths of its home directory on the PVC. After that, it is the usual Jenkins ritual: administrator account, password, pending plugins, and all the other traditional rites of passage.

A freshly started Jenkins is a strange creature.

On one hand, it can be armed to the teeth with plugins. On the other hand, it is as naked as a jaybird. It does not have any build tools installed — but that is fine. We do not really need them.

The actual work will be done by purpose-built containers, created only when needed.

Configuring the Kubernetes cloud

This is where the Jenkins Cloud configuration comes in.

Configure it like this:

Manage Jenkins → Clouds → Add a new cloud → Kubernetes
Name: kubernetes, or any other name you prefer
Kubernetes URL: https://kubernetes.default.svc
Kubernetes namespace: jenkins
Enable WebSocket
Jenkins URL: http://jenkins.jenkins.svc.cluster.local:8080

Save the configuration.

Now all that remains is to create a few useful images, wire them into Jenkins pipelines, and then use them, use them, and use them some more.

Base builder

The first image will be… a base image for building other, better base images.

What does that mean?

The plan is for Jenkins to build many different applications, and also run jobs that may not build anything at all, but still need containers with specific tools inside.

There are two ways to approach this problem:

create one large image with every possible tool inside,
create many small, specialized images and use each one only when needed.

The second approach is easier to maintain.

The Dockerfile for the base builder looks roughly like this:

FROM debian:bookworm-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
 ca-certificates \
 curl \
 wget \
 git \
 openssh-client \
 bash \
 make \
 jq \
 yq \
 tar \
 gzip \
 xz-utils \
 unzip \
 zip \
 rsync \
 file \
 procps \
 iputils-ping \
 dnsutils \
 && rm -rf /var/lib/apt/lists/*

RUN useradd -m -u 1000 builder

USER builder
WORKDIR /workspace

CMD ["sleep", "infinity"]

Build it like this:

docker buildx build \
 --platform linux/arm64 \
 -t registry.lab.local/base-builder:0.1 \
 --push .

Kaniko builder

The second image will contain kaniko, mainly for building container images.

Note: this is a lab setup. Kaniko works well enough for our purposes here, but before using it in production, check the current state of the project and consider alternatives such as Buildah or Podman. The CI/CD ecosystem moves quickly, usually while pretending it is doing so in a controlled manner.

Yes, kaniko is simpler to use than Docker in this setup. It does not require root privileges and it does not need a Docker daemon.

The Dockerfile can look like this:

FROM gcr.io/kaniko-project/executor:latest AS kaniko

FROM alpine:3.21

RUN apk add --no-cache \
 ca-certificates \
 bash \
 coreutils \
 curl \
 git

COPY --from=kaniko /kaniko /kaniko

ENV PATH="/kaniko:${PATH}"

WORKDIR /workspace

CMD ["sleep", "infinity"]

Build it like this:

docker buildx build \
 --platform linux/arm64 \
 -t registry.lab.local/kaniko-builder:0.1 \
 --push .

These two images need to be built manually.

All the other ones will be created later by Jenkins itself.

If you managed to get here in one piece, that is already quite good. In the next episode, we will learn how to make actual use of our newly assembled toys.

2026-06-30

Tue, 30 Jun 2026 19:10:00 +0200

Suit Up and March On!

What is Helm?

Kubernetes is popular enough that it was only a matter of time before someone came up with a way to install applications using one or two commands instead of manually assembling yet another small mountain of YAML files.

That tool is called Helm. It helps define, install, upgrade and manage applications running inside a Kubernetes cluster.

Think of it as the Kubernetes equivalent of apt, dnf or brew — except instead of installing packages on a machine, it installs applications into your cluster.

How does it work?

Helm introduces three basic concepts:

Chart — a package containing templates for all Kubernetes manifests required by an application.
Values — parameters used to customize those templates.
Release — a deployed instance of a chart inside a cluster. The first installation creates the initial revision, and every upgrade creates another one, making rollbacks straightforward.

In practice, you take a chart, adjust its values, install it into the cluster and get a named release that Helm can later upgrade, list, inspect or remove.

What do we need?

Not much:

a Kubernetes cluster,
kubectl,
helm.

Our first application will be Headlamp.

Kubernetes Dashboard still exists, but these days Headlamp is often the more pleasant choice, so that is what we will use here.

Headlamp is one of those rare applications that you install once and it simply works. Such things deserve appreciation.

Installing applications with Helm

Most applications are distributed through Helm repositories.

Adding one is simple:

helm repo add repository-name https://some.repo.url
helm repo update
helm install my-application repository-name/chart

repository-name is your local alias for the repository.

Likewise, my-application is the name of your release. It may look redundant at first, but imagine testing three different MariaDB installations from three different sources. Suddenly meaningful release names start looking less like ceremony and more like self-defense.

helm repo update works much like apt update: it refreshes the local cache with information about available chart versions.

You can also search through charts available in your configured repositories:

helm search repo headlamp
helm search repo bitnami

Where do the values come from?

The easiest way is to ask Helm for the default configuration:

helm show values repository-name/chart > values.yaml

Then edit the generated file and adjust it to your environment.

For Headlamp, the following values.yaml is enough for this lab:

config:
 inCluster: true

ingress:
 enabled: true
 ingressClassName: nginx
 hosts:
 - host: headlamp.lab.local
 paths:
 - path: /
 type: Prefix

nodeSelector:
 przeznaczenie: observability

We need to define the Ingress class, the hostname under which the application will be available, and optionally a nodeSelector if we want to control where the Pod runs.

As usual in this lab setup, we also need to add the hostname to /etc/hosts on every machine that will access the service, including the workstation:

10.10.10.24 headlamp.lab.local

Installing Headlamp

Now comes the easy part:

helm repo add headlamp https://kubernetes-sigs.github.io/headlamp/
helm repo update

helm install headlamp headlamp/headlamp \
 --namespace headlamp \
 --create-namespace \
 -f values.yaml

After opening the URL in a browser, we get a friendly login screen:

Headlamp asks for a Kubernetes authentication token.

Generating one is straightforward:

kubectl -n headlamp create token headlamp

The application itself is clean, tasteful and not overloaded with options. Most things are exactly where you expect them to be.

You can browse resources, inspect logs and click through various settings without leaving the browser.

Another excellent Kubernetes client is k9s. It runs entirely in the terminal, so you do not get fancy graphical topology views, but everything else is there — and it is fast.

As you can see, basic Helm usage is pleasantly simple.

Installing databases

There is not much point in overexplaining databases here. We are going to install a few of them from a popular public repository and see how little work is actually required.

We will use Bitnami’s repository:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

kubectl create namespace databases

The separate namespace is not strictly magical, but it keeps things tidy.

PostgreSQL

Example values.yaml:

global:
 storageClass: local-path

architecture: standalone

auth:
 postgresPassword: "some-password"
 username: "marcin"
 password: "some-other-password"
 database: "utensils"

primary:
 nodeSelector:
 przeznaczenie: heavy-worker

 persistence:
 enabled: true
 size: 20Gi

 service:
 type: NodePort
 ports:
 postgresql: 5432
 nodePorts:
 postgresql: 30432

The database will be exposed through NodePort 30432.

Since this is not a web application, we need to remember which node actually hosts the Pod. Thanks to the nodeSelector, that should be 10.10.10.25.

Installation:

helm install postgresql bitnami/postgresql \
 --namespace databases \
 -f values.yaml

Afterwards Helm prints a few useful notes, including commands for retrieving generated credentials:

To get the password for "postgres" run:

export POSTGRES_ADMIN_PASSWORD=$(kubectl get secret --namespace databases postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)

To get the password for "marcin" run:

export POSTGRES_PASSWORD=$(kubectl get secret --namespace databases postgresql -o jsonpath="{.data.password}" | base64 -d)

MariaDB

Example configuration:

global:
 storageClass: local-path

architecture: standalone

auth:
 rootPassword: "some-password"
 database: "utensils"
 username: "marcin"
 password: "some-other-password"

primary:
 nodeSelector:
 przeznaczenie: heavy-worker

 persistence:
 enabled: true
 size: 20Gi

 service:
 type: NodePort
 ports:
 mysql: 3306
 nodePorts:
 mysql: 30306

This database will listen on NodePort 30306, on the same node as PostgreSQL.

Installation:

helm install mariadb bitnami/mariadb \
 --namespace databases \
 -f values.yaml

Administrator credentials can be retrieved from the generated Secret:

Username: root
Password: $(kubectl get secret --namespace databases mariadb -o jsonpath="{.data.mariadb-root-password}" | base64 -d)

Redis

Example configuration:

architecture: standalone

auth:
 enabled: false

master:
 nodeSelector:
 przeznaczenie: heavy-worker

 persistence:
 enabled: true
 storageClass: local-path
 size: 5Gi

 resources:
 requests:
 cpu: 50m
 memory: 128Mi
 limits:
 memory: 512Mi

metrics:
 enabled: true
 serviceMonitor:
 enabled: true
 namespace: databases

Installation is exactly what you would expect:

helm install redis bitnami/redis \
 --namespace databases \
 -f values.yaml

At this point the cluster is starting to look like an actual environment instead of an empty playground.

As an exercise, I recommend installing a few well-known applications and checking how they are packaged, configured and exposed. There is no need to rediscover America when so many useful things are already available in public Helm repositories.

Updating an application

Sooner or later, you will change something in values.yaml.

Instead of uninstalling and installing the application again, upgrade the existing release:

helm upgrade headlamp headlamp/headlamp \
 --namespace headlamp \
 -f values.yaml

Helm applies the new configuration while preserving release history. If something goes sideways, that history becomes useful very quickly.

Removing applications

When an application outlives its usefulness — and why is it always Headlamp? — removing it is just as easy:

helm uninstall my-application

If it was installed in a specific namespace, add it explicitly:

helm uninstall my-application \
 --namespace namespace-name

Listing installed releases

This command shows releases from the current namespace:

helm list

Which is technically correct and practically disappointing most of the time.

Usually, this is what you want:

helm list -A

It lists releases across all namespaces.

And that is the nice thing about Helm: the basic workflow stays small.

Add a repository, inspect the values, install a release, upgrade it when needed, list what is running, remove what is no longer useful. Underneath, Kubernetes still does Kubernetes things, but at least you do not have to hand-write every single manifest like it is some YAML-based monastic ritual.

2026-06-26

Fri, 26 Jun 2026 11:15:00 +0200

Why Do We Need a Registry?

Sooner or later, every Kubernetes cluster starts running applications that were built locally.

At first, using Docker Hub seems perfectly reasonable.

Then the first custom application appears.

Then another.

Eventually every deployment references images that exist only on your workstation.

Clearly, this is not a sustainable solution.

A private container registry solves this problem by providing a permanent home for your own images. Since the home lab already has plenty of storage available, hosting the registry locally is both the simplest and the fastest option.

PersistentVolume and PersistentVolumeClaim

One of the original design goals of Kubernetes was that Pods should be ephemeral.

Applications come and go.

Containers are recreated.

Nodes may disappear.

Unfortunately, databases tend to have rather strong opinions about losing their contents.

To bridge the gap between ephemeral workloads and persistent storage, Kubernetes introduced two API objects:

PersistentVolume (PV)
PersistentVolumeClaim (PVC)

PersistentVolume

A PersistentVolume represents an actual storage resource available inside the cluster.

Unlike Pods, a PersistentVolume exists independently of any application using it. Deleting a Pod does not automatically remove the underlying storage.

Think of it as the physical disk—or, more precisely, a piece of storage that Kubernetes knows how to manage.

PersistentVolumeClaim

A PersistentVolumeClaim is exactly what its name suggests: a request for storage.

Rather than asking for a specific disk, an application simply requests:

a certain amount of space,
an access mode,
and, optionally, a storage class.

The Pod never mounts a PersistentVolume directly. Instead, it references a PVC and lets Kubernetes handle the rest.

Binding

Whenever a PersistentVolumeClaim is created, Kubernetes searches for a suitable PersistentVolume that satisfies the requested parameters.

If one is found, the claim becomes permanently bound to that volume.

From the application’s point of view, none of this complexity exists.

The Pod simply mounts the PVC as a directory inside the container.

Access Modes

Access modes describe how Kubernetes nodes are allowed to mount a volume.

Four access modes currently exist:

ReadWriteOnce (RWO) — the volume may be mounted for reading and writing by only one node at a time. Multiple Pods may still use it, provided they all run on that node.
ReadWriteMany (RWX) — the volume may be mounted simultaneously by multiple nodes.
ReadOnlyMany (ROX) — multiple nodes may mount the volume, but only for reading.
ReadWriteOncePod (RWOP) — introduced more recently, this mode guarantees that exactly one Pod in the entire cluster has read-write access to the volume.

Note: the underlying storage backend must actually support the requested access mode. Kubernetes cannot magically turn a local SSD into a distributed filesystem.

Where Should the Data Live?

By default, k3s ships with the Local Path Provisioner, which stores dynamically created PersistentVolumes under:

/var/lib/rancher/k3s/storage

Personally, I prefer keeping persistent application data somewhere more obvious, such as:

/srv/storage

To do that, create the directory on every node:

sudo mkdir -p /srv/storage
sudo chmod 777 /srv/storage

The permissions are intentionally generous. This is a home lab, not Fort Knox.

Next, edit the local-path-config ConfigMap:

kubectl -n kube-system edit configmap local-path-config

The interesting part looks like this:

apiVersion: v1
data:
 config.json: |-
 {
 "nodePathMap":[
 {
 "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES",
 "paths":["/var/lib/rancher/k3s/storage"]
 }
 ]
 }

Simply replace:

"paths":["/var/lib/rancher/k3s/storage"]

with:

"paths":["/srv/storage"]

Newly created PersistentVolumes will now live there instead.

Deploying the Registry

The registry will be available as:

registry.lab.local

Since there is no local DNS server yet, every machine—including your workstation—should receive the following /etc/hosts entry:

10.10.10.24 registry.lab.local

The registry should preferably run on the most capable machine in the cluster.

In my case, that is iridium.

Assign it an appropriate label:

kubectl label node iridium registry=true

The Deployment manifest below uses that label via a nodeSelector.

At first glance, a registry seems like a simple application.

Then you remember it needs persistent storage.

And a Service.

And an Ingress.

And perhaps its own namespace.

And a ConfigMap.

So, in the end, it turns out to require exactly six Kubernetes objects:

Namespace
PersistentVolumeClaim
ConfigMap
Deployment
Service
Ingress

Their complete definitions are shown below.

(YAML manifest unchanged.)

Save everything as:

registry.yaml

and deploy it using:

kubectl apply -f registry.yaml

Interesting Bits

The PVC requests persistent storage:

accessModes:
 - ReadWriteOnce

The registry is pinned to the node labelled earlier:

nodeSelector:
 registry: "true"

The ingress disables request size limits:

nginx.ingress.kubernetes.io/proxy-body-size: "0"

Without this annotation, pushing sufficiently large images eventually results in sadness, disappointment, and extensive arm waving.

The registry configuration itself is stored in a ConfigMap rather than a long list of environment variables.

Since the registry already expects a YAML configuration file, using a ConfigMap keeps the deployment considerably cleaner.

This setup stores all registry data on the local filesystem of a single node. That is perfectly adequate for a home lab. Production environments usually rely on distributed storage instead.

Running Without TLS

If the registry temporarily runs over plain HTTP, Docker must be told to trust it.

Create:

/etc/docker/daemon.json

containing:

{
 "insecure-registries": ["registry.lab.local"]
}

Likewise, every k3s node should receive:

/etc/rancher/k3s/registries.yaml

containing:

mirrors:
 "registry.lab.local":
 endpoint:
 - "http://registry.lab.local"

Finally, restart k3s:

sudo systemctl restart k3s || sudo systemctl restart k3s-agent

Don’t forget to update the ConfigMap as well if you’re switching from HTTPS to HTTP.

Verification

The usual suspects come first:

kubectl get pods -n registry
kubectl get pvc -n registry
kubectl get ingress -n registry
kubectl get svc -n registry
kubectl describe ingress registry -n registry
kubectl get configmap -n registry

If the PVC reports:

STATUS: Bound

everything is proceeding nicely:

the PersistentVolume exists,
the claim has been bound,
the Pod is successfully using it.

Smoke Test

Let’s put the registry to work.

docker pull php:8.4.22-zts-trixie

docker tag php:8.4.22-zts-trixie \
 registry.lab.local/php:8.4.22

docker push registry.lab.local/php:8.4.22

To make sure the image really came from the registry rather than your local cache, remove it first:

docker image rm registry.lab.local/php:8.4.22

docker pull registry.lab.local/php:8.4.22

Finally, query the registry directly:

curl https://registry.lab.local/v2/php/tags/list

which should return:

{"name":"php","tags":["8.4.22"]}

Or list every repository:

curl -s https://registry.lab.local/v2/_catalog | jq

Congratulations.

Your registry is officially alive.

Registry UI

A registry works perfectly well without a graphical interface.

Humans, however, tend to appreciate one.

The excellent Docker Registry UI provides browsing, searching, and deleting images from a web browser.

As before, add:

10.10.10.24 registry-ui.lab.local

to every /etc/hosts.

Deploy the accompanying Deployment, Service, and Ingress manifests (shown below), save them as:

registry-ui.yaml

and apply them:

kubectl apply -f registry-ui.yaml

The interface is intentionally simple.

It won’t win any design awards, but it makes exploring your registry significantly more pleasant than remembering API endpoints.

Wrapping Up

At this point, the cluster has become largely self-sufficient.

Applications can now be built locally, pushed into the private registry, and deployed without relying on Docker Hub.

It may not look particularly impressive yet, but from now on every new service added to the cluster can remain entirely within the home lab.

Which, after all, was the idea from the very beginning.

2026-06-24

Wed, 24 Jun 2026 11:15:00 +0200

k3s: Ingress

What exactly is an Ingress?

In the Kubernetes ecosystem, an Ingress is an API object that manages external access to applications running inside a cluster. It acts as an intelligent HTTP and HTTPS router, eliminating the need to create a separate load balancer for every service.

How Does Ingress Work?

Traditional service exposure methods, such as NodePort or LoadBalancer, typically require assigning an external IP address to each service. This quickly becomes tedious and difficult to manage as the cluster grows.

Ingress centralizes external access and operates at Layer 7 (the application layer) of the OSI model:

Single entry point — only one public IP address needs to be exposed to the outside world.
Rule-based routing — traffic is forwarded to the appropriate service based on the requested URL path (for example, my.domain/something) or hostname (for example, something.my.domain).
SSL/TLS termination — HTTPS certificates can be handled centrally, removing that responsibility from individual applications.

Ingress vs. Ingress Controller

This distinction is important.

An Ingress object itself is merely a collection of routing rules.

For those rules to have any effect, the cluster must run an Ingress Controller — a reverse proxy responsible for processing incoming traffic and applying the rules defined in Ingress manifests.

Popular examples include:

NGINX Ingress Controller
Traefik
HAProxy Ingress

This guide uses the NGINX Ingress Controller.

Domain Names

Since the cluster will eventually host various web applications, it makes sense to identify services by DNS names rather than IP addresses.

The examples below use:

.lab.local

as the local domain.

This allows services to be accessed using names such as:

grafana.lab.local
argocd.lab.local
whoami.lab.local

which is considerably more pleasant than remembering port numbers.

Certificates

Sooner or later, running everything without TLS becomes annoying.

For a home lab environment, generating your own Certificate Authority and issuing certificates locally is perfectly acceptable.

Root Certificate Authority

Create a dedicated CA:

mkdir -p certs
cd certs

openssl genrsa -out lab-ca.key 4096

openssl req -x509 -new -nodes \
 -key lab-ca.key \
 -sha256 \
 -days 3650 \
 -out lab-ca.crt \
 -subj "/CN=Lab Local CA"

Wildcard Certificate

A wildcard certificate can secure every host under:

*.lab.local

Create the configuration file lab-local.cnf:

[req]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
req_extensions = req_ext

[dn]
CN = *.lab.local

[req_ext]
subjectAltName = @alt_names

[alt_names]
DNS.1 = *.lab.local
DNS.2 = lab.local

Generate the key, request, and certificate:

openssl genrsa -out lab-local.key 2048

openssl req -new \
 -key lab-local.key \
 -out lab-local.csr \
 -config lab-local.cnf

openssl x509 -req \
 -in lab-local.csr \
 -CA lab-ca.crt \
 -CAkey lab-ca.key \
 -CAcreateserial \
 -out lab-local.crt \
 -days 825 \
 -sha256 \
 -extensions req_ext \
 -extfile lab-local.cnf

Namespace for the Ingress Controller

The ingress controller will live in its own namespace.

Create ingress-namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
 name: ingress-nginx

Apply it:

kubectl apply -f ingress-namespace.yaml

TLS Secret

The certificate must be stored inside Kubernetes as a TLS secret:

kubectl -n ingress-nginx create secret tls lab-local-tls \
 --cert=lab-local.crt \
 --key=lab-local.key

The secret is named:

lab-local-tls

and can be referenced as:

ingress-nginx/lab-local-tls

throughout the cluster configuration.

Installing the CA Certificate on Your Workstation

The root CA must also be trusted by your workstation:

sudo cp lab-ca.crt /usr/local/share/ca-certificates/lab-ca.crt
sudo update-ca-certificates

Firefox users should additionally import the CA certificate manually through the browser settings.

Deploying the Ingress Controller

A complete ingress controller deployment requires several Kubernetes resources:

ServiceAccount
IngressClass
ConfigMap
ClusterRole
ClusterRoleBinding
Deployment

The container image used in this setup is:

registry.k8s.io/ingress-nginx/controller:v1.14.0

which provides the familiar NGINX reverse proxy and load balancer functionality.

The deployment manifest is shown below.

# ingress.yaml
# (manifest unchanged from the original article)

Noteworthy Configuration

The ingress controller is pinned to the control-plane node:

nodeSelector:
 kubernetes.io/hostname: iridium

This is perfectly acceptable for a small home lab.

The controller also uses the wildcard certificate by default:

--default-ssl-certificate=ingress-nginx/lab-local-tls

which means every ingress can immediately benefit from TLS.

Verifying the Installation

Start with the usual checks:

kubectl get pods -n ingress-nginx -o wide
kubectl get rs -n ingress-nginx -o wide

Next, add a test host entry:

10.10.10.24 test.lab.local

to your workstation’s /etc/hosts.

Then run:

curl http://test.lab.local

Expected result:

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

Surprisingly, this is good news.

The DNS lookup works.

The ingress controller works.

NGINX works.

There simply isn’t any application behind the hostname yet.

The infrastructure is doing exactly what it should.

TLS Verification

Next, test HTTPS:

curl -v https://test.lab.local

The interesting part is not the final HTTP response, but the TLS negotiation itself.

Look for output similar to:

SSL connection using TLSv1.3
...
subject: CN=*.lab.local
...
issuer: CN=Lab Local CA
...
subjectAltName: "test.lab.local" matches cert's "*.lab.local"
...
SSL certificate verified via OpenSSL

If these messages appear, the certificate chain is functioning correctly and the wildcard certificate is being served as expected.

Finally, open the URL in Firefox and confirm that the browser displays the expected lock icon and certificate information.

Conclusion

At this point:

DNS works,
the ingress controller is running,
TLS works,
wildcard certificates work,
external traffic reaches the cluster.

No actual applications are exposed yet, but the foundation is now in place.

The cluster is finally ready to host something more interesting than infrastructure components.

Which, as every Kubernetes administrator eventually discovers, is usually the easy part.

2026-06-23

Tue, 23 Jun 2026 11:15:00 +0200

Installing k3s

Warning:

This article describes the preparation, installation, and initial configuration of a Kubernetes cluster.

You are reading at your own risk. There is a non-zero chance that your life will never be quite the same afterward.

All procedures described here have been tested.

Also, no animals were harmed during the writing of this article. Unless we count two flies, six mosquitoes, and one unfortunate snail removed from a home cucumber plantation.

Hardware

The cluster consists of the following machines:

Hostname	IP Address	SBC	Architecture	RAM	Storage	Operating System	Intended Role
palladium	10.10.10.20	Odroid N2	arm64	4 GB	128 GB	Armbian	tiny-int
vanadium	10.10.10.21	Raspberry Pi 5	arm64	8 GB	512 GB	Ubuntu Server	ci-cd
iridium	10.10.10.24	Odroid M2	arm64	16 GB	512 GB	Armbian	control-plane
ruthenium	10.10.10.25	Orange Pi 5 Plus	arm64	16 GB	512 GB	Armbian	heavy-worker
tantalum	10.10.10.26	Raspberry Pi 500	arm64	16 GB	256 GB	Ubuntu Server	observability

All IP addresses are assigned by the home router using static DHCP reservations based on MAC addresses.

Operating System and Required Software

All nodes run either Ubuntu Server or Armbian based on Ubuntu Noble.

Versions may differ slightly, but package management remains identical across the cluster.

The first step is ensuring that every machine can resolve the names of every other machine.

At the moment there is no local DNS server, so each host requires an /etc/hosts entry similar to the following:

10.10.10.20 palladium
10.10.10.21 vanadium
10.10.10.24 iridium
10.10.10.25 ruthenium
10.10.10.26 tantalum

It is also worth adding the same entries to the workstation or laptop used to manage the cluster.

Windows users may perform the traditional rite of passage and figure out where Microsoft decided to hide the hosts file this decade.

Useful Tools

It is highly recommended to install kubectl on your workstation.

The official installation instructions can be found here:

https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

On every cluster node, install a basic set of utilities:

sudo apt update && sudo apt install -y \
 curl \
 wget \
 vim \
 git \
 htop \
 net-tools \
 iproute2 \
 iptables \
 ca-certificates \
 gnupg \
 lsb-release \
 jq \
 socat \
 conntrack \
 open-iscsi \
 nfs-common \
 tmux \
 dnsutils \
 tcpdump \
 lsof \
 nc

If you prefer the Emacs lifestyle but do not enjoy “the Sixth Editor”, replace vim with mg.

You may also want:

sudo apt install mc

if navigating directories through a dual-panel interface reminds you of happier times.

On iridium, I additionally install:

sudo apt install -y sqlite3

Swap was already disabled during the preparation of the machines, but it is worth double-checking before proceeding.

Kernel Modules

Enable the required kernel modules:

cat <<EOF | sudo tee /etc/modules-load.d/k3s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

Then configure the required kernel parameters:

cat <<EOF | sudo tee /etc/sysctl.d/k3s.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
EOF

sudo sysctl --system

cgroups

It is also worth checking whether cgroups are properly enabled:

cat /proc/cgroups

Some systems may require additional kernel parameters in /boot/cmdline.txt:

cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

Kubernetes

The chosen distribution is k3s.

It is fully compatible with “serious” Kubernetes while requiring significantly fewer resources, making it an excellent choice for home labs, edge computing, and IoT deployments.

Small, lightweight, easy to install, and surprisingly capable. What more could one ask for?

Control Plane

iridium has been selected as the control-plane node because it is the most powerful machine in the cluster.

The remaining nodes will join as workers with specific responsibilities.

Install k3s on iridium:

curl -sfL https://get.k3s.io | sh -s - server \
 --node-name iridium \
 --node-ip 10.10.10.24 \
 --advertise-address 10.10.10.24 \
 --write-kubeconfig-mode 644 \
 --disable traefik

Run this command directly on the machine, either through SSH or a local terminal session.

What Do These Flags Do?

--node-ip — forces a specific IP address and therefore a specific network interface.
--advertise-address — makes the API server reachable from the LAN.
--disable traefik — k3s ships with Traefik as its default ingress controller, but this cluster will use NGINX later.
--write-kubeconfig-mode 644 — creates the kubeconfig file with world-readable permissions.

That’s it.

Really.

After installation, verify that everything is running correctly:

sudo systemctl status k3s

More importantly:

kubectl get nodes

should produce something similar to:

iridium Ready control-plane

An even better command is:

kubectl get nodes -w

The -w option continuously watches for changes.

This is useful because a node that repeatedly transitions between Ready and some other state usually indicates an incomplete kernel configuration.

The most common culprits are missing overlay or br_netfilter modules.

If everything looks healthy, retrieve the cluster configuration:

cat /etc/rancher/k3s/k3s.yaml

Copy it to your workstation.

You may store it as:

~/.kube/config

or keep multiple cluster configurations separately.

For Bash:

export KUBECONFIG=~/.kube/my-cluster-config

For Fish:

set -x KUBECONFIG ~/.kube/my-cluster-config

Before using the file, edit the server address:

apiVersion: v1
clusters:
- cluster:
 certificate-authority-data: ...
 server: https://127.0.0.1:6443

Replace:

127.0.0.1

with:

10.10.10.24

You should now be able to run:

kubectl get nodes

directly from your workstation and obtain the same results.

Joining Worker Nodes

The node registration token can be found on the control-plane node:

sudo cat /var/lib/rancher/k3s/server/node-token

Use it on each worker node:

curl -sfL https://get.k3s.io | \
K3S_URL=https://10.10.10.24:6443 \
K3S_TOKEN='YOUR_TOKEN_HERE' \
sh -s - agent \
 --node-name vanadium \
 --node-ip 10.10.10.21

This example joins vanadium.

Remember to adjust both the node name and IP address accordingly.

Run the command directly on the target machine.

Once all nodes have joined, verify the cluster:

kubectl get nodes -o wide

Every node should eventually report:

Ready

Simple?

Simple.

A little time-consuming, perhaps, but there are no lengthy computations involved.

Only now is it safe to reach for your favorite book.

2026-06-17

Wed, 17 Jun 2026 08:15:00 +0200

Continuation of the post Hardware and Software (I)

Odroid M2

The Odroid M2 deserves a dedicated article, as installing and configuring the operating system requires a few extra steps and a fair amount of patience.

The board vendor seems particularly fond of shipping museum-grade software as officially supported images. Personally, I am not a fan of running Ubuntu 20.04 with a 4.9 or 5.1 kernel in 2026. That is not exactly what modern workloads are looking for.

Instead, I strongly recommend using Armbian, whose developers release updates regularly. If you’re impatient, you can always build your own image.

Unfortunately, after installing the official image on an SD card, it quickly becomes apparent that Kubernetes is not particularly happy with the default kernel configuration. The kernel is built with only a minimal set of features enabled, so it is time to roll up your sleeves and build a custom version.

Storage

The Odroid M2 provides an M.2 NVMe slot on the underside of the board, making SSD installation straightforward. Simply insert the drive, secure it using the supplied screw, and enjoy your new storage.

Important: The board cannot boot directly from an NVMe SSD. You must use either an SD card or the built-in eMMC module for booting.

Take a close look at the connectors and switches on the board. One of them controls the SD/eMMC boot source selection. During installation and initial setup, leave it in the Micro SD position.

Building the System

Additional requirements: two SD cards (assuming limited prior experience and the possibility of rebuilding the image later).

Bootstrap

Start by installing the latest version of Armbian onto an SD card and booting from it.

Complete the initial system configuration (passwords, user accounts, locale, and everything else the setup wizard asks for).
Allow your user to execute sudo without a password by adding something similar to /etc/sudoers:

marcin ALL=(ALL) NOPASSWD:ALL

Install git.
Clone the build repository:

git clone --depth 1 https://github.com/armbian/build

Launch the main build script:

cd build
./compile.sh

The script handles the entire build process:

installs missing dependencies,
guides the user through the required configuration,
downloads and builds the kernel,
fetches all remaining packages,
generates a complete system image.

Configuration and Build

Important options in the configuration wizard:

Show a kernel configuration menu before compilation

The default kernel configuration is generally reasonable, but it is worth verifying that options required by Kubernetes are enabled, particularly various netfilter features.

Choose a board → odroidm2
Choose a kernel → current
Choose a release package base → noble

(resolute has recently appeared, but I have not tested it yet.)

Choose image type → Image with console interface
Choose image type (II) → Standard image with console interface

You may spend anywhere from a few minutes to several hours reviewing kernel options, which is why I recommend having a spare SD card available.

Once configuration is complete, the build system will spend quite some time compiling kernels, building packages, and assembling the final image.

During that time, you can:

go for a run,
visit the swimming pool,
ride a bicycle,
or, as a last resort, take a very long walk.

Once the build completes successfully, you should find something similar to the following in output/images:

Armbian-unofficial_26.08.0-trunk_Odroidm2_resolute_current_6.18.35.img

Yes, every locally built image automatically receives the -unofficial suffix.

Writing the Image

Since the device only provides a single SD card slot, you will need to transfer the generated image to your regular workstation and write it to the second SD card.

After that:

shut down the device,
replace the SD card,
power it on,
enjoy your shiny new system,
go through the initial configuration one more time.

Make sure the SSD is visible:

lsblk | grep nvme0

The output should look similar to:

nvme0n1

Installing the System onto eMMC and NVMe

Running from an SD card is useful for diagnostics and experimentation, but the built-in eMMC should be used for everyday operation.

To begin installation:

armbian-install

The most important choice during installation is:

boot from eMMC, root from NVMe

The installer will warn about potential data loss, create fresh partitions on both eMMC and SSD storage, and transfer the already configured system onto the SSD.

As a result, you will not need to repeat the entire setup process again.

Once installation completes:

shut down the device,
remove the SD card,
switch the boot selector to eMMC,
power the board on,
enjoy a significantly faster-booting Odroid M2.

Final System Configuration

The first thing to do is disable swap.

Check whether swap is active:

swapon --show
free -h

Typical output:

NAME TYPE SIZE USED PRIO
/swap.img file 8G 0B -2

Disable it completely:

sudo swapoff -a

If swapon --show reports something like:

/dev/zram0

then zram should be disabled as well:

sudo systemctl disable --now armbian-zram-config

Optionally remove the configuration:

sudo rm -f /etc/default/armbian-zram-config

Kubernetes does not particularly appreciate active swap devices, so it is best to disable them before proceeding further.

Your user account probably no longer has passwordless sudo access, so repeat the same sudoers modification as before.

Finally, replace your SSH keys:

ssh-copy-id marcin@odroidm2

In my case:

ssh-copy-id marcin@10.10.10.24

Voilà. The system is now ready for the next stage.

2026-06-15

Mon, 15 Jun 2026 19:15:00 +0200

This post provides a brief overview of the hardware, installation methods, and software configuration used in my home Kubernetes lab.

Hardware

The cluster currently consists of:

Odroid N2
Odroid M2
Raspberry Pi 5
Raspberry Pi 500
Orange Pi 5 Plus

Among these devices, only the Odroid N2 lacks an M.2 interface for an NVMe SSD and must rely on an SD card for storage. All the others support NVMe drives — well, almost all of them, but we’ll get to that in a moment.

I’m only interested in 2280 form-factor drives due to their lower cost and wider availability.

Let’s take a quick look at what is required to use — and boot from — an NVMe drive on each platform.

Raspberry Pi 5

The Raspberry Pi 5 supports M.2 NVMe storage through an additional PCIe expansion board (HAT). The best option in terms of price and quality is the one made by Pimoroni.

The Internet is full of installation guides, so I’ll keep it brief. The process consists of:

attaching the metal standoffs,
mounting the SSD using the supplied screw,
connecting the ribbon cable,
attaching the HAT to the underside of the Raspberry Pi.

The SSD ends up hidden inside the assembly, between the Raspberry Pi 5 and the HAT itself.

Operating System Installation

There are two ways to install the operating system: with or without an SD card. For the purposes of this project, we’re only interested in Ubuntu Server.

Method 1

The first method is straightforward:

write a standard Raspberry Pi OS image (or any other operating system) to an SD card and boot the Raspberry Pi,
verify that the system can see the SSD:

$ lsblk | grep nvme0

The command should return something similar to:

nvme0n1

Then:

install and launch rpi-imager,
select Ubuntu Server from the operating system list,
choose the NVMe drive as the target device,
start the image-writing process,
shut down the Raspberry Pi,
remove the SD card,
boot the system again.

There is no need to modify the boot order and make NVMe booting the default. It’s useful to retain the ability to boot from an SD card “just in case”.

The Raspberry Pi will still boot from the NVMe drive automatically, albeit a few seconds later. Those extra seconds are hardly a problem.

Method 2

The second method requires an M.2 to USB adapter.

In this case, the operating system image is written directly to the SSD before it is mounted onto the HAT.

Once installed, the Raspberry Pi boots exactly the same way as in the first method.

Configuration

There are two things worth doing immediately after installation:

disable swap,
create a dedicated user account, grant passwordless sudo access, and exchange SSH keys.

You can verify whether swap is enabled using:

$ swapon --show
$ free -h

If swap is active, you’ll likely see something similar to:

NAME TYPE SIZE USED PRIO
/swap.img file 8G 0B -2

Disable swap immediately:

$ sudo swapoff -a

Then remove the corresponding entry from /etc/fstab, for example:

/swap.img none swap sw 0 0

This prevents swap from being re-enabled after a reboot.

Creating a dedicated administrative user is equally simple. Assuming the new user is called marcin, add the following entry to /etc/sudoers:

# marcins and others:
marcin ALL=(ALL) NOPASSWD:ALL

Finally, exchange SSH keys:

$ ssh-copy-id marcin@raspberry-pi-5-address

In my case:

$ ssh-copy-id marcin@10.10.10.21
$ ssh-copy-id marcin@10.10.10.26

With that done, the machine is ready for automation with tools such as Ansible.

In the next installment: Odroid M2 — and there will be quite a bit to talk about.

2026-06-12

Fri, 12 Jun 2026 19:15:00 +0200

Home Kubernetes Lab

I’ve always been a fan of ARM-based single-board computers (SBCs). My journey started with a Raspberry Pi 4, which served as my primary workstation for nearly two years, followed by an Odroid N2. These are simple, reliable devices that are surprisingly versatile—even capable retro-gaming machines.

While the Raspberry Pi 4 has since been replaced by a Raspberry Pi 5, the Odroid N2 remains in active service. It hosts several lightweight databases and web applications and continues to perform flawlessly.

Over time, my collection has expanded to include:

Odroid M2 — the next generation of the Odroid product line,
Orange Pi 5 Plus,
Raspberry Pi 500 — purchased as an impulse bargain.

Hardware Overview

Hostname	palladium	vanadium	iridium	ruthenium	tantalum
SBC Model	ODROID N2	Raspberry Pi 5	ODROID M2	Orange Pi 5 Plus	Raspberry Pi 500
SoC	Amlogic S922X	Broadcom BCM2712	Rockchip RK3588S2	Rockchip RK3588	Broadcom BCM2712
CPU Architecture	4× Cortex-A73 + 2× Cortex-A53	4× Cortex-A76	4× Cortex-A76 + 4× Cortex-A55	4× Cortex-A76 + 4× Cortex-A55	4× Cortex-A76
Memory	4 GB	8 GB	16 GB	16 GB	16 GB
Storage Type	microSD	NVMe	NVMe + eMMC	NVMe	NVMe
Storage Capacity	128 GB	512 GB	512 GB + 64 GB	512 GB	256 GB
Operating System	Armbian	Ubuntu Server	Armbian	Armbian	Ubuntu Server

Retired Systems

Hostname	ferrum	thorium
SBC Model	Radxa Rock Pi S	FriendlyElec NanoPi Neo
SoC	Rockchip RK3308	Allwinner H3
Architecture	64-bit (ARMv8-A)	32-bit (ARMv7-A)
CPU	4× Cortex-A35 @ 1.3 GHz	4× Cortex-A7 @ 1.2 GHz
Memory	512 MB DDR3	512 MB DDR3
Storage Type	microSD	microSD
Storage Capacity	64 GB	64 GB
Operating System	DietPi	DietPi

Objectives

The goal of this project is to build a robust Kubernetes cluster capable of hosting both personal and widely used applications and databases. The cluster should also include a complete platform engineering stack, featuring:

comprehensive observability,
CI/CD pipelines,
GitOps-driven deployments.

Proposed Node Roles

iridium — Kubernetes control plane,
palladium — lightweight worker node and test environment,
vanadium — CI/CD workloads,
tantalum — observability stack,
ruthenium — heavy worker node for resource-intensive web applications.

Tooling

k3s — lightweight Kubernetes distribution,
Ansible — infrastructure automation and configuration management.