Moving my personal infrastructure to Kubernetes

Introduction

Over the course of a weekend I moved all of my personal infrastructure - including the system hosting this website - onto a single-node K3s Kubernetes cluster. This time I decided I would actually document the whole undertaking from the beginning. During previous migrations I often forgot to do this, much to the displeasure of my future self. In addition I have also decided to publish the document itself, for the purpose of helping other people wanting to run similar setups.

Before the Kubernetes migration I had one cheap Hetzner VPS running all of my infrastructure. This server was perfectly fine for what it was used for. It ran without (too) many issues and did what it needed to do. The only issue was that I often forgot to document changes to the server. This made consequent configuration changes a lot more difficult and time consuming than they had to be, as I had to figure out large parts of the setup from scratch again.

To avoid this issue I chose to migrate to Kubernetes. Admittedly, I also just wanted a good excuse to work with Kubernetes besides my job. Kubernetes is quite complex, and it caters towards usecases that you simply don't encounter in a small sideproject. It is simply not a good choice when you have much simpler alternatives like Docker Compose, which you could learn in an afternoon. I found however that once you already put in the work to learn Kubernetes, it is just better than the alternatives.

In this article you will find a thorough documentation of the kubernetes cluster setup, a description of the migration process from VPS to K3s cluster, a guide on how to use helm to manage your cluster, a discussion of which cloud provider to use, and plenty of other things I thought were worth mentioning.

Introduction
Table of contents
1. Setting up the Kubernetes cluster
2. Installing the cluster infrastructure
3. Managing the cluster
- 3.1. Using Helm for change management

1. Setting up the Kubernetes cluster

1.1. Choosing between managed and self-hosted Kubernetes

If you are reading this you probably know that Kubernetes comes in two flavours nowadays: self-hosted and managed. At my work I've always used managed Kubernetes solutions such as GKE and EKS, which mostly avoid the headaches that you get when you host your own cluster. I didn't have any experience with setting up Kubernetes from scratch, so this was my preferred option for this project.

Back in 2020, when I migrated to Kubernetes, the only three good options for managed Kubernetes were Google Kubernetes Engine, AWS Elastic Kubernetes Service and Azure Kubernetes service. In the last two years several smaller Cloud Platform companies like DigitalOcean and Scaleway have introduced their own managed Kubernetes offering.

When choosing between managed Kubernetes offerings you should mostly be comparing pricing and reliability. The functionality of a kubernetes service is - in theory - the same across providers, making managed kubernetes a commodity product. The only way these providers can compete is by making their service more reliable or cheaper; this is also what you should be looking at when making your decision.

To help compare I have compiled a comparison of the main kubernetes offerings as of 2023. The chart assumes one cluster with one cluster node. The cluster node is the cheapest option available, on-demand. Costs are per month. Location is western Europe.

Provider	Cluster fee	Node cost	Total cost	Node details	Free offering
AWS	$73	$13.43	$86.43	t4g.small (2vCPU, 2GB RAM)	No
Google Cloud	$0	$13.46	$13.46	e2.small (2vCPU, 2GB RAM)	$300 free credits
Azure	$0	$0	$36,21	B2s (2vCPU, 4GB RAM)	$200 free credits for first month
DigitalOcean	$0	$12	$12	Basic Node (1vCPU, 2GB RAM)	$200 credit for first 60 days.
Scaleway	$0	$7.02	$7.02	DEV-1S (2vCPU, 2GB RAM)	No SLA on the node availability.

Because of the relatively high costs associated with the managed kubernetes solutions I went looking for alternatives. The alternative I ended up going for was K3s: a light-weight kubernetes distribution, containing everything you could reasonably need for a humble single-node cluster. It's advantage over other distributions is that is built with production deployments in mind. There are other single-node kubernetes distributions like minikube or kind, but those are mostly meant to be used for local development.

For me the main advantage of using K3s was that it enabled me to use low-cost VPS providers, like Hetzner. Instances on AWS and GCP often come with a large price premium, presumably because of all the included cloud-related features. VPS providers can afford lower prices, exactly because they don't have to take care of all those secondary features. On Hetzner I am now paying €5.32 monthly for a VPS with 2vCPUs and 4 GB RAM, which costs less than half of what the cheapest managed kubernetes costs, plus it comes with a machine that is twice as powerful.

Of course using K3s is not all sunshine. It does come with the downsides of self-hosted Kubernetes that I was trying to avoid originally. That being said, it's still infinitely easier to set up a K3s server than it is to do it from scratch. In the next section I will explain the process of getting K3s up and running on a VPS.

1.2. Prerequisites

To get started you will need a VPS that has at least 1 full vCPU available and at least 2GB of RAM. In my experience K3s takes up about 600MB of RAM right from the start, with just the K3s services running. In this guide I am using Debian 11 (Bullseye), so be aware that the process might be different if you are using another linux distribution on your server. Make sure that you have SSH public key authentication set up on this machine before continuing.

1.3. Installing K3s

First of all we'll have to install K3s on the master node (and in this case alse the only node). This is a relatively simple process, and it could be even simpler if you didn't do any custom k3s configuration as we will in the next steps. Get started by SSHing into your VPS.

Installing K3s can be done by using the official installation script hosted on the K3s website. We will download it and run it directly on the machine,

curl -sfL https://get.k3s.io | sh -

If something goes wrong with the installation process from this point onwards, you can always recover by running /usr/local/bin/k3s-killall.sh and then reinstalling K3s as described above.

Back when I configured my cluster I ran into this particular issue. I was able to solve it as suggested in the github comments, by installing the apparmor package.

sudo apt update
sudo apt install -y apparmor apparmor-utils

A default K3s installation has the traefik ingress controller and a cloud controller running by default. We will disable this functionality, as we do not use the ingress resource, and we are not running on AWS, GCP or Azure. As a result the memory usage of K3s will decrease by a bit.

The K3s installation script that we ran earlier has created a systemd unit file. By modifying the command line arguments in there we can disable the ingress and cloud controllers.

# Modify K3s unit file
SERVICE_FILE="/etc/systemd/system/k3s.service"
LINE=$(awk '/ExecStart=\/usr\/local\/bin\/k3s/{print NR}' $SERVICE_FILE)
cat "$SERVICE_FILE" | head -n $(expr $LINE - 1) > $SERVICE_FILE
tee -a "$SERVICE_FILE" << EOF
ExecStart=/usr/local/bin/k3s server \
    --node-name "olivine" \
    --disable "traefik" \
    --default-local-storage-path "/storage" \
    --disable-cloud-controller
EOF

# Reload systemd unit files and restart the K3s service with updated configuration
systemctl daemon-reload
systemctl restart k3s

This next part is optional, but might be useful. If you want to administer your cluster from the master node itself, you can do that by installing kubectl and copying the kube config file to your user folder.

# Install kubectl, see https://Kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
mv kubectl /usr/local/bin
sudo chmod +x /usr/local/bin/kubectl

# Setup kubeconfig, see https://rancher.com/docs/k3s/latest/en/cluster-access/
# This is only needed if you want to run kubectl commands from the master node
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

Now that the master node is set up, we can configure our workstation to get remote access to the cluster.

On the workstation / Kubernetes client:

# Install kubectl, see https://Kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo mv kubectl /usr/local/bin
sudo chmod +x /usr/local/bin/kubectl

# Install helm, we will need this later to install some useful charts
HELM_VERSION=3.6.3
wget "https://get.helm.sh/helm-v$HELM_VERSION-linux-amd64.tar.gz"
tar -zxvf "helm-v$HELM_VERSION-linux-amd64.tar.gz"
sudo mv linux-amd64/helm /usr/local/bin/helm
rm helm-v$HELM_VERSION-linux-amd64.tar.gz
rm -r linux-amd64

# Setup kubeconfig. Sed is used there to set the correct cluster address. We grab our kubeconfig file from the master node, 
# which means that the cluster address is set to 127.0.0.1. That address needs to be replaced with the actual IP address of
# the master node to get the config file working on the workstation.
MASTER_NODE_IP=78.47.250.115
mkdir -p ~/.kube/ && ssh root@$MASTER_NODE_IP "cat /etc/rancher/k3s/k3s.yaml" \ | sed "s|server: https://127.0.0.1:6443|server: https://$MASTER_NODE_IP:6443|g" > ~/.kube.config

Once that is done we have our cluster ready, and we can manage it using kubectl from our workstation.

1.4. Configuring local storage

For permanent file storage we still need to configure a storage provider. Container storage is ephemeral, so any permanent storage needs to be provided externally. This is usually done by a storage provider. For example, if you have a cluster on AWS, there is a provider you can install that will make it possible to mount EBS volumes and EFS drives on a container.

Since we are running K3s on hetzner, we don't have that option. We do have the option of using the local storage of the VPS itself. Rancher (the company behind K3s), created a Kubernetes provisioner that allows us to do exactly that: the "Local Path Provisioner".

Installing it requires only one kubectl apply command:

kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.24/deploy/local-path-storage.yaml

Using the new provisioner we can now create persistent volumes that are mounted to a path on the local file system of the VPS:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-pv
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /example/path
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - server_name

1.5. Switching over

After all the required infrastructure was installed I switched over to the new cluster. This required some preparation. A day before I started the switch-over I set the Time-To-Live (TTL) of all the important DNS records to one minute. This step was necessary to propagate the DNS changes upstream. These changes, once fully propagated, allowed me to fall back to my old VPS within a minute. If something went wrong I could recover by just resetting the DNS records.

During the migration I moved all the DNS records to the new cluster. Luckily I didn't need to make use of the fallback measures, so after a day I set the DNS records back to their original TTL values.

1.6. Troubleshooting

Early on the cluster had some issues that required some thorough detective work to fix. One day my cluster become unreachable. It was still running according to the hetzner dashboard, but I just couldn't connect to it over SSH, nor could I reach any of the hosted services. After restarting the server I could connect over SSH again. I quickly found out that CPU usage was going through the roof - the culprit being an iptables process.

Apparently, as some other people had discovered on the k3s github issues, the iptables package that came with the debian buster installation on my server had a bug that caused a huge amount of duplicate iptables rules to be generated.In my case, over 40,000 rules were added for the short time my cluster had been running. The processing of all these rules lead to the very high CPU usage caused by iptables.

The solution itself was dead simple. Upgrading the debian install from buster to bullseye (which came with an iptables package including a bugfix) completely fixed the issue.

2. Installing the cluster infrastructure

An empty cluster is rather useless of course, so we're going to place some basic infrastructure services on it:

Certificate manager: All L7 traffic to the cluster should be encrypted. I use Lets Encrypt for this, but those certificates need to be refreshed every 90 days (trust me, this is a lot more annoying than it sounds like). We can automate this using certmanager.
Docker registry: Most of our cluster interaction will involve building images for services that will run on the cluster. It makes sense to set up a private docker registry for ease of use, and to keep any personal code private.
Git server: This one might not make much sense with services like github, but as with the docker registry, sometimes you have code that you want to keep private. Also, its also good to back up any external repositories you might have.
Reverse proxy: Once we have some services set up we will need a way to get incoming traffic to the correct services. Most people do this with the Kubernetes ingress resource, but at the time of writing this that only works for L7 protocols (i.e. HTTP). I also needed to reverse proxy some stuff at the transport layer so ingress was not a possibility. Instead I chose to set up an nginx reverse proxy which servers as a gateway for all incoming traffic. Nginx also does SSL termination for us.

2.1. Certificate manager

Having to renew certificates is annoying as hell. To avoid having to do this every 90 days I set up certificate manager to renew the certificates for me. Certificate manager requires a DNS provider with API access, so it can automatically configure the DNS to make the ACME challenge succeed. Because of this I used cloudflare as a DNS provider, seeing as it is free and has a DNS API.

The installation is pretty straightforward with helm:

kubectl create namespace cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.4.0 \
  --set installCRDs=true

We then need to create a cloudflare API key for certmanager. Make sure this API key has the appropriate access rights, so you don't end up in too much trouble if it is compromised. The key needs to be stored in a kubernetes secret:

apiVersion: v1
kind: Secret
metadata:
    name: cloudflare-api-token-secret
    namespace: cert-manager
type: Opaque
stringData:
    api-token: { { .Values.cloudflare.secret_token }}

We also need a ClusterIssuer. The cluster issuer will watch our cluster for certificate objects and renew those if necessary.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
    name: letsencrypt-production
spec:
    acme:
        # The ACME server URL
        server: https://acme-v02.api.letsencrypt.org/directory
        # Email address used for ACME registration
        # Let's Encrypt will use this to contact you about expiring
        # certificates, and issues related to your account.
        email: example@test.com
        # Name of a secret used to store the ACME account private key
        privateKeySecretRef:
            name: letsencrypt-production
        # Enable the DNS-01 challenge provider
        solvers:
            - dns01:
                cloudflare:
                    email: example@test.com
                    apiTokenSecretRef:
                        name: cloudflare-api-token-secret
                        key: api-token

Once that is all set up, we can start creating certificates in our cluster:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-com
  namespace: infra
spec:
  duration: 2160h0m0s
  renewBefore: 720h0m0s
  secretName: example-com-tls
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
  dnsNames:
  - '*.example.com'
  - example.com

This certificate object will create a wildcard cert that is renewed 30 days before expiration. The resulting certificate will be stored in the example-com-tls kubernetes secret.

This secret can then be mounted as a volume on any one of your pods:

spec:
  template:
    spec:
      volumes:
        - name: cert-example-volume
          secret:
            secretName: example-com-tls

For more precise instructions, you can take a look at the official documentation.

2.2. Reverse proxy

Every service in the k3s cluster is only accessible from the VPS on which it is deployed, with exception of the nginx reverse proxy which is publicly accessible. This is to make sure I have a good overview of what exactly is accessible from the public internet. All traffic flowing into the cluster has to go through the reverse proxy. It serves as a gate from which I can control and secure cluster traffic ingress. One example of this is the cluster-wide HTTP-to-HTTPS redirection. Unsecure HTTP traffic is not allowed, it is instead redirected to the more secure HTTPS.

The reverse proxy uses kubernetes service DNS names to point to services instead of the clusterIP of the services. ClusterIPs are randomly assigned and can therefore change if we accidentally delete the service, the service DNS name wont. The DNS name comes in the form of service-name.namespace-name.

As you've seen in the certmanager setup, we use LetsEncrypt to provide our certificates. These certificates need to be renewed every 90 days, which cert-manager already takes care of. The certificates are in Kubernetes secrets which are mounted in the reverse-proxy pod filesystem. Nginx won't pick up on any changes in the mounted certificates, however. We need to manually reload nginx to get nginx to serve the renewed certificate. We do this using an extra cron process in the container. To maintain both the nginx process and the cron process in one container we use supervisor. For more details, please check the docker documentation.

2.3. Docker registry

To keep the code for my personal projects private I set up a docker registry. This was a lot easier than it sounds. Docker (the company) itself provides a docker image that can be used to self-host your own registry. Hosting the registry requires nothing more than just running the docker container.

Because we are using kubernetes we do need some extra configuration. The registry needs a place to store its images. We can use a localpath PV (see previous header) to take care of that. We also want to password protect the registry so not just anybody can use it. This password can be generated like this:

htpasswd -B htpasswd username

We then provide the registry container with the following configuration file to require the password:

version: 0.1
storage:
  filesystem:
    rootdirectory: /var/lib/registry
auth:
  htpasswd:
    realm: basic-realm
    path: /etc/registry/htpasswd
http:
  addr: 0.0.0.0:5000
  host: https://docker.example.com

In order for kubelet to automatically be able to pull images from the cluster, we will need to create a .dockercfg file on all cluster nodes:

tee -a "/var/lib/kubelet/config.json" << EOF
{
  "auths": {
    "your.registry.com": {
      "auth": "base64(user:password)"
    }
  }
}
EOF

As mentioned before I am using a nginx reverse proxy to make the docker registry available outside of the cluster. The nginx configuration for requires a bit more work than just a plain http reverse proxy. We need to use the client_max_body_size and the chunked_transfer_encoding directives to make sure the uploads are correctly proxied.

# Reverse proxy for docker.example.com
server {
    listen 443 ssl;
    listen [::]:443 ssl;

    # Disable any limits to avoid HTTP 413 for large uploads
    client_max_body_size 0;

    # Required to avoid HTTP 411: see Issue #1486 (https://github.com/moby/moby/issues/1486)
    chunked_transfer_encoding on;

    server_name docker.example.com;

    ssl_certificate /etc/nginx/ssl/example.com/tls.crt;
    ssl_certificate_key /etc/nginx/ssl/example.com/tls.key;
    include /etc/nginx/snippets/ssl-params.conf;

    location / {
        proxy_pass http://docker-registry.infra:5000;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 900;
    }
}

With all this configuration we can now use our own docker registry when deploying containers to the cluster. Be sure to use the internal cluster DNS, otherwise it will use up costly bandwidth by making traffic go over the internet, instead of just remaining local.

2.4. Git server

For the git server I took a bit of an unorthodox approach. Usually I would just create a bare repository on a host and then access it over SSH. In the case of our cluster that approach would not work, however. As a matter of policy I wanted every service to run within the cluster. The only thing actually running on the VPS should be the k3s processes. To conform to my own rules I created a git server docker image:

FROM nginx:1.21.1

RUN apt-get update -y && apt-get install -y git gitweb fcgiwrap spawn-fcgi supervisor perl openssh-server

# Gitweb
COPY gitweb.conf /etc/gitweb.conf
COPY nginx.conf /etc/nginx/conf.d/default.conf

# Git over ssh
RUN mkdir /var/run/sshd
RUN adduser git && \
    cd /home/git && \ 
    mkdir .ssh && \
    chmod 700 .ssh && \
    chown git:git .ssh
RUN mkdir /repo && \
    ln -s /repo /home/git/ && \
    chown git /home/repo

COPY supervisor.conf /etc/supervisor/conf.d/supervisord.conf

CMD ["/usr/bin/supervisord"]

In this dockerfile we use supervisord to run both an ssh server and a gitweb HTTP interface. Git repositories can be created and pulled over port 22 on the container, while the web interface can be accessed over port 80. You might notice that this creates a problem in our scenario. We already use port 22 on the VPS host for general administration. We cannot redirect this port to the git container. Instead we will redirect a different port to the git container. We do this by using the nginx stream module:

load_module /usr/lib/nginx/modules/ngx_stream_module.so;


stream {
    server {
        listen 2222;
        proxy_pass git-server.infra:22;
    }
}

3. Managing the cluster

3.1. Using Helm for change management

Managing more than one kubernetes application with just kubectl tends to get a bit unwieldy. It is often better to go for a tool such as helm to help with deployments. Helm allows you to create things called "charts", which are basically groups of kubernetes manifests that can be installed, updated and removed as one unit. Instead of many kubectl commands, now you only need to run helm install example-chart ./chart-folder. Helm even has a basic templating language for the purpose of adding variable parameters in the charts.