Cluster Bootstrap Overview

Cluster Bootstrap Overview#

The bootstrap is split into multiple stages:

  • Bringing up the base infrastructure like the hardware, the operating system and a Kubernetes cluster. This provides a functional but empty cluster.

    See the the folder /ansible regarding a playbook to automate this process.

    • Install the base operating system. This is NixOS as configured in /nixos. This also deploys and configures the k0s distribution.

    • Join the Nodes into the Cluster. This is done via Ansible. See /ansible.

    • Install basic extensions into the cluster. At the moment this is Cilium. This is deployed during the cluster creation by Ansible and then updated later by Flux.

  • bootstrap is about making FluxCD operational and providing a Git server so that FluxCD can start to take over. This group does contain a minimal set of applications so that FluxCD is able to work.

    • Bootstrap the FluxCD system according to the instance README.

    • Gitea - This is an internal Git server with a small footprint. It hosts a clone of this repository, so that FluxCD can access it.

    • FluxCD - This is the continuous delivery system in use. It does reconcile the cluster state based on the state in this repository.

  • cluster-base-apps groups together the applications which are required so that the cluster is fully able to manage itself. The following applications are in this group:

    • Vault - This is the backend for the secrets handling, it is used to provide secrets via External Secrets.

    • External Secrets - This is used to map secrets from external sources into the cluster. Currently this is used to map secrets from the cluster internal Vault instance.

    • Gitlab - This is used as the automation engine to host and build software artifacts. It also serves as the source for the GitOps approach.

      • PostgreSQL - The bundled database. Intentionally kept, so that there is no dependency into a central database service.

      • MinIO - The bundled S3 compatible storage interface. Intentionally kept, so that there is no dependency into a central S3 compatible service.

    • VolSync - This is used to backup and restore persistent volumes from an external S3 compatible source.

  • cluster-apps are all other services and applications.

The general principle is that a stage is only allowed to depend on the previous stages, but never on anything from the following stages.

This is also the reason why the Gitlab deployment has its own database and minio storage.

Secrets bootstrapping#

The applications in cluster-base-apps cannot yet make usage of Vault. This means that everything needed to restore Vault has to be made available via initial SOPS based secrets.

  • flux-system.sops-age - SOPS Age key. The cluster requires this to decrypt the SOPS secrets.

  • secrets.backup-repo - Secret to encrypt and decrypt the backup repository.

  • secrets.backup-s3 - The S3 access credentials to store and restore the backup snapshots.

Pending#

  • Vault needs a solution to automatically unseal itself.

Gitea bootstrapping#

Gitea is bootstrapped in the following steps:

  1. The application itself is deployed by the bootstrap folder.

  2. The repository is created by a Job which is also deployed from the manifests in the bootstrap folder.

  3. The repository content is pushed in manually.

  4. As part of the regular cluster-apps deployment a Receiver for FluxCD is set up together with the webhook configuration in Gitea, so that every future push to the repository will trigger the webhook to FluxCD.

Improvements#

The Gitlab instance is heavy for code and artifact hosting. Forgejo is being integrated as a more lightweight replacement.

Options to further reduce resource usage include external S3 storage and initially deploying from there, or deploying a lightweight git server plus a registry for the required artifacts (mainly container images).