Setting up K3s all-in-one Development Environment

This documentation guides you through creating an all-in-one development environment using a single K3s cluster (ArgoCD is deployment in the same cluster as a K8TRE dev deployment).

These instructions are also used for automatically testing K8TRE in GitHub actions. For more hands-on instructions with explanations see k3s.md.

Prerequisites

This assumes you already have a Ubuntu 24.04 virtual machine.

Configure and install K3s

Create a K3s configuration file, then install K3S. Although most commands can be passed to the command line installer it is more convenient to define them in a K3S configuration file.

sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml << EOF
node-name: k8tre-dev
tls-san:
  - k8tre-dev
cluster-init: true
disable: traefik
EOF

curl -sfSL https://get.k3s.io | INSTALL_K3S_VERSION=v1.32.4+k3s1 sh -

Setup Kubeconfig file

mkdir -p ~/.kube
sudo cat /etc/rancher/k3s/k3s.yaml > ~/.kube/config

3.2 Enable Required Add-ons

Enable MetalLB and hostpath-storage on both VMs:

# kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
# kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
# kubectl apply -f https://raw.githubusercontent.com/helm/charts/master/stable/hostpath-provisioner/hostpath-provisioner.yaml

# # For MetalLB, provide an IP address range (adjust based on your network)
# kubectl patch configmap config -n metallb-system --type merge -p '{"data": {"config": "address-pools:\n- name: default\n  protocol: layer2\n  addresses:\n  - 192.168.123.50-192.168.123.100\n"}}'

Setup ArgoCD

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.14.11/manifests/install.yaml
sleep 10
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s

Install ArgoCD CLI

sudo curl -sfSL https://github.com/argoproj/argo-cd/releases/download/v2.14.11/argocd-linux-amd64 -o /usr/local/bin/argocd
sudo chmod a+x /usr/local/bin/argocd

Configure ArgoCD

kubectl port-forward svc/argocd-server -n argocd 8080:443 --address 0.0.0.0 &
sleep 1

# Get the initial admin password
ARGOCD_PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)

# Login, disable certificate validation
argocd login localhost:8080 --username=admin --password="$ARGOCD_PASSWORD" --insecure

Mark the current cluster as the ArgoCD dev environment

argocd cluster set in-cluster --label environment=dev --label secret-store=kubernetes
argocd cluster get in-cluster

Configure ArgoCD for Kustomize Helm Overlays by applying a ConfigMap patch:

cat << EOF > argocd-cm-patch.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  kustomize.buildOptions: "--enable-helm --load-restrictor LoadRestrictionsNone"
EOF

kubectl apply -f argocd-cm-patch.yaml

Restart the ArgoCD repo server to apply changes:

kubectl rollout restart deployment argocd-repo-server -n argocd

List CRDs

kubectl get crd

Create secrets for use by External Secrets Operator

uv run ci/create-ci-secrets.py --context $(kubectl config current-context)

Deploy the app-of-apps

Edit the app-of-apps to point to you GitHub fork $GITHUB_REPOSITORY and branch/commit $GITHUB_SHA

sed -i -e "s%/k8tre/k8tre%/${GITHUB_REPOSITORY}%" -e "s%main%${GITHUB_SHA}%" app_of_apps/root-app-of-apps.yaml
git diff

Deploy the app-of-apps

kubectl apply -f app_of_apps/root-app-of-apps.yaml

Wait for app-of-apps to be created

sleep 60
kubectl -n argocd wait --for=jsonpath='{.status.health.status}'=Healthy application root-app-of-apps --timeout=300s

kubectl get appprojects -A
kubectl get applicationsets -A
kubectl get applications -A

Wait for applications in app-of-apps to be created

kubectl -n argocd wait --for=jsonpath='{.status.health.status}'=Healthy application --all --timeout=300s

kubectl get deployment -A
kubectl get daemonset -A
kubectl get crd

TODO: - Check all references to k8tre/k8tre and main are changed in all applications - Check number of applications is as expected - Check all applications are synchronised - Check deployments/daemonsets etc are healthy and ready - Playright test to login and launch workspace