Setting up K3s Development Environment

This documentation guides you through creating a development environment using K3s. Alternative approaches include KinD, vCluster, and others.

Overview

This guide will help you set up two Ubuntu 24.04 VMs with K3s: - A management cluster with ArgoCD installed - A development cluster registered with ArgoCD for application deployment

ArgoCD on the management cluster will deploy and manage applications on the development cluster.

Prerequisites

Ubuntu 24.04 on your local development machine
kubectl (with kustomize)
ArgoCD CLI
Helm
QEMU/KVM and Virtual Machine Manager

Step 1: Creating Virtual Machines

1.1 Create Two VMs

Create two identical VMs (QEMU/KVM based) using Virtual Machine Manager:

Open Virtual Machine Manager
Click "Create new virtual machine"
Select "Local install media" and choose Ubuntu 24.04 ISO
Configure with at least 2 CPUs, 4GB RAM, and 20GB storage
Complete the installation process for both VMs

1.2 Set Hostname and Hosts

Name the VMs vm-mgmt and vm-dev respectively by editing these files:

# On the management VM
sudo nano /etc/hostname
# Change to: vm-mgmt

sudo nano /etc/hosts
# Add: 127.0.1.1 vm-mgmt

sudo hostnamectl set-hostname vm-mgmt
sudo reboot

# On the development VM
sudo nano /etc/hostname
# Change to: vm-dev

sudo nano /etc/hosts
# Add: 127.0.1.1 vm-dev

sudo hostnamectl set-hostname vm-dev
sudo reboot

Step 2: Network Configuration

2.1 Create a Static Network

Create a new network in Virtual Machine Manager with static IP addresses:

Open Virtual Machine Manager
Go to Edit → Connection Details → Virtual Networks → +
Or create the network XML file directly:

<network connections="2">
  <name>static-network</name>
  <uuid>0f73f3fd-38f6-4e72-aa94-7984c2606054</uuid>
  <forward mode="nat">
    <nat>
      <port start="1024" end="65535"/>
    </nat>
  </forward>
  <bridge name="virbr1" stp="on" delay="0"/>
  <mac address="52:54:00:4f:af:6c"/>
  <domain name="xk8tre.org"/>
  <ip address="192.168.123.1" netmask="255.255.255.0">
    <dhcp>
      <range start="192.168.123.2" end="192.168.123.254"/>
      <host mac="52:54:00:c8:e0:2f" name="dev" ip="192.168.123.62"/>
      <host mac="52:54:00:07:36:8a" name="mgmt" ip="192.168.123.52"/>
    </dhcp>
  </ip>
</network>

Note: The IP address 192.168.123.1 refers to your host machine's IP on this virtual network.

2.2 Attach VMs to Network

Edit each VM's configuration
Change the network interface to the newly created static-network
Make sure the MAC addresses match those specified in the network XML
Restart the VMs so they get their new IP addresses

Step 3: Installing and Configuring K3s

3.1 Install K3s

Install K3s on both VMs:

# On both VMs
curl -sfL https://get.k3s.io | sh -
sudo usermod -aG docker $USER
newgrp docker

3.2 Enable Required Add-ons

Enable MetalLB and hostpath-storage on both VMs:

# On both VMs
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
kubectl apply -f https://raw.githubusercontent.com/helm/charts/master/stable/hostpath-provisioner/hostpath-provisioner.yaml

# For MetalLB, provide an IP address range (adjust based on your network)
kubectl patch configmap config -n metallb-system --type merge -p '{"data": {"config": "address-pools:\n- name: default\n  protocol: layer2\n  addresses:\n  - 192.168.123.50-192.168.123.100\n"}}'

Learn more about K3s add-ons.

Step 4: Export and Merge Kubeconfig

4.1 Export Kubeconfig from VMs

# On the management VM
k3s kubectl config view --raw > ~/.kube/config

# On the development VM
k3s kubectl config view --raw > ~/.kube/config

4.2 Copy Kubeconfig Files to Host

# From your host machine
scp <username>@192.168.123.52:~/.kube/config ~/.kube/mgmt-config
scp <username>@192.168.123.62:~/.kube/config ~/.kube/dev-config

4.3 Edit and Merge Kubeconfig Files

Edit each kubeconfig file to rename clusters and users:

# Edit mgmt-kubeconfig.yaml to change:
# - cluster name to "k3s-mgmt" 
# - user name to "admin-mgmt"

# Edit dev-kubeconfig.yaml to change:
# - cluster name to "k3s-dev"
# - user name to "admin-dev"

Then merge them:

KUBECONFIG=~/.kube/config:~/.kube/mgmt-config:~/.kube/dev-config kubectl config view --flatten > ~/.kube/merged-config
mv ~/.kube/merged-config ~/.kube/config
chmod 600 ~/.kube/config

Step 5: Setting Up ArgoCD

5.1 Install ArgoCD on Management Cluster

# Switch to management cluster context
kubectl config use-context k3s-mgmt

# Create namespace
kubectl create namespace argocd

# Install ArgoCD
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for all pods to be ready
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s

5.2 Access ArgoCD UI

# Port-forward ArgoCD server (run this in a separate terminal)
kubectl port-forward svc/argocd-server -n argocd 8080:443 --address 0.0.0.0

# Get the initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Visit https://localhost:8080 to access the ArgoCD UI.

5.3 Configure ArgoCD CLI

# Login to ArgoCD from your host
argocd login localhost:8080

# Or if you've set up /etc/hosts entries:
argocd login mgmt.xk8tre.org:8080

5.4 Register Development Cluster with ArgoCD

# Add the development cluster to ArgoCD
argocd cluster add k3s-dev --name dev --label environment=dev

5.5 Configure ArgoCD for Kustomize Helm Overlays

Create and apply a ConfigMap patch:

cat << EOF > argocd-cm-patch.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  kustomize.buildOptions: "--enable-helm --load-restrictor LoadRestrictionsNone"
EOF

kubectl apply -f argocd-cm-patch.yaml

Restart the ArgoCD repo server to apply changes:

kubectl rollout restart deployment argocd-repo-server -n argocd

Learn more about ArgoCD configuration options.

Step 6: Convenience Configurations

6.1 Context-Switching Aliases

Add to your ~/.bashrc or ~/.zshrc:

# For easy context switching
alias kdev='kubectl config use-context k3s-dev'
alias kmgmt='kubectl config use-context k3s-mgmt'

# Optionally add kubectl aliases
alias k='kubectl'
alias kgp='kubectl get pods'
alias kgs='kubectl get services'

Then source your file: source ~/.bashrc

6.2 Host File Entries

Add the following entries to /etc/hosts on your host machine:

sudo bash -c 'cat << EOF >> /etc/hosts
# VMs running K3s
192.168.123.52 mgmt.xk8tre.org
192.168.123.62 dev.xk8tre.org
EOF'

Note: Be careful not to assign IP addresses that might conflict with other devices on your network.

Step 7: Verification

7.1 Test Cluster Access

# Test management cluster
kmgmt
kubectl get nodes

# Test development cluster
kdev
kubectl get nodes

7.2 Verify ArgoCD Setup

# Check that ArgoCD can communicate with both clusters
argocd cluster list

Additional Resources

Troubleshooting

If you encounter issues:

Check cluster status: k3s kubectl get nodes
View K3s logs: journalctl -u k3s
Check ArgoCD logs: kubectl logs -n argocd deployment/argocd-server
For networking issues, verify the VM network configuration.