vyking-devops

End-to-end DevOps demo for the Vyking practical task. A local Kubernetes cluster (k3d, multi-node), GitOps continuous delivery (Argo CD installed by Terraform), application packaging (Helm), and a stateful workload (MySQL) with verifiable automated backups.

Architecture

                        github.com/dayanstef/vyking-devops
                                       |
                                       | pulled by Argo CD
                                       v
  +--------------------- k3d (1 server + 3 agents) ----------------------+
  |                                                                      |
  |  +---- argocd ns ----+                                               |
  |  |   Argo CD (Helm)  |   installed by Terraform helm_release         |
  |  +---------+---------+                                               |
  |            | manages                                                 |
  |            +-> Application "infrastructure" (path: infrastructure/)  |
  |            |      +- Bitnami MySQL (subchart dependency)             |
  |            |      +- Backup PVC (separate from MySQL data)           |
  |            |      +- CronJob mysql-backup (every 5 min)              |
  |            |                                                         |
  |            +-> Application "applications"   (path: applications/)    |
  |                   +- Backend Deployment (Go API)                     |
  |                   +- Backend Service                                 |
  |                   +- Frontend Deployment (Nginx + static HTML)       |
  |                   +- Frontend Service                                |
  |                   +- Ingress (Traefik, vyking.localhost:8080)        |
  |                                                                      |
  +----------------------------------------------------------------------+

Request flow: browser -> Traefik Ingress -> Service frontend -> nginx proxies /api/* to Service backend -> Go API -> Service mysql. Items posted from the UI persist in MySQL; every 5 minutes the CronJob writes a mysqldump gzip to a separate PersistentVolumeClaim with retention of the last 20 backups.

Repository layout

vyking-devops/
├── apps/                 # Source for the FE + BE container images
│   ├── backend/          # Go API (go.mod, main.go, main_test.go, Dockerfile)
│   └── frontend/         # Static SPA served by Nginx (index.html, nginx.conf, Dockerfile)
├── terraform/            # IaC: providers, Argo CD install, App-of-Apps
├── infrastructure/       # Helm chart: Bitnami MySQL subchart + backup PVC + CronJob
├── applications/         # Helm chart: backend + frontend Deployments/Services/Ingress
├── Makefile              # All ops behind named targets
├── .gitignore .editorconfig .dockerignore
└── README.md             # This file

Prerequisites

Tool	Min version	Install (macOS)
Docker	24+	https://docs.docker.com/desktop/install/mac-install/
k3d	5.6+	brew install k3d
kubectl	1.28+	comes with Docker Desktop, or brew install kubectl
helm	3.13+	brew install helm
terraform	1.6+	brew install terraform
make	any	(system)

Docker Desktop must be running before any make target.

Quick start

make all

That target runs, in order: make cluster -> make images -> make tf-apply. Each step is also runnable individually below.

Step by step

1. Multi-node cluster

make cluster
kubectl get nodes -o wide

1 control-plane node and 3 worker nodes, all Ready.

2. Build and push the application images

The images are already published as public packages at ghcr.io/dayanstef/vyking-backend and ghcr.io/dayanstef/vyking-frontend, so the cluster can pull them anonymously. If you want to rebuild and push your own (e.g., to a fork), you need a GitHub Personal Access Token with write:packages:

echo $GHCR_PAT | docker login ghcr.io -u <your-user> --password-stdin
make images TAG=v0.1.3 REGISTRY=ghcr.io/<your-user>

After the first push, set both packages to public at https://github.com/users/<your-user>/packages.

3. Terraform: install Argo CD and create the App-of-Apps

make tf-apply

This installs Argo CD from the official Helm chart and creates two Argo CD Application resources that monitor infrastructure/ and applications/ in this repo.

4. Watch Argo CD reconcile

make app-status

Within 2-3 minutes both apps should be Synced + Healthy.

5. Argo CD UI

make argocd-ui          # http://localhost:8443
make argocd-password    # prints the admin password

Username admin. Both apps render as a graph of green resources.

6. Use the application

make app-ui   # port-forwards http://localhost:8080

Open http://localhost:8080. Type something, press Add. Items persist in MySQL.

Via curl:

curl -sS -H 'content-type: application/json' -d '{"text":"hello"}' http://localhost:8080/api/items
curl -sS http://localhost:8080/api/items

7. Backups

The CronJob runs every 5 minutes. To skip the wait:

make backup-now
make backup-list

You should see mysql-YYYYMMDD-HHMMSS.sql.gz files. Retention keeps the most recent 20.

Integrity check on the latest backup:

POD=$(kubectl -n vyking get pods -l job-name -o jsonpath='{.items[-1:].metadata.name}')
kubectl -n vyking exec "$POD" -- /bin/bash -c 'BACKUP=$(ls -t /backups/mysql-*.sql.gz | head -1); gunzip -t "$BACKUP" && echo "$BACKUP is a valid gzip"'

8. Demonstrate GitOps self-heal

Drift the cluster manually; Argo CD reverts it:

kubectl -n vyking scale deploy backend --replicas=1
sleep 20
kubectl -n vyking get deploy backend

Back to READY 2/2 because selfHeal is on.

9. Tear it all down

make clean

Runs make tf-destroy then make cluster-delete.

Why these design choices

• gavinbunney/kubectl_manifest for the Argo CD Application CRs. The HashiCorp Kubernetes provider's kubernetes_manifest does schema validation at plan time, so a fresh terraform apply fails on first run when Argo CD's CRDs do not yet exist. kubectl_manifest applies raw YAML at apply time and works in a single shot. Single-apply bootstrap was a priority.
• Single vyking namespace for app + DB. Kubernetes Secrets are namespace-scoped. Co-locating MySQL, FE, BE, and the CronJob in one namespace removes the need for secret reflectors. The App-of-Apps separation remains logical (infra vs apps), implemented as two Argo CD Applications watching different Git paths.
• infrastructure/ is an umbrella Helm chart. Bitnami MySQL is declared as a subchart dependency. The backup PVC and CronJob are templates alongside it. A single Argo CD Application can sync the whole thing.
• Backups on a separate PVC. Per the task: backup storage must not share volume with MySQL data.
• Custom images pushed to ghcr.io. Lets the verification step exercise a real FE -> BE -> DB flow rather than a static-only frontend.
• bitnamilegacy/ registry override on the MySQL subchart. In August 2025 Bitnami moved their free public images out of docker.io/bitnami/ into a paid catalog. The 12.x chart still defaults to the old path. The override in infrastructure/values.yaml points the chart at docker.io/bitnamilegacy/mysql so the cluster can pull anonymously.

Troubleshooting

Symptom	Likely cause / fix
make cluster errors: port 8080 in use	Stop the other process, or HOST_PORT=8081 make cluster
make tf-apply: CRD not found	Rare race; re-run make tf-apply. The 15s time_sleep usually prevents it.
Argo CD app stuck OutOfSync	Click Sync in the UI, or kubectl -n argocd patch app <name> --type=merge -p '{"operation":{"sync":{}}}'
infrastructure shows OutOfSync but Healthy, with the StatefulSet flagged	Cosmetic Argo CD behavior under ServerSideApply for StatefulSets. argocd app diff infrastructure returns empty, all child resources are Synced, the workload runs. Safe to ignore.
Pods ImagePullBackOff on Bitnami MySQL	Bitnami moved free images to bitnamilegacy/. The values file already overrides this; confirm it persisted.
Pods ImagePullBackOff on backend/frontend	If you forked and pushed to your own ghcr, set the packages public, or configure an imagePullSecret.
vyking.localhost does not resolve	Add 127.0.0.1 vyking.localhost to /etc/hosts, or use make app-ui.
Backend CrashLoopBackOff	MySQL still booting; wait 60s. kubectl -n vyking logs deploy/backend.

Appendix: keep ghcr packages private

If you fork this repo and want to keep your ghcr packages private, create an imagePullSecret and reference it:

kubectl -n vyking create secret docker-registry ghcr-pull \
  --docker-server=ghcr.io \
  --docker-username=<your-user> \
  --docker-password=$GHCR_PAT

Add to applications/values.yaml:

imagePullSecrets:
  - name: ghcr-pull

And update each Deployment template to honour .Values.imagePullSecrets.