(#971) - add Argo Workflows manifests (staging)

terraform/environments/eks/k8s-manifests-staging/argo-workflow/README.md

@@ -0,0 +1,33 @@
+# Argo Workflows
+
+These manifests install a minimal Argo Workflows control plane into the shared `credreg-staging` namespace. The controller and server components rely on a shared PostgreSQL database (for example, the RDS modules under `terraform/environments/eks`) for workflow persistence.
+
+## Components
+- `externalsecret.yaml` – syncs the AWS Secrets Manager entry `credreg-argo-workflows` into a Kubernetes Secret named `argo-postgres`.
+- `configmap.yaml` – controller configuration that enables Postgres-based persistence; set the host/database here, while credentials come from the synced secret.
+- `rbac.yaml` – service accounts plus the RBAC needed by the workflow controller and Argo server.
+- `workflow-controller-deployment.yaml` – runs `workflow-controller` with the standard `argoexec` image.
+- `argo-server.yaml` – exposes the Argo UI/API inside the cluster on port `2746`.
+- `argo-basic-auth-externalsecret.yaml` – syncs the AWS Secrets Manager entry `credreg-argo-basic-auth` (or similar) to supply the base64-encoded `user:password` string for ingress auth.
+- `argo-server-ingress.yaml` – optional HTTPS ingress + certificate (via cert-manager + Let’s Encrypt) and basic auth for external access to the Argo UI.
+
+## Before applying
+1. **Provision or reference a PostgreSQL instance.** Ensure the desired environment has a reachable database endpoint.
+2. **Create the Secrets Manager entry.** Create `credreg-argo-workflows` (or adjust the `remoteRef.key` value) with JSON keys `host`, `port`, `database`, `username`, `password`, `sslmode`. The External Secrets Operator will sync it into the cluster and the controller/server pick them up via env vars.
+3. **Update `configmap.yaml`.** Set `persistence.postgresql.host` (and database/table names if they differ) for the target environment. Even though credentials are secret-backed, Argo still requires the host in this config.
+4. **Install Argo CRDs.** Apply the upstream CRDs from https://github.com/argoproj/argo-workflows/releases (required only once per cluster) before rolling out these manifests.
+5. **Configure DNS if using the ingress.** Update `argo-server-ingress.yaml` with the desired hostname(s) and point the DNS record at the ingress controller’s load balancer.
+
+## Apply order
+```bash
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/externalsecret.yaml
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/rbac.yaml
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/configmap.yaml
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/workflow-controller-deployment.yaml
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-server.yaml
+# Optional ingress / certificate
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-basic-auth-externalsecret.yaml
+kubectl apply -f terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-server-ingress.yaml
+```
+
+Once the `argo-postgres` secret is synced and the controller connects to Postgres successfully, `kubectl get wf -n credreg-staging` should show persisted workflows even after pod restarts.

terraform/environments/eks/k8s-manifests-staging/argo-workflow/TRIGGER-DUMMY-WORKFLOW.md

@@ -0,0 +1,79 @@
+# Triggering Argo Workflows via port-forward + curl
+
+Use this guide when you need to submit a workflow from your workstation without going through the ingress (no basic auth). The flow is:
+
+1. **Port-forward the Argo server service**
+   ```bash
+   kubectl port-forward -n credreg-staging svc/argo-server 2746:2746
+   ```
+   Leave this running in a separate terminal; it exposes `https://localhost:2746`.
+
+2. **Mint a service-account token**
+   ```bash
+   BEARER=$(kubectl create token argo-server -n credreg-staging)
+   ```
+   Any SA with workflow submit/list permissions works (`argo-server` or `argo-workflow-controller`).
+
+3. **Create the workflow payload**
+   ```bash
+   cat > wf.json <<'EOF'
+   {
+     "workflow": {
+       "apiVersion": "argoproj.io/v1alpha1",
+       "kind": "Workflow",
+       "metadata": { "generateName": "rest-test-" },
+       "spec": {
+      "serviceAccountName": "argo-workflow-controller",
+      "entrypoint": "hello",
+         "templates": [
+           {
+          "name": "hello",
+          "container": {
+            "image": "public.ecr.aws/docker/library/debian:stable-slim",
+            "command": ["bash", "-c"],
+            "args": [
+              "apt-get update >/dev/null && DEBIAN_FRONTEND=noninteractive apt-get install -y cowsay >/dev/null && /usr/games/cowsay \"hello from REST\""
+            ]
+          }
+        }
+      ]
+    }
+     }
+   }
+   EOF
+   ```
+
+4. **Submit the workflow (cURL)**
+   ```bash
+   curl -sk https://localhost:2746/api/v1/workflows/credreg-staging \
+     -H "Authorization: Bearer $BEARER" \
+     -H 'Content-Type: application/json' \
+     -d @wf.json
+   ```
+   A successful response echoes the workflow metadata (UID, status, etc.).
+
+## Trigger via Postman
+
+1. Keep the port-forward running: `kubectl port-forward -n credreg-staging svc/argo-server 2746:2746`.
+2. Generate a Bearer token: `kubectl create token argo-server -n credreg-staging` (copy the value).
+3. In Postman:
+   - **Method:** `POST`
+   - **URL:** `https://localhost:2746/api/v1/workflows/credreg-staging`
+   - **Headers:**
+     - `Authorization: Bearer <token>`
+     - `Content-Type: application/json`
+   - **Body:** raw JSON from `wf.json` (same payload as above).
+4. Disable SSL verification in Postman (Settings → General → “SSL certificate verification” off) or import the Argo server cert so the self-signed TLS passes.
+5. Send the request; you should see the workflow metadata returned. Use the same token for subsequent requests until it expires.
+
+5. **Verify status**
+   ```bash
+   kubectl get wf -n credreg-staging
+   kubectl logs -n credreg-staging wf/<workflow-name>
+   ```
+
+6. **Clean up**
+   - `kubectl delete wf <workflow-name> -n credreg-staging` (optional)
+   - Stop the `kubectl port-forward` process.
+
+> Tip: For ad-hoc tests, this approach avoids ingress auth entirely. When you’re ready to call the public endpoint, add the ingress basic-auth header and keep using the Bearer token in parallel.

terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-basic-auth-externalsecret.yaml

@@ -0,0 +1,18 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: argo-basic-auth
+  namespace: credreg-staging
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: aws-secret-manager
+    kind: ClusterSecretStore
+  target:
+    name: argo-basic-auth
+    creationPolicy: Owner
+  data:
+    - secretKey: auth
+      remoteRef:
+        key: credreg-argo-basic-auth
+        property: auth

terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-server-ingress.yaml

@@ -0,0 +1,46 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: argo-server-cert
+  namespace: credreg-staging
+spec:
+  secretName: argo-server-tls
+  issuerRef:
+    kind: ClusterIssuer
+    name: letsencrypt-prod
+  dnsNames:
+    - argo-staging.credentialengineregistry.org
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: argo-server
+  namespace: credreg-staging
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
+    nginx.ingress.kubernetes.io/ssl-redirect: "true"
+    nginx.ingress.kubernetes.io/auth-type: "basic"
+    nginx.ingress.kubernetes.io/auth-secret: "argo-basic-auth"
+    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
+    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
+    nginx.ingress.kubernetes.io/whitelist-source-range: "71.212.64.155/32,129.224.215.205/32"
+spec:
+  ingressClassName: nginx
+  tls:
+    - hosts:
+        - argo-staging.credentialengineregistry.org
+      secretName: argo-server-tls
+  rules:
+    - host: argo-staging.credentialengineregistry.org
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: argo-server
+                port:
+                  number: 2746

terraform/environments/eks/k8s-manifests-staging/argo-workflow/argo-server.yaml

@@ -0,0 +1,83 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: argo-server
+  namespace: credreg-staging
+  labels:
+    app.kubernetes.io/name: argo-server
+    app.kubernetes.io/part-of: argo-workflows
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: argo-server
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: argo-server
+        app.kubernetes.io/part-of: argo-workflows
+    spec:
+      serviceAccountName: argo-server
+      containers:
+        - name: argo-server
+          image: quay.io/argoproj/argocli:v3.7.7
+          imagePullPolicy: IfNotPresent
+          args:
+            - server
+            - --auth-mode
+            - server
+            - --namespaced
+            - --namespace
+            - credreg-staging
+            - --configmap
+            - workflow-controller-configmap
+          envFrom:
+            - secretRef:
+                name: argo-postgres
+          ports:
+            - containerPort: 2746
+              name: web
+          livenessProbe:
+            httpGet:
+              scheme: HTTPS
+              path: /healthz
+              port: web
+              httpHeaders:
+                - name: Host
+                  value: localhost
+            initialDelaySeconds: 10
+            periodSeconds: 30
+          readinessProbe:
+            httpGet:
+              scheme: HTTPS
+              path: /healthz
+              port: web
+              httpHeaders:
+                - name: Host
+                  value: localhost
+            initialDelaySeconds: 10
+            periodSeconds: 15
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              cpu: 500m
+              memory: 512Mi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: argo-server
+  namespace: credreg-staging
+  labels:
+    app.kubernetes.io/name: argo-server
+    app.kubernetes.io/part-of: argo-workflows
+spec:
+  selector:
+    app.kubernetes.io/name: argo-server
+  type: ClusterIP
+  ports:
+    - name: web
+      port: 2746
+      targetPort: web

terraform/environments/eks/k8s-manifests-staging/argo-workflow/configmap.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: workflow-controller-configmap
+  namespace: credreg-staging
+data:
+  config: |
+    metricsConfig:
+      enabled: false
+      secure: true
+    telemetryConfig:
+      enabled: false
+      secure: true
+    namespace: credreg-staging
+    persistence:
+      archive: true
+      nodeStatusOffload: true
+      postgresql:
+        host: argo-workflows-staging.cwdkv5tua6nq.us-east-1.rds.amazonaws.com
+        port: 5432
+        database: argo_workflows
+        tableName: argo_workflows
+        sslMode: require
+        userNameSecret:
+          name: argo-postgres
+          key: username
+        passwordSecret:
+          name: argo-postgres
+          key: password

terraform/environments/eks/k8s-manifests-staging/argo-workflow/externalsecret.yaml

@@ -0,0 +1,38 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: argo-postgres
+  namespace: credreg-staging
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: aws-secret-manager
+    kind: ClusterSecretStore
+  target:
+    name: argo-postgres
+    creationPolicy: Owner
+  data:
+    - secretKey: host
+      remoteRef:
+        key: credreg-argo-workflows-staging
+        property: host
+    - secretKey: port
+      remoteRef:
+        key: credreg-argo-workflows-staging
+        property: port
+    - secretKey: database
+      remoteRef:
+        key: credreg-argo-workflows-staging
+        property: database
+    - secretKey: username
+      remoteRef:
+        key: credreg-argo-workflows-staging
+        property: username
+    - secretKey: password
+      remoteRef:
+        key: credreg-argo-workflows-staging
+        property: password
+    - secretKey: sslmode
+      remoteRef:
+        key: credreg-argo-workflows-staging
+        property: sslmode