[feature]: Add log aggregation to the kube-prometheus-stack observability setup

[devantler]

Summary

Extend the existing kube-prometheus-stack deployment to include centralized log aggregation, enabling Kubernetes audit logs and container logs to be queryable in a self-hosted Grafana instance.

Motivation

• The monitoring stack is currently metrics-only (Prometheus + Alertmanager). Grafana is disabled (grafana.enabled: false).
• Kubernetes audit logs (added in feat: add Talos security hardening patches #1506) are written to files on the EPHEMERAL partition — only accessible via talosctl on individual nodes.
• No centralized log search or correlation between logs and metrics exists.
• A self-hosted Grafana with Loki would enable audit log investigation, pod log search, and metric/log correlation — all from one UI.

Architecture

kube-prometheus-stack does not include Loki or Alloy as sub-charts. However, Grafana is a sub-chart that can be enabled. The approach is:

1. Enable Grafana in kube-prometheus-stack via grafana.enabled: true + datasource configuration
2. Deploy Loki as a separate HelmRelease (Grafana Helm repo) for log storage/query
3. Deploy Alloy as a separate HelmRelease (Grafana Helm repo) as the log collector

Note: Promtail is deprecated (EOL March 2026). Grafana Alloy is the official successor — an OpenTelemetry-based collector for logs, metrics, and traces.

┌─────────────────────────────────────────────────┐
│              kube-prometheus-stack               │
│  ┌───────────┐  ┌─────────────┐  ┌───────────┐ │
│  │ Prometheus │  │ Alertmanager│  │  Grafana   │ │
│  │  (metrics) │  │  (webhooks) │  │  (UI/viz)  │ │
│  └─────┬─────┘  └─────────────┘  └──┬────┬───┘ │
│        │                             │    │     │
└────────│─────────────────────────────│────│─────┘
         │  ┌──────────────────────────┘    │
         │  │  datasource: prometheus       │ datasource: loki
         │  │                               │
   ┌─────┴──┴───┐                    ┌──────┴──────┐
   │  Prometheus │◄── scrape ──┐     │    Loki     │
   │   (metrics) │             │     │   (logs)    │
   └─────────────┘             │     └──────▲──────┘
                               │            │ push
                         ┌─────┴────────────┴─────┐
                         │      Grafana Alloy      │
                         │  (DaemonSet collector)  │
                         │  • container logs       │
                         │  • audit log files      │
                         └─────────────────────────┘

Proposed Solution

1. Enable Grafana in kube-prometheus-stack

Update helm-release.yaml values:

grafana:
  enabled: true
  additionalDataSources:
    - name: Loki
      type: loki
      url: http://loki-gateway.monitoring.svc.cluster.local
      access: proxy

2. Deploy Loki (separate HelmRelease)

Deploy Loki in monitoring namespace as a companion to kube-prometheus-stack. Use the grafana/loki Helm chart in single-binary or simple-scalable mode (homelab scale).

3. Deploy Alloy (separate HelmRelease)

Deploy Grafana Alloy as a DaemonSet to collect:

• Container logs from all pods (standard Kubernetes log collection)
• Audit log files from control-plane nodes by mounting /var/log/kubernetes/audit/ (host path from Talos audit-logging patch)

4. Optional: Audit webhook backend

Add audit-webhook-* API server flags to talos/cluster/audit-logging.yaml for real-time streaming to Loki (in addition to the file backend). This is optional — Alloy tailing the audit log files achieves the same result with better resilience.

Acceptance Criteria

• Grafana enabled in kube-prometheus-stack with Prometheus + Loki datasources
• Loki deployed as a separate HelmRelease in monitoring namespace
• Alloy deployed as a DaemonSet HelmRelease collecting container logs + audit logs
• Kubernetes audit logs queryable via LogQL in Grafana
• Container logs from all namespaces searchable in Grafana
• Existing Prometheus metrics remain accessible in Grafana
• CiliumNetworkPolicy allows Alloy → Loki and Grafana → Loki traffic

Additional Context

• Current kube-prometheus-stack chart: v84.5.0 (prometheus-community)
• Current audit logging setup: talos/cluster/audit-logging.yaml (file backend, 30-day rotation)
• Helm charts needed: grafana/loki, grafana/alloy
• The file-based audit backend should be kept as the reliable primary — Loki provides the search/query layer on top
• Reference: Grafana Alloy docs, Loki docs

[devantler]

🤖 Generated by Platform AI Assistant — an automated AI assistant for this repository.

Thoughtful, well-scoped proposal. A few notes from checking the current manifests, in case they help when implementing:

• Chart version moved on. kube-prometheus-stack is now 85.1.1 (the issue references v84.5.0) — k8s/bases/infrastructure/controllers/kube-prometheus-stack/helm-release.yaml:14. The "enable Grafana" approach is unaffected.
• Enabling Grafana reverses a documented decision. That HelmRelease carries an inline rationale for grafana.enabled: false — "alerting-only … operators look at logs / kubectl directly" — and runs Prometheus with no persistence (storageSpec: {}, emptyDir, 14d/5GiB). This is a deliberate "no UI, no persistence" stance, so the PR (and docs/dr/alerting.md) should note it's intentionally superseded.
• Storage is the main open design question. Loki needs durable storage for logs to outlive pod restarts. Per .github/instructions/talos-patches.instructions.md, local Talos disk is ephemeral while prod uses Hetzner CSI volumes, so the storage backend differs per environment — single-binary filesystem on emptyDir would drop logs on restart (same trade-off Prometheus accepts, but audit-log retention may warrant a PVC or object store).
• Network policy is largely already covered. The existing allow-monitoring CiliumNetworkPolicy (networkpolicy.yaml, endpointSelector: {}) already permits intra-namespace ingress + egress, so Alloy→Loki and Grafana→Loki are allowed if all three land in the monitoring namespace. The net-new networking is mainly any Loki egress to external object storage; Alloy's access to audit/container log files is a hostPath concern, not a network one.
• Structure / prereq. Loki and Alloy are HelmReleases → they belong under k8s/bases/infrastructure/controllers/<component>/ (infrastructure-controllers Flux layer), alongside kube-prometheus-stack. There's no Grafana HelmRepository in the repo yet, so adding one (https://grafana.github.io/helm-charts, shared by both charts) is a prerequisite.
• Consider landing it incrementally — one concern per PR: (1) enable Grafana + Loki + datasource, (2) Alloy for container logs, (3) audit-log file tailing. Each is independently verifiable in the CI cluster test.

[devantler]

🤖 Generated by the Daily AI Assistant

Labelling this roadmap — in a first-ever platform strategy review, this is the observability epic. Both prior milestones (Platform – Reimagined, Kubestronaut Journey) are 100% complete, so the platform's two live roadmap themes going forward are:

1. Observability — centralized logs (this issue): the stack is metrics-only today (grafana.enabled: false); enabling Grafana + Loki + Alloy gives audit-log search, pod-log search, and metric/log correlation in one UI. The 3-phase plan in the description (enable Grafana + Loki datasource → Loki → Alloy) is a clean, independently-shippable decomposition — each phase can land as its own draft PR.
2. Supply-chain verification — Supply-chain: cosign-verify the infrastructure Flux OCI artifact (extend #1559 beyond apps) #1570: extending the cosign verification from feat(apps): verify cosign signatures on app OCI artifacts #1559 (apps) to the infrastructure Flux artifact.

No scope change proposed here — just elevating this well-specified, maintainer-filed request to the roadmap so it's tracked as the observability north star. The roadmap label + a tight 2-epic set keeps the roadmap current rather than long and stale.