AWS Cloudwatch Logs not showing up in HyperDX

[vinfang]

I'm using Cloudstack Helm chart to deploy all the components (HyperDX, Clickhouse, MongoDB, Open Telemetry Collector) into our AWS Kubernetes cluster. We're also using the Open Telemetry helm chart to deploy an agent that will poll our AWS Cloudwatch logs to send to the Clickstack's otel collector.

Following the guide here, https://clickhouse.com/docs/use-cases/observability/clickstack/integrations/aws-cloudwatch-logs, I'm pretty sure we got our open telemetry agent working correctly as the debug logs from the agent pod show it's finding those log streams:

Open Telemetry Agent Config

config:
  exporters:
    otlphttp:
      endpoint: http://clickstack-otel-collector.clickhouse.svc.cluster.local:4318/
      headers:
        authorization: "my-ingestion-api-key"
    debug:
      verbosity: detailed
  processors:
      batch:
        timeout: 10s
      memory_limiter:
        check_interval: 5s
        limit_percentage: 80
        spike_limit_percentage: 25
receivers: 
    awscloudwatch:
      region: us-east-1
      logs:
        poll_interval: 1m
        max_events_per_request: 100
        groups:
          named:
            /aws/lambda/my-lambda:
            API-Gateway-Execution-Logs_xxx:
    jaeger: null
    otlp: null
    prometheus: null
    zipkin: null
  service:
    pipelines:
      logs:
        receivers: [awscloudwatch]
        processors: [batch]
        exporters: [otlphttp, debug]
      metrics: null
      traces: null

I don't see any errors saying there are IAM permission issues, no errors saying it can't connect to our open telemetry collecting in the clickstack, and no errors with the authorization in regards to the ingestion API Key.

However, no AWS cloudwatch logs show up in the HyperDX at all, only logs from HyperDX with the service name hdx-oss-api like HTTP GET /health or HTTP POST /clickhouse-proxy

I've turned on the OTEL_SUPERVISOR_LOGS to true in the ClickStack helm chart for the otel collector to see if there's anything going on but I only see this warning which doesn't make a lot of sense to me

{"level":"warn","ts":"2026-03-18T01:38:10.239Z","caller":"ottl@v0.145.0/parser.go:410","msg":"failed to execute statement","resource":{"host.arch":"amd64","host.name":"clickstack-otel-collector-54dc5f4f4b-mkp8g","os.description":"Alpine 3.23.3","os.type":"linux","service.instance.id":"019cfe95-8243-7e1d-b223-7df740cfda3c","service.name":"otelcol-contrib","service.version":"0.145.0"},"otelcol.component.id":"transform","otelcol.component.kind":"processor","otelcol.pipeline.id":"logs/out-default","otelcol.signal":"logs","error":"error getting value in ottl.StandardPMapGetter[*github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog.TransformContext]: expected string but got nil","statement":"merge_maps(log.attributes, ParseJSON(log.cache[\"0\"]), \"upsert\") where IsMap(log.cache)"}

and no other errors.

Am I missing some other extra step to get the AWS Cloudwatch logs to show up in the ClickStack's HyperDX? Are there any additional logs to turn on look at for more clues?
Helm Chart version is clickstack-1.1.2

[vinfang]

Found out the issue will and will explain my steps to help out others if they need troubleshooting.

Apparently the logs being sent to our clickstack from the AWS cloudwatch logs are dated in 2025 and the year is 2026 now. If you adjust the HyperDX time range to go back into time you'll see the logs show up in HyperDX.

Additionally, if you want to confirm the logs are in the clickhouse database in kubernetes, ssh into the clickhouse pod

kubectl exec --stdin --tty clickhouse_pod_name -n clickhouse_namespace -- /bin/bash

and then run the clickhouse-client command in the pod to access the database and look at the default.otel_logs table to see if your logs actually got inserted into or not

So there's nothing wrong with the helm chart or clickstack, just that our logs are dated from last year for some reason.

READING:
https://clickhouse.com/docs/interfaces/cli