[New Integration] Kafka OpenTelemetry Input Package

[zmoog]

Summary

• New OTel input package (kafka_input_otel) that uses the kafkareceiver to consume OTLP-encoded logs, metrics, and traces from Kafka-compatible brokers, including Azure Event Hub
• Supports optional per-signal topics (logs, metrics, traces) with dynamic_signal_types: true
• Authentication via SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512) — covers Event Hub connection string and plain Kafka clusters
• Encoding limited to otlp_json / otlp_proto (EDOT kafkareceiver constraint)

This is a POC to explore using the OTel Kafka receiver as a transport layer for ingesting telemetry from Azure Event Hub.

Key design decisions

• Single package, multiple signals: Each signal type is optional — enabled by providing a topic name. One Event Hub per signal type.
• Consumer groups: Shared default ($Default) with optional per-signal overrides.
• Not Azure-specific: Works with any Kafka-compatible broker, though Event Hub is the primary use case.

Known limitations

• Only otlp_json and otlp_proto encodings (EDOT restriction)
• No OAUTHBEARER auth (Azure encoding extension not in EDOT yet)
• Policy tests skipped pending elastic-package multi-signal support (elastic/elastic-package#3298)

Test plan

• elastic-package lint passes
• elastic-package build succeeds
• Deploy with Elastic Agent + EDOT against an Azure Event Hub namespace
• Verify OTLP JSON logs flow through kafkareceiver → Elasticsearch
• Verify OTLP JSON metrics flow through kafkareceiver → Elasticsearch
• Verify OTLP JSON traces flow through kafkareceiver → Elasticsearch
• Test with only one signal type enabled (e.g., logs only)

🤖 Generated with Claude Code

[github-actions]

Vale Linting Results

Summary: 1 warning, 1 suggestion found

⚠️ Warnings (1)

File	Line	Rule	Message
packages/kafka_input_otel/docs/README.md	36	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.

💡 Suggestions (1)

File	Line	Rule	Message
packages/kafka_input_otel/docs/README.md	21	Elastic.Wordiness	Consider using 'because' instead of 'since'.

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

[macroscopeapp]

🟡 Medium input/input.yml.hbs:1

When none of logs_topic, metrics_topic, or traces_topic are configured, the template produces an invalid OpenTelemetry Collector configuration with empty receivers: and pipelines: sections. The Collector fails at startup because both sections require at least one entry. Consider adding a top-level guard (e.g., wrap everything in {{#if}} checking at least one topic) or making one topic required in the manifest.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file packages/kafka_input_otel/agent/input/input.yml.hbs around line 1:

When none of `logs_topic`, `metrics_topic`, or `traces_topic` are configured, the template produces an invalid OpenTelemetry Collector configuration with empty `receivers:` and `pipelines:` sections. The Collector fails at startup because both sections require at least one entry. Consider adding a top-level guard (e.g., wrap everything in `{{#if}}` checking at least one topic) or making one topic required in the manifest.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 73491dc

Failed CI Steps

• ✅ Check go sources

cc @zmoog