feat(otel): export OTLP traces

docs/diagnostics/integrations.md

@@ -6,7 +6,7 @@ title: "Integrations"
 
 EventStoreDB supports several methods to integrate with external monitoring and observability tools. Those include:
 
-- [OpenTelemetry](#opentelemetry-exporter): export metrics to an OpenTelemetry-compatible endpoint
+- [OpenTelemetry](#opentelemetry-exporter): export telemetry to an OpenTelemetry-compatible endpoint
 - [Prometheus](#prometheus): collect metrics in Prometheus
 - [Datadog](#datadog): monitor and measure the cluster with Datadog
 - [ElasticSearch](#elasticsearch): this section describes how to collect EventStoreDB logs in ElasticSearch
@@ -21,7 +21,7 @@ Older versions can be monitored by Prometheus using the community-supported expo
 
 ## OpenTelemetry Exporter
 
-EventStoreDB passively exposes metrics for scraping on the `/metrics` endpoint. It can also actively export metrics and logs using the [OpenTelemetry Protocol](https://opentelemetry.io/docs/specs/otel/protocol/) (OTLP).
+EventStoreDB passively exposes metrics for scraping on the `/metrics` endpoint. It can also actively export logs, metrics, and traces using the [OpenTelemetry Protocol](https://opentelemetry.io/docs/specs/otel/protocol/) (OTLP).
 
 A number of APM providers natively support OTLP, so you might be able to send EventStoreDB telemetry directly to your APM provider. Alternatively, you can export to the OpenTelemetry Collector, which can then fan out to a variety of backends. You can find out more about the [OpenTelemetry collector](https://opentelemetry.io/docs/collector/).
 
@@ -44,13 +44,19 @@ Sample JSON configuration:
         "Otlp": {
           "Endpoint": "http://metrics-collector:4317"
         }
+      },
+      "Traces": {
+        "Enabled": true,
+        "Otlp": {
+          "Endpoint": "http://traces-collector:4317"
+        }
       }
     }
   }
 }
 ```
 
-The shared `EventStore:OpenTelemetry:Otlp` section provides defaults for every enabled OTLP signal. The `EventStore:OpenTelemetry:Logs:Otlp` and `EventStore:OpenTelemetry:Metrics:Otlp` sections can override only the settings that need to differ for each signal.
+The shared `EventStore:OpenTelemetry:Otlp` section provides defaults for every enabled OTLP signal. The `EventStore:OpenTelemetry:Logs:Otlp`, `EventStore:OpenTelemetry:Metrics:Otlp`, and `EventStore:OpenTelemetry:Traces:Otlp` sections can override only the settings that need to differ for each signal.
 
 All OpenTelemetry environment variables are optional. Configure them only in deployment environments that should export telemetry to an OTLP collector.
 
@@ -64,6 +70,8 @@ The configuration can specify:
 | EventStore__OpenTelemetry__Logs__Otlp__Endpoint    | Optional log-specific OTLP destination                         |
 | EventStore__OpenTelemetry__Metrics__Enabled        | Enables OTLP metric export from runtime configuration          |
 | EventStore__OpenTelemetry__Metrics__Otlp__Endpoint | Optional metric-specific OTLP destination                      |
+| EventStore__OpenTelemetry__Traces__Enabled         | Enables OTLP trace export from runtime configuration           |
+| EventStore__OpenTelemetry__Traces__Otlp__Endpoint  | Optional trace-specific OTLP destination                       |
 
 Headers are key-value pairs separated by commas. For example:
 ```:no-line-numbers
@@ -82,7 +90,8 @@ The interval is taken from the `ExpectedScrapeIntervalSeconds` value in `metrics
 |------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|
 | Logs are not exported                    | Check that `EventStore__OpenTelemetry__Logs__Enabled` is set to `true`.                                                                        |
 | Metrics are not exported                 | Check that `EventStore__OpenTelemetry__Metrics__Enabled` is `true`, any `EventStore__OpenTelemetry__Metrics__Otlp__*` key is set, or `Otlp.Enabled` is `true` in `metricsconfig.json`. |
-| Telemetry arrives at the wrong collector | Check whether a per-signal `Logs:Otlp` or `Metrics:Otlp` section is overriding the shared `EventStore:OpenTelemetry:Otlp` destination.         |
+| Traces are not exported                  | Check that `EventStore__OpenTelemetry__Traces__Enabled` is `true` or any `EventStore__OpenTelemetry__Traces__Otlp__*` key is set.             |
+| Telemetry arrives at the wrong collector | Check whether a per-signal `Logs:Otlp`, `Metrics:Otlp`, or `Traces:Otlp` section is overriding the shared `EventStore:OpenTelemetry:Otlp` destination. |
 
 ## Datadog
 

qodana.yaml

@@ -77,3 +77,9 @@ dependencyOverrides:
     licenses:
       - key: "Apache-2.0"
         url: "https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/LICENSE.TXT"
+
+  - name: "OpenTelemetry.Instrumentation.AspNetCore"
+    version: "1.15.2"
+    licenses:
+      - key: "Apache-2.0"
+        url: "https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/main/LICENSE"

src/Directory.Packages.props

@@ -62,6 +62,7 @@
     <PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.15.3" />
     <PackageVersion Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.15.3-beta.1" />
     <PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.15.3" />
+    <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.15.2" />
     <PackageVersion Include="Quickenshtein" Version="1.5.1" />
     <PackageVersion Include="Scrutor" Version="5.1.2" />
     <PackageVersion Include="Serilog" Version="4.3.0" />

src/EventStore.Core.XUnit.Tests/OpenTelemetry/OtlpTracingConfigurationTests.cs

@@ -0,0 +1,84 @@
+using System;
+using System.Collections.Generic;
+using EventStore.Core.Configuration;
+using FluentAssertions;
+using Microsoft.Extensions.Configuration;
+using OpenTelemetry.Exporter;
+using Xunit;
+
+namespace EventStore.Core.XUnit.Tests.OpenTelemetry;
+
+public class OtlpTracingConfigurationTests
+{
+	[Fact]
+	public void IsDisabledByDefault()
+	{
+		var configuration = new ConfigurationBuilder().Build();
+
+		configuration.OtlpTracesEnabled().Should().BeFalse();
+	}
+
+	[Fact]
+	public void DoesNotEnableTracesWhenOnlySharedOpenTelemetryOtlpSectionExists()
+	{
+		var configuration = new ConfigurationBuilder()
+			.AddInMemoryCollection(new Dictionary<string, string>
+			{
+				["EventStore:OpenTelemetry:Otlp:Endpoint"] = "http://shared:4317",
+			})
+			.Build();
+
+		configuration.OtlpTracesEnabled().Should().BeFalse();
+	}
+
+	[Fact]
+	public void EnablesTracesWhenRuntimeSwitchUsesSharedOpenTelemetryOtlpSettings()
+	{
+		var configuration = new ConfigurationBuilder()
+			.AddInMemoryCollection(new Dictionary<string, string>
+			{
+				["EventStore:OpenTelemetry:Traces:Enabled"] = "true",
+				["EventStore:OpenTelemetry:Otlp:Endpoint"] = "http://shared:4317",
+			})
+			.Build();
+
+		var options = configuration.GetOtlpExporterOptions(OpenTelemetryConfiguration.OtlpTracesOtlpPrefix);
+
+		configuration.OtlpTracesEnabled().Should().BeTrue();
+		options.Endpoint.Should().Be(new Uri("http://shared:4317"));
+	}
+
+	[Fact]
+	public void EnablesTracesWhenPerSignalOpenTelemetryOtlpSectionExists()
+	{
+		var configuration = new ConfigurationBuilder()
+			.AddInMemoryCollection(new Dictionary<string, string>
+			{
+				["EventStore:OpenTelemetry:Traces:Otlp:Endpoint"] = "http://traces:4317",
+			})
+			.Build();
+
+		configuration.OtlpTracesEnabled().Should().BeTrue();
+	}
+
+	[Fact]
+	public void PerSignalTracesOtlpOverridesSharedSettings()
+	{
+		var configuration = new ConfigurationBuilder()
+			.AddInMemoryCollection(new Dictionary<string, string>
+			{
+				["EventStore:OpenTelemetry:Otlp:Endpoint"] = "http://shared:4317",
+				["EventStore:OpenTelemetry:Otlp:Headers"] = "key=shared",
+				["EventStore:OpenTelemetry:Traces:Otlp:Endpoint"] = "http://traces:4317",
+				["EventStore:OpenTelemetry:Traces:Otlp:Headers"] = "key=traces",
+				["EventStore:OpenTelemetry:Traces:Otlp:Protocol"] = "HttpProtobuf",
+			})
+			.Build();
+
+		var options = configuration.GetOtlpExporterOptions(OpenTelemetryConfiguration.OtlpTracesOtlpPrefix);
+
+		options.Endpoint.Should().Be(new Uri("http://traces:4317"));
+		options.Headers.Should().Be("key=traces");
+		options.Protocol.Should().Be(OtlpExportProtocol.HttpProtobuf);
+	}
+}

src/EventStore.Core/ClusterVNodeStartup.cs

@@ -25,6 +25,7 @@
 using OpenTelemetry;
 using OpenTelemetry.Metrics;
 using OpenTelemetry.Resources;
+using OpenTelemetry.Trace;
 using Operations = EventStore.Core.Services.Transport.Grpc.Operations;
 using ClusterGossip = EventStore.Core.Services.Transport.Grpc.Cluster.Gossip;
 using ClientGossip = EventStore.Core.Services.Transport.Grpc.Gossip;
@@ -231,6 +232,7 @@ public void ConfigureServicesOnly(IServiceCollection services)
 
 			.AddOpenTelemetry()
 			.WithMetrics(meterOptions => ConfigureMetrics(meterOptions, metricsConfiguration, _configuration))
+			.WithTracing(tracerOptions => ConfigureTracing(tracerOptions, _configuration))
 			.Services
 
 			// gRPC
@@ -338,6 +340,22 @@ private static void ConfigureOtlpMetrics(
 		});
 	}
 
+	private static void ConfigureTracing(
+		TracerProviderBuilder tracerOptions,
+		IConfiguration configuration)
+	{
+		tracerOptions.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("eventstore"));
+
+		if (!configuration.OtlpTracesEnabled())
+			return;
+
+		tracerOptions
+			.AddAspNetCoreInstrumentation()
+			.AddOtlpExporter(exporterOptions => configuration.BindOtlpExporterOptions(
+				OpenTelemetryConfiguration.OtlpTracesOtlpPrefix,
+				exporterOptions));
+	}
+
 	public void Handle(SystemMessage.SystemReady _) => _ready = true;
 
 	public void Handle(SystemMessage.BecomeShuttingDown _) => _ready = false;

src/EventStore.Core/Configuration/OpenTelemetryConfiguration.cs

@@ -14,6 +14,8 @@ public static class OpenTelemetryConfiguration
 	public const string OtlpLogsOtlpPrefix = $"{OpenTelemetryPrefix}:Logs:Otlp";
 	public const string OtlpMetricsPrefix = $"{OpenTelemetryPrefix}:Metrics";
 	public const string OtlpMetricsOtlpPrefix = $"{OpenTelemetryPrefix}:Metrics:Otlp";
+	public const string OtlpTracesPrefix = $"{OpenTelemetryPrefix}:Traces";
+	public const string OtlpTracesOtlpPrefix = $"{OpenTelemetryPrefix}:Traces:Otlp";
 
 	public static bool OtlpLogsEnabled(this IConfiguration configuration) =>
 		configuration.GetValue<bool>($"{OtlpLogsPrefix}:Enabled");
@@ -23,6 +25,10 @@ public static bool OtlpMetricsEnabled(this IConfiguration configuration, Metrics
 		configuration.GetSection(OtlpMetricsOtlpPrefix).Exists() ||
 		configuration.GetValue<bool>($"{OtlpMetricsPrefix}:Enabled");
 
+	public static bool OtlpTracesEnabled(this IConfiguration configuration) =>
+		configuration.GetSection(OtlpTracesOtlpPrefix).Exists() ||
+		configuration.GetValue<bool>($"{OtlpTracesPrefix}:Enabled");
+
 	public static OtlpExporterOptions GetOtlpExporterOptions(
 		this IConfiguration configuration,
 		string perSignalOtlpPrefix,

src/EventStore.Core/EventStore.Core.csproj

@@ -20,6 +20,7 @@
 		<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" />
 		<PackageReference Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" />
 		<PackageReference Include="OpenTelemetry.Extensions.Hosting" />
+		<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" />
 		<PackageReference Include="Quickenshtein" />
 		<PackageReference Include="Scrutor" />
 		<PackageReference Include="Serilog.Sinks.OpenTelemetry" />

src/qodana.yaml

@@ -34,3 +34,8 @@ dependencyOverrides:
     licenses:
       - key: "Apache-2.0"
         url: "https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/LICENSE.TXT"
+  - name: "OpenTelemetry.Instrumentation.AspNetCore"
+    version: "1.15.2"
+    licenses:
+      - key: "Apache-2.0"
+        url: "https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/main/LICENSE"