Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,21 @@
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
</PropertyGroup>
<ItemGroup>
<PackageVersion Include="Itmo.Dev.Platform.Persistence.Abstractions" Version="1.2.364" />
<PackageVersion Include="Itmo.Dev.Platform.Persistence.Postgres" Version="1.2.364" />
<PackageVersion Include="Itmo.Dev.Platform.Persistence.Abstractions" Version="1.2.384" />
<PackageVersion Include="Itmo.Dev.Platform.Persistence.Postgres" Version="1.2.384" />
<PackageVersion Include="Itmo.Dev.Editorconfig" Version="1.0.14" />
<PackageVersion Include="Microsoft.Extensions.Configuration.UserSecrets" Version="10.0.3" />
<PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.3" />
<PackageVersion Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="10.0.3" />
<PackageVersion Include="Microsoft.Extensions.Options.DataAnnotations" Version="10.0.3" />
<PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.15.2" />
<PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.15.2" />
<PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.15.1" />
<PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.15.0" />
<PackageVersion Include="OpenTelemetry.Instrumentation.Runtime" Version="1.15.0" />
<PackageVersion Include="Refit.HttpClientFactory" Version="10.0.1" />
<PackageVersion Include="Scrutor" Version="7.0.0" />
<PackageVersion Include="SourceKit.Generators.Builder" Version="1.2.56" />
<PackageVersion Include="SourceKit.Generators.Builder" Version="1.2.58" />
<PackageVersion Include="Spectre.Console.Cli" Version="0.53.1" />
<PackageVersion Include="Spectre.Console.Registrars.Microsoft-Di" Version="0.6.0" />
<PackageVersion Include="prometheus-net" Version="8.2.1" />
Expand Down
60 changes: 58 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# Configuration Service (Labs 2-3-4)
# Configuration Service (Labs 2-3-4-5)

## Оглавление / Table of Contents
- [Lab 2: Локальный запуск API](#lab-2-локальный-запуск-api)
- [Lab 3: Метрики и Grafana](#lab-3-метрики-и-grafana)
- [Lab 4: Логирование и LogQL](#lab-4-логирование-и-logql)
- [Lab 5: Трейсы и TraceQL](#lab-5-трейсы-и-traceql)

Сервис конфигураций с двумя методами:
- `POST /api/configurations` - сохранить/обновить набор конфигураций
Expand Down Expand Up @@ -107,4 +108,59 @@ docker compose -f ./docker/docker-compose.yml up --build
![logs-explore-400](docs/images/lab4/logs-explore-400.png)

Среднее количество логов об неудачных запросах в час
![logs-average-400-per-hour](docs/images/lab4/logs-average-400-per-hour.png)
![logs-average-400-per-hour](docs/images/lab4/logs-average-400-per-hour.png)

## Lab 5: Трейсы и TraceQL

### Что добавлено
- Генерация трейсов в приложении:
- автотрейсинг ASP.NET Core запросов через OpenTelemetry
- ручные спаны в `ConfigurationController`:
- `configurations.set`
- `configurations.get`
- custom-теги для фильтрации (`configurations.entries_count`, `configurations.page_size`, `configurations.has_page_token`)
- Отправка трейсов:
- OTLP exporter из приложения в Tempo (`OTEL_EXPORTER_OTLP_ENDPOINT`)
- Хранение и просмотр:
- Tempo как backend для трейсов
- Grafana Explore как UI просмотра трейсов
- Язык запросов:
- TraceQL для поиска и фильтрации трейсов в Grafana

### Инфраструктура
В `docker/docker-compose.yml` добавлен сервис:
- `tempo` (`grafana/tempo`)

Также добавлены:
- `docker/tempo.yaml` - конфигурация Tempo
- `monitoring/grafana/provisioning/datasources/tempo.yml` - datasource Tempo в Grafana
- переменная окружения `OTEL_EXPORTER_OTLP_ENDPOINT` у `app`

Запуск:
```bash
docker compose -f ./docker/docker-compose.yml up --build
```

### Примеры TraceQL-запросов
- Все трейсы сервиса:
- `{ .service.name = "configurations-service" }`
- Только ручные спаны записи конфигураций:
- `{ name = "configurations.set" }`
- Только ручные спаны чтения конфигураций:
- `{ name = "configurations.get" }`
- Запросы, где размер page > 10:
- `{ span.configurations.page_size > 10 }`
- Запросы, где был передан page token:
- `{ span.configurations.has_page_token = true }`
- Ошибочные спаны:
- `{ status = error }`

### Скриншоты ЛР5
- Explore с трейсами сервиса
![traces-explore-all](docs/images/lab5/traces-explore-all.png)
- Фильтрация через TraceQL по ручным спанам
![traces-traceql-set-get](docs/images/lab5/traces-traceql-set-get.png)
- Поиск ошибок/фильтрация по статусу
![traces-traceql-errors](docs/images/lab5/traces-traceql-errors.png)
- Пример трейса с >1 span
![trace-with-spans](docs/images/lab5/trace-with-spans.png)
18 changes: 18 additions & 0 deletions docker/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,14 @@ services:
Infrastructure__Persistence__Postgres__SslMode: Prefer
Infrastructure__Persistence__Postgres__Pooling: "true"
LOKI_URL: http://loki:3100
OTEL_EXPORTER_OTLP_ENDPOINT: http://tempo:4317
ports:
- "5215:8080"
depends_on:
postgres:
condition: service_healthy
tempo:
condition: service_started

victoriametrics:
image: victoriametrics/victoria-metrics:latest
Expand Down Expand Up @@ -70,6 +73,19 @@ services:
volumes:
- loki_data:/loki

tempo:
image: grafana/tempo:2.7.2
container_name: lab5-tempo
command:
- "-config.file=/etc/tempo.yaml"
ports:
- "3200:3200"
- "4317:4317"
- "4318:4318"
volumes:
- ../docker/tempo.yaml:/etc/tempo.yaml:ro
- tempo_data:/var/tempo

grafana:
image: grafana/grafana:11.6.1
container_name: lab3-grafana
Expand All @@ -85,9 +101,11 @@ services:
depends_on:
- victoriametrics
- loki
- tempo

volumes:
postgres_data:
vm_data:
loki_data:
tempo_data:
grafana_data:
29 changes: 29 additions & 0 deletions docker/tempo.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
server:
http_listen_port: 3200

distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318

ingester:
max_block_duration: 5m

compactor:
compaction:
block_retention: 24h

storage:
trace:
backend: local
local:
path: /var/tempo/traces

overrides:
defaults:
metrics_generator:
processors: [service-graphs, span-metrics]
Binary file added docs/images/lab5/trace-with-spans.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/lab5/traces-explore-all.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/lab5/traces-traceql-errors.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/lab5/traces-traceql-set-get.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 10 additions & 0 deletions monitoring/grafana/provisioning/datasources/tempo.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
apiVersion: 1

datasources:
- name: Tempo
uid: tempo
type: tempo
access: proxy
url: http://tempo:3200
isDefault: false
editable: false
5 changes: 5 additions & 0 deletions src/Configurations/Configurations.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@
<ProjectReference Include="..\Presentation\Configurations.Presentation.Http\Configurations.Presentation.Http.csproj" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="OpenTelemetry.Extensions.Hosting" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" />
<PackageReference Include="OpenTelemetry.Instrumentation.Runtime" />
<PackageReference Include="prometheus-net.AspNetCore" />
<PackageReference Include="Serilog.AspNetCore" />
<PackageReference Include="Serilog.Sinks.Grafana.Loki" />
Expand Down
20 changes: 19 additions & 1 deletion src/Configurations/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,15 @@
using Configurations.Infrastructure.Persistence;
using Configurations.Presentation.Http;
using Itmo.Dev.Platform.Common.Extensions;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using Prometheus;
using Serilog;
using Serilog.Sinks.Grafana.Loki;

string lokiUrl = Environment.GetEnvironmentVariable("LOKI_URL") ?? "http://loki:3100";
string otlpEndpoint = Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT") ?? "http://tempo:4317";
const string serviceName = "configurations-service";

Log.Logger = new LoggerConfiguration()
.MinimumLevel.Information()
Expand All @@ -27,6 +31,17 @@

builder.Host.UseSerilog();
builder.Services.AddPlatform(platform => platform.WithNewtonsoftSerialization());
builder.Services.AddOpenTelemetry()
.ConfigureResource(resource => resource
.AddService(serviceName: serviceName))
.WithTracing(tracing => tracing
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddSource("Configurations.Presentation.Http")
.AddOtlpExporter(options =>
{
options.Endpoint = new Uri(otlpEndpoint);
}));

builder.Services.AddSwaggerGen(c =>
{
Expand All @@ -47,7 +62,10 @@
app.UsePresentationHttp();
app.MapMetrics();

Log.Information("Configuration service started with Loki URL {LokiUrl}", lokiUrl);
Log.Information(
"Configuration service started with Loki URL {LokiUrl} and OTLP endpoint {OtlpEndpoint}",
lokiUrl,
otlpEndpoint);
await app.RunAsync();
}
catch (Exception ex)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
using System.Text.Json;
using System.Diagnostics;
using Configurations.Application.Contracts.Configurations;
using Configurations.Application.Contracts.Configurations.Operations;
using Configurations.Application.Model;
Expand All @@ -14,6 +15,8 @@ namespace Configurations.Presentation.Http.Controllers;
[Route("api/configurations")]
public sealed class ConfigurationController : ControllerBase
{
private static readonly ActivitySource ActivitySource = new("Configurations.Presentation.Http");

private readonly static Counter SetRequestsTotal = Metrics.CreateCounter(
"configurations_set_requests_total",
"Total number of set configurations requests.");
Expand Down Expand Up @@ -62,6 +65,8 @@ public async Task<IActionResult> SetConfigurationsAsync(
[FromBody] SetConfigurationsRequest request,
CancellationToken cancellationToken)
{
using Activity? activity = ActivitySource.StartActivity("configurations.set");

try
{
SetRequestsTotal.Inc();
Expand All @@ -75,20 +80,36 @@ public async Task<IActionResult> SetConfigurationsAsync(
_logger.LogWarning("Set configurations request received with empty entries collection");
}

activity?.SetTag("configurations.entries_count", entries.Length);
_logger.LogInformation("Set configurations request accepted with {EntriesCount} entries", entries.Length);

using (Activity? validateActivity = ActivitySource.StartActivity("configurations.validate"))
{
validateActivity?.SetTag("configurations.entries_count", entries.Length);
validateActivity?.SetStatus(entries.Length == 0 ? ActivityStatusCode.Error : ActivityStatusCode.Ok);
if (entries.Length == 0)
validateActivity?.SetStatus(ActivityStatusCode.Error, "Empty entries collection");
}

var applicationRequest = new SetConfigurations.Request(entries);

await _configurationService.SetConfigurationsAsync(applicationRequest, cancellationToken);
using (Activity? dbActivity = ActivitySource.StartActivity("configurations.db.set"))
{
dbActivity?.SetTag("configurations.entries_count", entries.Length);
await _configurationService.SetConfigurationsAsync(applicationRequest, cancellationToken);
dbActivity?.SetStatus(ActivityStatusCode.Ok);
}

EntriesWrittenTotal.Inc(entries.Length);
SetBatchSize.Observe(entries.Length);
activity?.SetStatus(ActivityStatusCode.Ok);

_logger.LogInformation("Set configurations request completed successfully, written {EntriesCount} entries", entries.Length);
return Ok();
}
catch (Exception ex)
{
activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
_logger.LogError(ex, "Set configurations request failed");
throw;
}
Expand All @@ -99,9 +120,13 @@ public async Task<ActionResult<GetConfigurationsResponse>> GetConfigurationsAsyn
[FromQuery] GetConfigurationsRequest request,
CancellationToken cancellationToken)
{
using Activity? activity = ActivitySource.StartActivity("configurations.get");

try
{
GetRequestsTotal.Inc();
activity?.SetTag("configurations.page_size", request.PageSize);
activity?.SetTag("configurations.has_page_token", request.PageToken is not null);
_logger.LogInformation(
"Get configurations request accepted with page size {PageSize}, page token provided: {HasPageToken}",
request.PageSize,
Expand All @@ -113,9 +138,21 @@ public async Task<ActionResult<GetConfigurationsResponse>> GetConfigurationsAsyn

var applicationRequest = new GetConfigurations.Request(request.PageSize, pageToken);

GetConfigurations.Response applicationResponse = await _configurationService.GetConfigurationAsync(
applicationRequest,
cancellationToken);
using (Activity? validateActivity = ActivitySource.StartActivity("configurations.validate"))
{
validateActivity?.SetTag("configurations.page_size", request.PageSize);
validateActivity?.SetTag("configurations.has_page_token", request.PageToken is not null);
validateActivity?.SetStatus(ActivityStatusCode.Ok);
}

GetConfigurations.Response applicationResponse;
using (Activity? dbActivity = ActivitySource.StartActivity("configurations.db.get"))
{
applicationResponse = await _configurationService.GetConfigurationAsync(
applicationRequest,
cancellationToken);
dbActivity?.SetStatus(ActivityStatusCode.Ok);
}

IEnumerable<GetConfigurationsResponse.ConfigurationEntry> entries = applicationResponse.Entries
.Select(entry => new GetConfigurationsResponse.ConfigurationEntry
Expand All @@ -130,6 +167,8 @@ public async Task<ActionResult<GetConfigurationsResponse>> GetConfigurationsAsyn

EntriesReadTotal.Inc(applicationResponse.Entries.Count);
GetResultSize.Observe(applicationResponse.Entries.Count);
activity?.SetTag("configurations.entries_count", applicationResponse.Entries.Count);
activity?.SetStatus(ActivityStatusCode.Ok);

_logger.LogInformation(
"Get configurations request completed, returned {EntriesCount} entries, next page token provided: {HasNextPageToken}",
Expand All @@ -144,6 +183,7 @@ public async Task<ActionResult<GetConfigurationsResponse>> GetConfigurationsAsyn
}
catch (JsonException ex)
{
activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
_logger.LogWarning(ex, "Invalid page token format in get configurations request");
return BadRequest(new
{
Expand All @@ -153,6 +193,7 @@ public async Task<ActionResult<GetConfigurationsResponse>> GetConfigurationsAsyn
}
catch (Exception ex)
{
activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
_logger.LogError(ex, "Get configurations request failed");
throw;
}
Expand Down