Skip to content

feat(sdk-logs): add maxConcurrentExports to enable concurrent batch e…#8250

Closed
TruongNguyen1305 wants to merge 1 commit intoopen-telemetry:mainfrom
TruongNguyen1305:improvement/BatchLogRecordProcessor/concurrent-export
Closed

feat(sdk-logs): add maxConcurrentExports to enable concurrent batch e…#8250
TruongNguyen1305 wants to merge 1 commit intoopen-telemetry:mainfrom
TruongNguyen1305:improvement/BatchLogRecordProcessor/concurrent-export

Conversation

@TruongNguyen1305
Copy link
Copy Markdown
Contributor

Summary

This PR introduces a new configuration option maxConcurrentExports for BatchLogRecordProcessor to allow multiple export operations to run concurrently.

By enabling controlled parallelism, this change improves log export throughput under high load while maintaining backpressure.


Motivation

Currently, BatchLogRecordProcessor performs exports sequentially, allowing only a single in-flight export at any given time. While this simplifies flow control, it can become a bottleneck in high-throughput scenarios, especially when:

  • Export latency is non-trivial (e.g., network-bound OTLP exporters)
  • Log production rate exceeds single-threaded export capacity
  • The underlying exporter already supports asynchronous or multi-threaded execution

At the same time, switching to fully asynchronous export without limits can overwhelm downstream systems (e.g., OTLP receivers), leading to errors such as HTTP/2 REFUSED_STREAM.


Changes

  • Add a new configuration parameter:

    int maxConcurrentExports

  • Allow multiple export batches to be processed in parallel, bounded by maxConcurrentExports

  • Preserve existing behavior by default:

    • maxConcurrentExports = 1 (sequential export)

Design Considerations

Controlled Concurrency

Instead of unbounded async exports, this approach ensures:

  • Bounded number of in-flight export requests
  • Protection against overwhelming downstream systems
  • Stable behavior under burst traffic

Backpressure

When the concurrency limit is reached:

  • Additional batches will be processed synchronously
  • This maintains backpressure semantics similar to the current implementation

Executor Utilization

This change allows better utilization of the underlying LogRecordExporter, particularly when:

  • The exporter uses an internal ExecutorService
  • The exporter performs non-blocking or async I/O (e.g., OTLP over gRPC)

By enabling concurrent exports at the processor level, we avoid underutilizing available execution resources.


Compatibility

  • Default behavior remains unchanged (maxConcurrentExports = 1)
  • No breaking API changes
  • Fully backward compatible

@TruongNguyen1305 TruongNguyen1305 requested a review from a team as a code owner April 5, 2026 04:39
@TruongNguyen1305 TruongNguyen1305 deleted the improvement/BatchLogRecordProcessor/concurrent-export branch April 5, 2026 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant