input: add thread.ring_buffer.retry_limit option

[jinyongchoi]

When a threaded input plugin's ring buffer is full, the input thread retries writing to the buffer before dropping data. Previously, this retry limit was hardcoded to 10 (1 second total with 100ms sleep between retries).

This patch makes the retry limit configurable via the new 'thread.ring_buffer.retry_limit' option. The default value remains 10 for backward compatibility. Increasing this value allows the system to handle temporary backpressure situations without dropping data.

Fixes #11393

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

[SERVICE]
    flush 1
    grace 600
    log_level info

    storage.path /tmp/storage
    storage.metrics on
    storage.max_chunks_up 512
    storage.sync full
    storage.checksum off
    storage.backlog.mem_limit 100M

[INPUT]
    Name tail
    Path /var/log/app/*.log
    Tag app_logs
    Key message

    Read_from_Head true
    Refresh_Interval 3

    Buffer_Chunk_Size 1MB
    Buffer_Max_Size 16MB

    Mem_Buf_Limit 256MB
    storage.type memory
    storage.pause_on_chunks_overlimit true

    DB /tmp/storage/tail.db
    DB.sync normal

    Threaded true
    thread.ring_buffer.capacity  128
    thread.ring_buffer.retry_limit  1000

[OUTPUT]
    Name kafka
    Match app_logs
    Brokers kafka:9092
    Topics logs
    timestamp_format iso8601

    rdkafka.compression.codec gzip
    rdkafka.compression.level 7
    rdkafka.queue.buffering.max.messages 200000
    rdkafka.batch.size 1000000
    rdkafka.linger.ms 100

    workers 4

• [N/A] Debug log output from testing the change
• [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• [N/A] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features

  • Configurable retry limit for threaded input ring-buffer writes via setting thread.ring_buffer.retry_limit (default: 10).

• Bug Fixes / Observability

  • Enqueue failures now increment a failure metric and the configured retry value is validated (>0).

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a per-input configurable retry limit for threaded input ring buffer writes: new ring_buffer_retry_limit field on flb_input_instance, default macro and config property, parsing/validation during input setup, and use of the instance field in enqueue retry logic.

Changes

Cohort / File(s)	Summary
Input struct definition include/fluent-bit/flb_input.h	Added int ring_buffer_retry_limit field to struct flb_input_instance.
Configuration & initialization src/flb_input.c	Added FLB_INPUT_RING_BUFFER_RETRY_LIMIT (default 10); registered thread.ring_buffer.retry_limit in input properties; initialize ins->ring_buffer_retry_limit to default; parse and validate property (>0) in flb_input_set_property.
Ring buffer retry logic src/flb_input_chunk.c	Replaced local hardcoded retry limit with ins->ring_buffer_retry_limit in enqueue retry loop; incremented ring buffer retry failure counter on enqueue failure.

Sequence Diagram(s)

(Skipped — changes are focused and do not introduce a multi-component sequential flow requiring visualization.)

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Suggested reviewers

• edsiper
• koleini

Poem

🐇 I nudged a limit, soft and slight,
Threads may try a little more each night.
Rings wait patient, retries hold true,
Fewer drops fall, carrots for you. 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'input: add thread.ring_buffer.retry_limit option' clearly and concisely describes the main change: introducing a new configuration option for controlling ring buffer retry limits in threaded input plugins.
Linked Issues check	✅ Passed	The pull request fully implements all coding requirements from issue #11393: adds the thread.ring_buffer.retry_limit configuration option with default value 10, implements per-instance retry limit logic, and applies it across all threaded input plugins.
Out of Scope Changes check	✅ Passed	All changes are directly within scope: adding the configuration option to the header, implementing the macro and config map entry, and updating the retry logic to use the per-instance limit instead of hardcoded value.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

This PR adds a configurable retry limit for threaded input plugins' ring buffer writes. Previously, the retry limit was hardcoded to 10 attempts (1 second total with 100ms sleep between retries). This caused data loss during temporary backpressure situations that lasted longer than 1 second.

Changes:

• Added thread.ring_buffer.retry_limit configuration option with default value of 10 to maintain backward compatibility
• Modified ring buffer write logic to use the configurable retry limit instead of hardcoded value
• Updated documentation to explain the new configuration option

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
include/fluent-bit/flb_input.h	Added ring_buffer_retry_limit field to flb_input_instance struct
src/flb_input.c	Added configuration map entry, initialization, and property parsing for the new retry limit option
src/flb_input_chunk.c	Removed hardcoded retry limit and updated to use instance-level configurable value

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jinyongchoi]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 👍

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[jinyongchoi]

@cosmo0920 @edsiper
Hello,
This PR is ready for review. It adds a new thread.ring_buffer.retry_limit configuration option for threaded input plugins, replacing the hardcoded retry limit (10) with a user-configurable value. This allows users to tune retry attempts for handling temporary backpressure situations without dropping data.
Thanks!