Add bulk_message_request_threshold to ES output plugin#1946
Open
madmecodes wants to merge 2 commits intofluent:masterfrom
Open
Add bulk_message_request_threshold to ES output plugin#1946madmecodes wants to merge 2 commits intofluent:masterfrom
madmecodes wants to merge 2 commits intofluent:masterfrom
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR exposes Fluentd Elasticsearch output plugin’s bulk_message_request_threshold setting via the Fluent Operator CRDs (for both elasticsearch and elasticsearch_data_stream) and ensures it’s rendered into the generated Fluentd configuration, alongside regenerated CRDs and updated plugin docs.
Changes:
- Add
bulkMessageRequestThresholdtoElasticsearchCommonso it’s available to both ES output variants. - Render
bulk_message_request_thresholdinto Fluentd output config when the field is set. - Regenerate CRDs/manifests and update Elasticsearch output plugin documentation.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| manifests/setup/setup.yaml | Regenerated install manifest schema to include bulkMessageRequestThreshold for ES outputs. |
| manifests/setup/fluent-operator-crd.yaml | Regenerated CRD bundle with the new ES output field. |
| docs/plugins/fluentd/output/es.md | Documents the newly exposed ES output parameter. |
| config/crd/bases/fluentd.fluent.io_outputs.yaml | Regenerated Output CRD schema including the new field. |
| config/crd/bases/fluentd.fluent.io_clusteroutputs.yaml | Regenerated ClusterOutput CRD schema including the new field. |
| charts/fluent-operator/crds/fluentd.fluent.io_outputs.yaml | Helm CRDs updated to include the new field. |
| charts/fluent-operator/crds/fluentd.fluent.io_clusteroutputs.yaml | Helm CRDs updated to include the new field. |
| charts/fluent-operator-fluentd-crds/templates/fluentd.fluent.io_outputs.yaml | Helm templated CRDs updated to include the new field. |
| charts/fluent-operator-fluentd-crds/templates/fluentd.fluent.io_clusteroutputs.yaml | Helm templated CRDs updated to include the new field. |
| apis/fluentd/v1alpha1/plugins/output/types.go | Adds config rendering for bulk_message_request_threshold. |
| apis/fluentd/v1alpha1/plugins/output/es.go | Adds the API field bulkMessageRequestThreshold to ElasticsearchCommon. |
| IlmPolicyOverwrite *bool `json:"ilmPolicyOverride,omitempty"` | ||
| // Optional, Enable logging of 400 reason without enabling debug log level | ||
| LogEs400Reason *bool `json:"logEs400Reason,omitempty"` | ||
| // Optional, Configure bulk_message request splitting threshold size. |
Comment on lines
396
to
400
| params.InsertPairs(parent, "ilm_policy", cmn.IlmPolicy) | ||
| params.InsertPairs(parent, "ilm_policy_overwrite", cmn.IlmPolicyOverwrite) | ||
| params.InsertPairs(parent, "log_es_400_reason", cmn.LogEs400Reason) | ||
| params.InsertPairs(parent, "bulk_message_request_threshold", cmn.BulkMessageRequestThreshold) | ||
|
|
Collaborator
|
@madmecodes can you help fix the dco issue? |
Signed-off-by: madmecodes <ayushguptadev1@gmail.com>
Signed-off-by: madmecodes <ayushguptadev1@gmail.com>
madmecodes
force-pushed
the
feat/es-bulk-message-threshold
branch
from
e77c7ca to
b4a9b48
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bulk_message_request_thresholdfield to theElasticsearchCommonstruct, making it available for bothelasticsearchandelasticsearch_data_streamoutput pluginsFixes #1905
Test plan
bulkMessageRequestThresholdunder bothelasticsearchandelasticsearchDataStreamoutput specsbulkMessageRequestThresholdset and confirm the generated fluentd config containsbulk_message_request_threshold