Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions pipeline/outputs/s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@
| `s3_key_format` | Format string for keys in S3. This option supports a UUID, strftime time formatters, a syntax for selecting parts of the Fluent log tag using a syntax inspired by the `rewrite_tag` filter. Add `$UUID` in the format string to insert a random string. Add `$INDEX` in the format string to insert an integer that increments each upload. The `$INDEX` value saves in the `store_dir`. Add `$TAG` in the format string to insert the full log tag. Add `$TAG[0]` to insert the first part of the tag in the S3 key. The tag is split into parts using the characters specified with the `s3_key_format_tag_delimiters` option. Add the extension directly after the last piece of the format string to insert a key suffix. To specify a key suffix in `use_put_object` mode, you must specify `$UUID`. See [S3 Key Format](#s3-key-format-and-tag-delimiters). Time in `s3_key` is the timestamp of the first record in the S3 file. | `/fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S` |
| `s3_key_format_tag_delimiters` | A series of characters which will be used to split the tag into `parts` for use with the s3_key_format option. | `.` |
| `send_content_md5` | Send the Content-MD5 header with `PutObject` and UploadPart requests, as is required when Object Lock is enabled. | `false` |
| `sse` | Server-side encryption for S3 objects. Set to `AES256` for S3-managed keys (SSE-S3), `aws:kms` for AWS KMS-managed keys (SSE-KMS), or `aws:kms:dsse` for dual-layer server-side encryption with KMS (DSSE-KMS). | _none_ |

Check warning on line 83 in pipeline/outputs/s3.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Acronyms] Spell out 'KMS', if it's unfamiliar to the audience. Raw Output: {"message": "[FluentBit.Acronyms] Spell out 'KMS', if it's unfamiliar to the audience.", "location": {"path": "pipeline/outputs/s3.md", "range": {"start": {"line": 83, "column": 245}}}, "severity": "INFO"}

Check warning on line 83 in pipeline/outputs/s3.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Acronyms] Spell out 'DSSE', if it's unfamiliar to the audience. Raw Output: {"message": "[FluentBit.Acronyms] Spell out 'DSSE', if it's unfamiliar to the audience.", "location": {"path": "pipeline/outputs/s3.md", "range": {"start": {"line": 83, "column": 240}}}, "severity": "INFO"}

Check warning on line 83 in pipeline/outputs/s3.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Acronyms] Spell out 'KMS', if it's unfamiliar to the audience. Raw Output: {"message": "[FluentBit.Acronyms] Spell out 'KMS', if it's unfamiliar to the audience.", "location": {"path": "pipeline/outputs/s3.md", "range": {"start": {"line": 83, "column": 235}}}, "severity": "INFO"}

Check warning on line 83 in pipeline/outputs/s3.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Acronyms] Spell out 'SSE', if it's unfamiliar to the audience. Raw Output: {"message": "[FluentBit.Acronyms] Spell out 'SSE', if it's unfamiliar to the audience.", "location": {"path": "pipeline/outputs/s3.md", "range": {"start": {"line": 83, "column": 164}}}, "severity": "INFO"}

Check warning on line 83 in pipeline/outputs/s3.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Acronyms] Spell out 'KMS', if it's unfamiliar to the audience. Raw Output: {"message": "[FluentBit.Acronyms] Spell out 'KMS', if it's unfamiliar to the audience.", "location": {"path": "pipeline/outputs/s3.md", "range": {"start": {"line": 83, "column": 146}}}, "severity": "INFO"}

Check warning on line 83 in pipeline/outputs/s3.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Acronyms] Spell out 'SSE', if it's unfamiliar to the audience. Raw Output: {"message": "[FluentBit.Acronyms] Spell out 'SSE', if it's unfamiliar to the audience.", "location": {"path": "pipeline/outputs/s3.md", "range": {"start": {"line": 83, "column": 119}}}, "severity": "INFO"}
| `sse_kms_key_id` | AWS KMS key ARN for server-side encryption. Only applicable when `sse` is set to `aws:kms` or `aws:kms:dsse`. If not specified, the default AWS-managed KMS key for S3 will be used. | _none_ |

Check warning on line 84 in pipeline/outputs/s3.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Acronyms] Spell out 'KMS', if it's unfamiliar to the audience. Raw Output: {"message": "[FluentBit.Acronyms] Spell out 'KMS', if it's unfamiliar to the audience.", "location": {"path": "pipeline/outputs/s3.md", "range": {"start": {"line": 84, "column": 195}}}, "severity": "INFO"}

Check warning on line 84 in pipeline/outputs/s3.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Acronyms] Spell out 'KMS', if it's unfamiliar to the audience. Raw Output: {"message": "[FluentBit.Acronyms] Spell out 'KMS', if it's unfamiliar to the audience.", "location": {"path": "pipeline/outputs/s3.md", "range": {"start": {"line": 84, "column": 47}}}, "severity": "INFO"}
| `static_file_path` | Disables behavior where UUID string appends to the end of the S3 key name when `$UUID` isn't provided in `s3_key_format`. `$UUID`, time formatters, `$TAG`, and other dynamic key formatters all work as expected while this feature is set to true. | `false` |
| `store_dir` | Directory to locally buffer data before sending. Plugin uses the S3 Multipart upload API to send data in chunks of 5 MB at a time. | `/tmp/fluent-bit/s3` |
| `store_dir_limit_size` | S3 plugin has its own buffering system with files in the `store_dir`. Use the `store_dir_limit_size` to limit the amount of data S3 buffers in the `store_dir` to limit disk usage. If the limit is reached, data will be discarded. Default is 0 which means unlimited. | `0` |
Expand Down Expand Up @@ -574,8 +576,48 @@
{% endtab %}
{% endtabs %}

An example using SSE-KMS encryption:

Check warning on line 579 in pipeline/outputs/s3.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Acronyms] Spell out 'KMS', if it's unfamiliar to the audience. Raw Output: {"message": "[FluentBit.Acronyms] Spell out 'KMS', if it's unfamiliar to the audience.", "location": {"path": "pipeline/outputs/s3.md", "range": {"start": {"line": 579, "column": 22}}}, "severity": "INFO"}

Check warning on line 579 in pipeline/outputs/s3.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Acronyms] Spell out 'SSE', if it's unfamiliar to the audience. Raw Output: {"message": "[FluentBit.Acronyms] Spell out 'SSE', if it's unfamiliar to the audience.", "location": {"path": "pipeline/outputs/s3.md", "range": {"start": {"line": 579, "column": 18}}}, "severity": "INFO"}

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:

outputs:
- name: s3
match: '*'
bucket: your-bucket
region: us-east-1
store_dir: /home/ec2-user/buffer
total_file_size: 50M
upload_timeout: 10m
sse: aws:kms
sse_kms_key_id: arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[OUTPUT]
Name s3
Match *
bucket your-bucket
region us-east-1
store_dir /home/ec2-user/buffer
total_file_size 50M
upload_timeout 10m
sse aws:kms
sse_kms_key_id arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012
```

{% endtab %}
{% endtabs %}

## AWS for Fluent Bit


Amazon distributes a container image with Fluent Bit and plugins.

### GitHub
Expand Down
Loading