Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion pipeline/outputs/firehose.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ This plugin uses the following configuration parameters:
| Key | Description | Default |
| :--- | :--- | --------- |
| `auto_retry_requests` | Immediately retry failed requests to AWS services once. This option doesn't affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput when there are transient/random networking issues. | `true` |
| `compression` | Compression type for Firehose records. Each log record is individually compressed and sent to Firehose. Supported values: `gzip`, `arrow`. `arrow` is only available if Apache Arrow was enabled at compile time. | _none_ |
| `compression` | Compression type for records sent to Firehose. Supported values: `gzip`, `zstd`, `snappy`. `arrow` is also available if Apache Arrow was enabled at compile time. See the [Compression](#compression). | _none_ |
| `delivery_stream` | The name of the Kinesis Firehose Delivery stream that you want log records sent to. | _none_ |
| `endpoint` | Specify a custom endpoint for the Firehose API. | _none_ |
| `external_id` | Specify an external ID for the STS API. You can use this option with the `role_arn` parameter if your role requires an external ID. | _none_ |
Expand Down Expand Up @@ -91,6 +91,16 @@ The following AWS IAM permissions are required to use this plugin:
}
```

## Compression

When you enable compression using the `compression` parameter, records are compressed before upload to Kinesis Firehose.

{% hint style="info" %}

Fluent Bit compresses each log record individually before sending to Firehose. Firehose then buffers multiple records and delivers them as complete files to the destination. Consumers receive these files (not individual records). If destination-level compression is enabled in Firehose (for example, S3 `CompressionFormat`), consumers must first decompress the file and then decompress each record. If destination-level compression is disabled, only per-record decompression is required.

{% endhint %}

### Worker support

Fluent Bit 1.7 added a new feature called `workers` which enables outputs to have dedicated threads. This `kinesis_firehose` plugin fully supports workers.
Expand Down
12 changes: 12 additions & 0 deletions pipeline/outputs/kinesis.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ For information about how AWS credentials are fetched, see [AWS credentials](../
| Key | Description | Default |
| --- | ----------- | ------- |
| `auto_retry_requests` | Immediately retry failed requests to AWS services once. This option doesn't affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which might help improve throughput when there are transient/random networking issues. | `true` |
| `compression` | Compression type for records sent to Kinesis Data Streams. Supported values: `gzip`, `zstd`, `snappy`. See [Compression](#compression). | _none_ |
| `endpoint` | Specify a custom endpoint for the Kinesis API. | _none_ |
| `external_id` | Specify an external ID for the STS API. You can use this option with the `role_arn` parameter if your role requires an external ID. | _none_ |
| `log_key` | By default, the whole log record will be sent to Kinesis. If you specify a key name with this option, then only the value of that key will be sent to Kinesis. For example, if you are using the Fluentd Docker log driver, you can specify `log_key log` and only the log message will be sent to Kinesis. | _none_ |
Expand Down Expand Up @@ -91,6 +92,17 @@ The following AWS IAM permissions are required to use this plugin:
}
```

## Compression

When you enable compression using the `compression` parameter, records are compressed before upload to Kinesis Data Streams.


{% hint style="info" %}

Each log record is individually compressed by Fluent Bit before sending. Consumers must decompress each record using the same compression format.

{% endhint %}

## Container images

Amazon distributes a container image with Fluent Bit and these plugins.
Expand Down
12 changes: 11 additions & 1 deletion pipeline/outputs/s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ The [Prometheus success/retry/error metrics values](../../administration/monitor
| `blob_database_file` | Absolute path to a database file to be used to store blob files contexts. | _none_ |
| `bucket` | S3 Bucket name | _none_ |
| `canned_acl` | [Predefined Canned ACL policy](https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl) for S3 objects. | _none_ |
| `compression` | Compression/format for S3 objects. Supported: `gzip` (always available) and `parquet` (requires Arrow build). For `gzip`, the `Content-Encoding` header is set to `gzip`. `parquet` is available **only when Fluent Bit is built with `-DFLB_ARROW=On`** and Arrow GLib/Parquet GLib are installed. Parquet is typically used with `use_put_object On`. | _none_ |
| `compression` | Compression type for S3 objects. Supported values: `gzip`, `zstd`, `snappy`. `arrow` and `parquet` are also available if Apache Arrow was enabled at compile time. See [Compression](#compression). | _none_ |
| `content_type` | A standard MIME type for the S3 object, set as the Content-Type HTTP header. | _none_ |
| `endpoint` | Custom endpoint for the S3 API. Endpoints can contain scheme and port. | _none_ |
| `external_id` | Specify an external ID for the STS API. Can be used with the `role_arn` parameter if your role requires an external ID. | _none_ |
Expand Down Expand Up @@ -112,6 +112,16 @@ The [Prometheus success/retry/error metrics values](../../administration/monitor

To skip TLS verification, set `tls.verify` as `false`. For more details about the properties available and general configuration, refer to [TLS/SSL](../../administration/transport-security.md).

## Compression

When you enable compression using the `compression` parameter, S3 objects are compressed before upload.

{% hint style="info" %}

Fluent Bit compresses data before uploading to S3. Consumers must decompress the objects using the same compression format.

{% endhint %}

## Permissions

The plugin requires the following AWS IAM permissions:
Expand Down