Skip to content

Add Observability Pipelines Amazon S3 Generic destination #3296

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
api-clients-generation-pipeline wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add Observability Pipelines Amazon S3 Generic destination #3296

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
195 changes: 186 additions & 9 deletions .generator/schemas/v2/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41994,6 +41994,191 @@ components:
type: string
x-enum-varnames:
- AMAZON_S3
ObservabilityPipelineAmazonS3GenericBatchSettings:
description: Event batching settings
properties:
batch_size:
description: Maximum batch size in bytes.
example: 100000000
format: int64
type: integer
timeout_secs:
description: Maximum number of seconds to wait before flushing the batch.
example: 900
format: int64
type: integer
type: object
ObservabilityPipelineAmazonS3GenericCompression:
description: Compression algorithm applied to encoded logs.
oneOf:
- $ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericCompressionZstd'
- $ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericCompressionGzip'
- $ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericCompressionSnappy'
ObservabilityPipelineAmazonS3GenericCompressionGzip:
description: Gzip compression.
properties:
level:
description: Gzip compression level.
example: 6
format: int64
type: integer
type:
$ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericCompressionGzipType'
required:
- type
- level
type: object
ObservabilityPipelineAmazonS3GenericCompressionGzipType:
default: gzip
description: The compression type. Always `gzip`.
enum:
- gzip
example: gzip
type: string
x-enum-varnames:
- GZIP
ObservabilityPipelineAmazonS3GenericCompressionSnappy:
description: Snappy compression.
properties:
type:
$ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericCompressionSnappyType'
required:
- type
type: object
ObservabilityPipelineAmazonS3GenericCompressionSnappyType:
default: snappy
description: The compression type. Always `snappy`.
enum:
- snappy
example: snappy
type: string
x-enum-varnames:
- SNAPPY
ObservabilityPipelineAmazonS3GenericCompressionZstd:
description: Zstd compression.
properties:
level:
description: Zstd compression level.
example: 3
format: int64
type: integer
type:
$ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericCompressionZstdType'
required:
- type
- level
type: object
ObservabilityPipelineAmazonS3GenericCompressionZstdType:
default: zstd
description: The compression type. Always `zstd`.
enum:
- zstd
example: zstd
type: string
x-enum-varnames:
- ZSTD
ObservabilityPipelineAmazonS3GenericDestination:
description: 'The `amazon_s3_generic` destination sends your logs to an Amazon
S3 bucket.


**Supported pipeline types:** logs'
properties:
auth:
$ref: '#/components/schemas/ObservabilityPipelineAwsAuth'
batch_settings:
$ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericBatchSettings'
bucket:
description: S3 bucket name.
example: my-bucket
type: string
compression:
$ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericCompression'
encoding:
$ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericEncoding'
id:
description: Unique identifier for the destination component.
example: generic-s3-destination
type: string
inputs:
description: A list of component IDs whose output is used as the `input`
for this component.
example:
- filter-processor
items:
type: string
type: array
key_prefix:
description: Optional prefix for object keys.
type: string
region:
description: AWS region of the S3 bucket.
example: us-east-1
type: string
storage_class:
$ref: '#/components/schemas/ObservabilityPipelineAmazonS3DestinationStorageClass'
type:
$ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericDestinationType'
required:
- id
- type
- inputs
- bucket
- region
- storage_class
- encoding
- compression
type: object
x-pipeline-types:
- logs
ObservabilityPipelineAmazonS3GenericDestinationType:
default: amazon_s3_generic
description: The destination type. Always `amazon_s3_generic`.
enum:
- amazon_s3_generic
example: amazon_s3_generic
type: string
x-enum-varnames:
- GENERIC_ARCHIVES_S3
ObservabilityPipelineAmazonS3GenericEncoding:
description: Encoding format for the destination.
oneOf:
- $ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericEncodingJson'
- $ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericEncodingParquet'
ObservabilityPipelineAmazonS3GenericEncodingJson:
description: JSON encoding.
properties:
type:
$ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericEncodingJsonType'
required:
- type
type: object
ObservabilityPipelineAmazonS3GenericEncodingJsonType:
default: json
description: The encoding type. Always `json`.
enum:
- json
example: json
type: string
x-enum-varnames:
- JSON
ObservabilityPipelineAmazonS3GenericEncodingParquet:
description: Parquet encoding.
properties:
type:
$ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericEncodingParquetType'
required:
- type
type: object
ObservabilityPipelineAmazonS3GenericEncodingParquetType:
default: parquet
description: The encoding type. Always `parquet`.
enum:
- parquet
example: parquet
type: string
x-enum-varnames:
- PARQUET
ObservabilityPipelineAmazonS3Source:
description: 'The `amazon_s3` source ingests logs from an Amazon S3 bucket.

Expand Down Expand Up @@ -42269,6 +42454,7 @@ components:
- $ref: '#/components/schemas/ObservabilityPipelineHttpClientDestination'
- $ref: '#/components/schemas/ObservabilityPipelineAmazonOpenSearchDestination'
- $ref: '#/components/schemas/ObservabilityPipelineAmazonS3Destination'
- $ref: '#/components/schemas/ObservabilityPipelineAmazonS3GenericDestination'
- $ref: '#/components/schemas/ObservabilityPipelineAmazonSecurityLakeDestination'
- $ref: '#/components/schemas/AzureStorageDestination'
- $ref: '#/components/schemas/ObservabilityPipelineCloudPremDestination'
Expand Down Expand Up @@ -46462,15 +46648,6 @@ components:
description: Optional name of the Splunk index where logs are written.
example: main
type: string
indexed_fields:
description: List of log field names to send as indexed fields to Splunk
HEC. Available only when `encoding` is `json`.
example:
- service
- host
items:
type: string
type: array
inputs:
description: A list of component IDs whose output is used as the `input`
for this component.
Expand Down
105 changes: 105 additions & 0 deletions docs/datadog_api_client.v2.model.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18365,6 +18365,111 @@ datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_destination\_
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_batch\_settings module
--------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_batch_settings
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_compression module
----------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_compression
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_compression\_gzip module
----------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_compression_gzip
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_compression\_gzip\_type module
----------------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_compression_gzip_type
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_compression\_snappy module
------------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_compression_snappy
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_compression\_snappy\_type module
------------------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_compression_snappy_type
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_compression\_zstd module
----------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_compression_zstd
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_compression\_zstd\_type module
----------------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_compression_zstd_type
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_destination module
----------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_destination
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_destination\_type module
----------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_destination_type
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_encoding module
-------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_encoding
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_encoding\_json module
-------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_encoding_json
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_encoding\_json\_type module
-------------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_encoding_json_type
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_encoding\_parquet module
----------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_encoding_parquet
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_generic\_encoding\_parquet\_type module
----------------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_amazon_s3_generic_encoding_parquet_type
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_amazon\_s3\_source module
--------------------------------------------------------------------------------

Expand Down
45 changes: 45 additions & 0 deletions src/datadog_api_client/v2/model/observability_pipeline_amazon_s3_generic_batch_settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License.
# This product includes software developed at Datadog (https://www.datadoghq.com/).
# Copyright 2019-Present Datadog, Inc.
from __future__ import annotations

from typing import Union

from datadog_api_client.model_utils import (
ModelNormal,
cached_property,
unset,
UnsetType,
)


class ObservabilityPipelineAmazonS3GenericBatchSettings(ModelNormal):
@cached_property
def openapi_types(_):
return {
"batch_size": (int,),
"timeout_secs": (int,),
}

attribute_map = {
"batch_size": "batch_size",
"timeout_secs": "timeout_secs",
}

def __init__(
self_, batch_size: Union[int, UnsetType] = unset, timeout_secs: Union[int, UnsetType] = unset, **kwargs
):
"""
Event batching settings

:param batch_size: Maximum batch size in bytes.
:type batch_size: int, optional

:param timeout_secs: Maximum number of seconds to wait before flushing the batch.
:type timeout_secs: int, optional
"""
if batch_size is not unset:
kwargs["batch_size"] = batch_size
if timeout_secs is not unset:
kwargs["timeout_secs"] = timeout_secs
super().__init__(kwargs)
Loading
Loading