[Bug]: PubsubUnboundedSink not following 1000 messages per batch limit

[stankiewicz]

What happened?

PubsubUnboundedSink's WriterFn is triggered every batchSize. WriterFn can receive many more messages in one go, leading to pubsub rejecting batch.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[stankiewicz]

resource limit - https://docs.cloud.google.com/pubsub/quotas#resource_limits

[stankiewicz]

missing check https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSink.java#L357