[Feature Request]: Support large pipeline options in Python SDK

[shunping]

What would you like to happen?

Currently, if we run a pipeline with a large pipeline option in Python on Dataflow, we would see an error Python exited: fork/exec /usr/local/bin/python: argument list too long.

We have solved similar issues in Java and Go(#27839 (comment)), and we should also support this in Python.

Internal bug id: 472722314

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Midhat-kazmi]

.take-issue

[Midhat-kazmi]

I see that PR #37379 is actively addressing this issue.
I’ll step back for now and follow the progress there.
Happy to help review or test if needed.