[Feature Request]: Add a Variant type to Beam Schemas

[ahmedabu98]

What would you like to happen?

PCollections of Beam Rows are required to have a fixed schema, making it hard to read or write records with varying logical schemas in the same PCollection.

We need a semi-structured type that can contain a nested level of variable columns. Transforms can choose to unwrap this type to reconstruct the variable columns accordingly.

The Variant type is becoming a good standard across different projects (see Parquet, Spark, Flink). It could also be a good candidate for Beam.

Bonus if such a type is made portable so that other SDKs can use it too

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[chamikaramj]

Noting that lakehouse solutions Delta Lake and Iceberg recently introduced support for the Variant type.

https://www.databricks.com/blog/introducing-open-variant-data-type-delta-lake-and-apache-spark

https://www.snowflake.com/en/engineering-blog/apache-iceberg-v3-variant-type/

So probably a good idea to add this support natively to Beam as well.