-
Notifications
You must be signed in to change notification settings - Fork 4.5k
Description
What happened?
I had a Iceberg Row which had a column called "eventTime". The eventTime column datatype was org.apache.beam.sdk.schemas.logicaltypes.DateTime.
I wanted to convert the Row DataType to my java class and for this purpose, i was trying to use the Convert.FromRow transform
There is no test case that converts a logical beam type to java type in ConvertTest.java
I have tried many data types and options but I was not able to convert it to a Java DataType. I have tried Java Instant, Java LocalDateTime, Joda Instant / LocalDateTime but was still not able to convert it. One of the main reasons that i observed was that the Static Schema Inference class that JavaFieldSchema schemaProvider class uses to determine the Beam Type from a Java Type does not have support for Beam Logical Type. Checkout function fieldFromType in this file
This was the error that i was continuously facing when i was trying to convert it -
"Cannot convert between types that don't have equivalent schemas."
which is present in the ConvertHelpers.java class.
Requesting Support from the beam community to help fix this. I am currently relying on jackson object mapper to parse all fields and their values from the beam row instance and put it to a map which i m then using jackson again to convert the map to my pojo object.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner