Skip to content

[Bug]: JavaFieldSchema does not support for Beam Logical Types #37524

@barunkumaracharya

Description

@barunkumaracharya

What happened?

I had a Iceberg Row which had a column called "eventTime". The eventTime column datatype was org.apache.beam.sdk.schemas.logicaltypes.DateTime.

I wanted to convert the Row DataType to my java class and for this purpose, i was trying to use the Convert.FromRow transform
There is no test case that converts a logical beam type to java type in ConvertTest.java

I have tried many data types and options but I was not able to convert it to a Java DataType. I have tried Java Instant, Java LocalDateTime, Joda Instant / LocalDateTime but was still not able to convert it. One of the main reasons that i observed was that the Static Schema Inference class that JavaFieldSchema schemaProvider class uses to determine the Beam Type from a Java Type does not have support for Beam Logical Type. Checkout function fieldFromType in this file

This was the error that i was continuously facing when i was trying to convert it -
"Cannot convert between types that don't have equivalent schemas."
which is present in the ConvertHelpers.java class.

Requesting Support from the beam community to help fix this. I am currently relying on jackson object mapper to parse all fields and their values from the beam row instance and put it to a map which i m then using jackson again to convert the map to my pojo object.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions