What happened?
When writing to an Iceberg table via IcebergIO on Dataflow, the pipeline fails with IllegalArgumentException during encoding of FileWriteResult. The root cause is:
java.lang.RuntimeException: Null value set on non-nullable field
Field{name=keyMetadata, description=, type=BYTES NOT NULL, options={{}}}
The SerializableDataFile schema at runtime marks all fields as NOT NULL, despite fields 6–13 (keyMetadata, splitOffsets, columnSizes, valueCounts, nullValueCounts, nanValueCounts, lowerBounds, upperBounds) being declared with @Nullable in the source code.
The @Nullable annotation used is org.checkerframework.checker.nullness.qual.Nullable (a TYPE_USE annotation), which AutoValueSchema / FieldValueTypeInformation.hasNullableReturnType() appears to not detect at runtime.
The error surfaces on keyMetadata because it is the first field that is actually null (normal for unencrypted Iceberg tables), but the issue affects all nullable fields in SerializableDataFile.
Issue Priority
Priority: 1 (data loss / total loss of function)
Issue Components
What happened?
When writing to an Iceberg table via
IcebergIOon Dataflow, the pipeline fails withIllegalArgumentExceptionduring encoding ofFileWriteResult. The root cause is:The
SerializableDataFileschema at runtime marks all fields as NOT NULL, despite fields 6–13 (keyMetadata,splitOffsets,columnSizes,valueCounts,nullValueCounts,nanValueCounts,lowerBounds,upperBounds) being declared with@Nullablein the source code.The
@Nullableannotation used isorg.checkerframework.checker.nullness.qual.Nullable(aTYPE_USEannotation), whichAutoValueSchema/FieldValueTypeInformation.hasNullableReturnType()appears to not detect at runtime.The error surfaces on
keyMetadatabecause it is the first field that is actuallynull(normal for unencrypted Iceberg tables), but the issue affects all nullable fields inSerializableDataFile.Issue Priority
Priority: 1 (data loss / total loss of function)
Issue Components