Skip to content

bug: OOM error inserting parquet with partition #121

@subkanthi

Description

@subkanthi
../../ice/ice/target/ice-jar insert ontime.ontime_by_month -p ../../../datasets/ontime/parquet/ontime.parquet --partition='[{"column":"FlightDate","transform":"month"}]' --insecure
2026-02-26 11:32:56 [main] WARN com.altinity.ice.cli.Main > SSL certificate verification is DISABLED. This is insecure and should only be used for development.
2026-02-26 11:32:57 [main] INFO c.a.i.c.internal.cmd.Insert > Updating write.distribution-mode to "range"
2026-02-26 11:32:57 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > ../../../datasets/ontime/parquet/ontime.parquet: processing
2026-02-26 11:32:57 [-5-thread-1] WARN c.a.i.c.internal.cmd.Insert > ../../../datasets/ontime/parquet/ontime.parquet: File contains multiple partition values for 'FlightDate' (min: 213, max: 643). Falling back to full scan (slow)
2026-02-26 11:32:57 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > ../../../datasets/ontime/parquet/ontime.parquet: partitioning
2026-02-26 11:35:58 [main] ERROR com.altinity.ice.cli.Main > Fatal
java.io.IOException: Error processing file(s)
        at com.altinity.ice.cli.internal.cmd.Insert.run(Insert.java:231)
        at com.altinity.ice.cli.Main.insert(Main.java:512)
        at com.altinity.ice.cli.Main.lambda$main$1(Main.java:779) [9 skipped]
        at com.altinity.ice.cli.Main.main(Main.java:786) [1 skipped]
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.iceberg.data.GenericRecord.<init>(GenericRecord.java:71) [1 skipped]
        at org.apache.iceberg.data.GenericRecord.copy(GenericRecord.java:134)
        at org.apache.iceberg.parquet.ParquetValueReaders$RecordReader.newStructData(ParquetValueReaders.java:1005)
        at org.apache.iceberg.parquet.ParquetValueReaders$RecordReader.newStructData(ParquetValueReaders.java:990)
        at org.apache.iceberg.parquet.ParquetValueReaders$StructReader.read(ParquetValueReaders.java:913)
        at org.apache.iceberg.parquet.ParquetReader$FileIterator.next(ParquetReader.java:130)
        at com.altinity.ice.cli.internal.iceberg.Partitioning.partition(Partitioning.java:266)
        at com.altinity.ice.cli.internal.cmd.Insert.copyPartitionedAndSorted(Insert.java:597)
        at com.altinity.ice.cli.internal.cmd.Insert.processFile(Insert.java:505)
        at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$2(Insert.java:194)
        at com.altinity.ice.cli.internal.cmd.Insert$$Lambda/0x00007b02f830df28.call(Unknown Source)
 [5 skipped]

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingp1

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions