-
Notifications
You must be signed in to change notification settings - Fork 5
Open
Labels
Description
When a file contains only a single partition value (e.g. partition by Year column and file only has value '1988' for Year), ice can infer the partition value and copy the entire parquet file directly. However, when copying the file, ice assumes the table is not partitioned and copies it directly to the table_name/data folder instead of the partitioned folder table_name/data/Year=1988.
Example:
ice insert ontime.ontime_by_year -p s3://example-bucket/ontime_1988.parquet --partition='[{"column":"Year"}]'
2026-02-26 12:55:34 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > s3://example-bucket/ontime_1988.parquet: processing
2026-02-26 12:55:35 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > s3://example-bucket/ontime_1988.parquet: using inferred partition key [1988]
2026-02-26 12:55:35 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > s3://example-bucket/ontime_1988.parquet: copying to s3://result-bucket/ontime/ontime_by_year/data/1772128534442-3b1f0bab2b12cb575867f61727c1e6f5715c9c203221a933e3ae60e6d7bc3fa9.parquet
2026-02-26 12:56:15 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > s3://example-bucket/ontime_1988.parquet: adding data file (copy took 40s)
2026-02-26 12:56:16 [main] INFO o.a.i.SnapshotProducer > Committed snapshot 6165517712435602678 (MergeAppend)
Reactions are currently unavailable