Skip to content

Inserting file with inferred partition key does not write to partition folder #119

@xieandrew

Description

@xieandrew

When a file contains only a single partition value (e.g. partition by Year column and file only has value '1988' for Year), ice can infer the partition value and copy the entire parquet file directly. However, when copying the file, ice assumes the table is not partitioned and copies it directly to the table_name/data folder instead of the partitioned folder table_name/data/Year=1988.

Example:

ice insert ontime.ontime_by_year -p s3://example-bucket/ontime_1988.parquet --partition='[{"column":"Year"}]'

2026-02-26 12:55:34 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > s3://example-bucket/ontime_1988.parquet: processing
2026-02-26 12:55:35 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > s3://example-bucket/ontime_1988.parquet: using inferred partition key [1988]
2026-02-26 12:55:35 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > s3://example-bucket/ontime_1988.parquet: copying to s3://result-bucket/ontime/ontime_by_year/data/1772128534442-3b1f0bab2b12cb575867f61727c1e6f5715c9c203221a933e3ae60e6d7bc3fa9.parquet
2026-02-26 12:56:15 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > s3://example-bucket/ontime_1988.parquet: adding data file (copy took 40s)
2026-02-26 12:56:16 [main] INFO o.a.i.SnapshotProducer > Committed snapshot 6165517712435602678 (MergeAppend)

Metadata

Metadata

Assignees

No one assigned

    Labels

    iceRelates to icep1

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions