[Parquet]: GH-563: Make path_in_schema optional

[etseidl]

Which issue does this PR close?

none

Rationale for this change

This is a proof of concept implementation for apache/parquet-format#563

What changes are included in this PR?

Since version 57.0.0, this crate has been tolerant of a missing path_in_schema. This PR adds options to cease writing the field as well. The option defaults to continuing to write the field.

See related discussion on parquet mailing list: https://lists.apache.org/thread/czm2bk45wwtkhhpqxqvmx9dk5wkwk1kt

Are these changes tested?

Not as yet

Are there any user-facing changes?

No, this only adds an optional behavior change that defaults to no change

Related PRs

• Make path_in_schema optional parquet-format#563
• GH-563: Make ColumnMetaData.path_in_schema optional parquet-format#564
• [Parquet]: GH-563: Make path_in_schema optional parquet-java#3470

[alamb]

less bloat for the win!

[jhorstmann]

A lot of other parquet implementations require this field, due to their generated thrift parser, even if they do not actually use the field for anything. I would be totally in favor of deprecating and skipping the field, but maybe a more compatible alternative would be to write the field as an empty list instead.

[etseidl]

A lot of other parquet implementations require this field, due to their generated thrift parser, even if they do not actually use the field for anything. I would be totally in favor of deprecating and skipping the field, but maybe a more compatible alternative would be to write the field as an empty list instead.

Well, parquet-java actually uses the field to populate its version of ColumnDescriptor, so an empty list will be just as damaging to an old version. The whole idea is to change the field to optional in the thrift definition, and give the ecosystem a few years for that change to percolate. After some reasonable time has passed we can default to not writing the field. But in the meantime, users who have data sensitive to metadata bloat and know they have up-to-date tooling can help themselves earlier.

[alamb]

A lot of other parquet implementations require this field, due to their generated thrift parser, even if they do not actually use the field for anything. I would be totally in favor of deprecating and skipping the field, but maybe a more compatible alternative would be to write the field as an empty list instead.

Well, parquet-java actually uses the field to populate its version of ColumnDescriptor, so an empty list will be just as damaging to an old version. The whole idea is to change the field to optional in the thrift definition, and give the ecosystem a few years for that change to percolate. After some reasonable time has passed we can default to not writing the field. But in the meantime, users who have data sensitive to metadata bloat and know they have up-to-date tooling can help themselves earlier.

See also related mailing list discussion:

• https://lists.apache.org/thread/czm2bk45wwtkhhpqxqvmx9dk5wkwk1kt

[alamb]

TLDR is I think this is a great idea. I also think it woudl be ok to merge this into arrow-rs even if there is not a broader consensus on the mailing list that we should do it in the format itself

My thinking is that some usecases are basically using parquet with the same read/writer and compatibility with older java based implementations is not important. This is the same thing for some of the newer encodings too

Letting users choose between "better/faster/stronger" and "more compatible" I think is very much a good idea

[alamb]

If we choose to go this way I think it would help to give some more context here on what types of readers would be affected (basically all the parquet-java based readers prior to late 2026)

We could also perhaps provide a link to the discussion: https://lists.apache.org/thread/czm2bk45wwtkhhpqxqvmx9dk5wkwk1kt

[alamb]

I think it is worth making it more apparently that this will cause incompatibilities with some older readers:

Suggested change

	/// Should the writer should emit the `path_in_schema` element of the
	/// `ColumnMetaData` Thrift struct.
	/// Should the writer should emit the `path_in_schema` element of the
	/// `ColumnMetaData` Thrift struct. WARNING: this will make the parquet
	/// file unreadable by some older parquet readers. See LINK HERE for details

[alamb]

I added a list of related PRs to this issue's description:

• Make path_in_schema optional parquet-format#563
• GH-563: Make ColumnMetaData.path_in_schema optional parquet-format#564
• [Parquet]: GH-563: Make path_in_schema optional parquet-java#3470