[SPARK-56844][SQL] Support ArrayType / MapType / StructType in ConstantColumnVector and FileSourceMetadataAttribute

[mzhang]

What changes were proposed in this pull request?

Allow ArrayType, MapType, and StructType in file source constant metadata attributes. Concretely:

• FileSourceMetadataAttribute.isSupportedType allows complex types recursively, contingent on their element types being supported. Behavior for primitives, decimal, string, binary, interval, and variant is unchanged.
• ColumnVectorUtils.populate gains struct/array/map branches:
  • Struct: recurse into pre-allocated child ConstantColumnVectors.
  • Array/map: allocate a one-row OffHeapColumnVector backing and reuse the existing RowToColumnConverter (wrapped in a single-field struct schema) to write the constant value. The view is handed to the constant vector along with ownership of the backing.
• ConstantColumnVector gains optional ownership of a backing WritableColumnVector (closed by close()), exposed via new setArrayWithBacking / setMapWithBacking. The original setArray / setMap are unchanged (caller retains ownership).
• ConstantColumnVector's constructor pre-allocates struct children so populate's struct recursion has a target. setChild closes any previously-set child to avoid leaking the auto-allocated one.

Notes on correctness: the recursive copy for array and map reuses RowToColumnConverter, which already drives row-to-columnar conversion across all supported types (RowToColumnarExec). No new per-type dispatch logic is introduced.

Why are the changes needed?

FileSourceMetadataAttribute.isSupportedType is the lone gate preventing complex-typed file source constant metadata; the underlying machinery already supports them.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New cases in ColumnVectorUtilsSuite replacing the previous "not supported" assertions:

• fill array of ints
• fill array of strings
• fill map of int -> boolean
• fill struct
• fill nested array<struct> (covers element-level nulls)
• fill null array

Existing ConstantColumnVectorSuite cases continue to exercise the same paths.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude (Anthropic)

[cloud-fan]

The PR widens FileSourceMetadataAttribute.isSupportedType to accept ArrayType/MapType/StructType recursively, and teaches ColumnVectorUtils.populate how to write those types into a ConstantColumnVector. The underlying columnar infrastructure (RowToColumnConverter, OffHeapColumnVector) already supports the element types, so this is mostly plumbing.

Two intentional code paths:

• Struct recurses into per-field ConstantColumnVector children (auto-allocated by the constructor). Keeps getChild(i) returning a ConstantColumnVector, which existing code (e.g. setCalendarInterval, setVariant) relies on.
• Array/map allocates a WritableColumnVector(1, t) backing, drives it via RowToColumnConverter with a one-field wrapper struct schema, and hands the ColumnarArray/ColumnarMap view plus ownership of the backing to the ConstantColumnVector via the new setArrayWithBacking/setMapWithBacking. ConstantColumnVector.close() releases the owned backing.

This asymmetry is justified — array/map element data is variable-length and doesn't fit the "one constant value per row" model, so it needs real backing storage and explicit ownership transfer.

A few comments below — the main one is matching the codebase's on-heap/off-heap convention.

Coverage suggestion (not blocking): consider adding to ColumnVectorUtilsSuite (a) a struct with a null field — exercises the inner.isNullAt(i) ? null : ... branch in populate's struct path — and (b) a struct-of-struct case.

[cloud-fan]

LGTM. 3 addressed, 0 remaining, 1 new (1 newly introduced — import order). Thanks for threading MemoryMode through cleanly and adding the null-field / nested-struct test cases.