Skip to content

[Doc] CAST collated-string handling on Spark 4.0+ is implicit and untested #4489

@andygrove

Description

@andygrove

Describe the bug

Spark 4.0+ supports collated StringType (e.g. STRING COLLATE UTF8_LCASE). CometCast.isSupported matches both source and target string types via case (DataTypes.StringType, _) => ... and case (_, DataTypes.StringType) => ..., where DataTypes.StringType is the singleton default-collation StringType instance.

Scala pattern equality means a non-default-collation StringType instance should NOT match DataTypes.StringType, and the cast would fall through to the default unsupported(...) branch and fall back to Spark. This appears to be the intended (safe) behaviour, but it is implicit: there is no isStringCollationType guard like the other string-touching serdes use (arrays.scala::CometArrayIntersect, QueryPlanSerde::supportedScalarSortElementType). And there is no test that asserts the fallback for CAST(c AS STRING COLLATE UTF8_LCASE) and CAST(c AS STRING) when c is collated.

If the equality ever yields true (e.g. via a future refactor of StringType.equals), Comet would silently route collated-string casts through the byte-oriented native path, producing incorrect results for downstream collation-aware comparisons / aggregations / hashing.

Surfaced by the cast audit (collection PR queue). Tracked under the umbrella #2190 for Spark 4.0 collation support.

Expected behavior

Either:

  1. Add an explicit isStringCollationType guard in CometCast.isSupported so the fallback is declared rather than relying on Scala pattern semantics, OR
  2. Add tests asserting that CAST to/from a non-default StringType collation falls back to Spark and does not run native.

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions