Describe the bug
CometCast.isSupported has no MapType arm; any cast involving a MapType source or target falls into the default case _ => unsupported(fromType, toType) and falls back to Spark.
However, the native side already has a cast_map_to_map function (see native/spark-expr/src/conversion_funcs/cast.rs around line 423). The Scala side just never routes to it, so CAST(<map> AS MAP<...>) falls back unnecessarily.
Surfaced by the cast audit (collection PR queue).
Expected behavior
Add a (MapType, MapType) arm to CometCast.isSupported that recursively checks the source / target key and value types and returns Compatible (or Incompatible(reason)) when both inner casts are supported. Wire the recursion the same way Array -> Array and Struct -> Struct are handled in CometCast.scala.
Additional context
- Native impl:
native/spark-expr/src/conversion_funcs/cast.rs::cast_map_to_map
- Comet matrix:
CometCast.scala
- For
Map -> String, Spark formats as {k1 -> v1, k2 -> v2}; the native path would need a cast_map_to_string analogous to the existing cast_struct_to_string. Out of scope here; this issue is only the Map -> Map case.
Describe the bug
CometCast.isSupportedhas noMapTypearm; any cast involving aMapTypesource or target falls into the defaultcase _ => unsupported(fromType, toType)and falls back to Spark.However, the native side already has a
cast_map_to_mapfunction (seenative/spark-expr/src/conversion_funcs/cast.rsaround line 423). The Scala side just never routes to it, soCAST(<map> AS MAP<...>)falls back unnecessarily.Surfaced by the cast audit (collection PR queue).
Expected behavior
Add a
(MapType, MapType)arm toCometCast.isSupportedthat recursively checks the source / target key and value types and returnsCompatible(orIncompatible(reason)) when both inner casts are supported. Wire the recursion the same wayArray -> ArrayandStruct -> Structare handled inCometCast.scala.Additional context
native/spark-expr/src/conversion_funcs/cast.rs::cast_map_to_mapCometCast.scalaMap -> String, Spark formats as{k1 -> v1, k2 -> v2}; the native path would need acast_map_to_stringanalogous to the existingcast_struct_to_string. Out of scope here; this issue is only theMap -> Mapcase.