You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Spark 4.1 introduced TimeType (HH:mm:ss). Spark 4.1's Cast.scala adds many TimeType arms (StringType <-> TimeType, TimeType <-> TimeType, TimeType -> IntegralType / DecimalType, etc.). CometCast.isSupported has no TimeType arm at all, so every TimeType-involving cast falls into the default case _ => unsupported(fromType, toType) and falls back to Spark.
The fallback is correct, but it is implicit, and getUnsupportedReasons() on CometCast does not enumerate TimeType as one of the unsupported branches, so the auto-generated compatibility guide doesn't mention it.
Surfaced by the cast audit (collection PR queue).
Expected behavior
Either:
Add an explicit TimeType arm to CometCast.isSupported that returns Unsupported(Some("Cast to/from TimeType is not yet supported")), OR
Describe the bug
Spark 4.1 introduced
TimeType(HH:mm:ss). Spark 4.1'sCast.scalaadds manyTimeTypearms (StringType <-> TimeType,TimeType <-> TimeType,TimeType -> IntegralType / DecimalType, etc.).CometCast.isSupportedhas noTimeTypearm at all, so every TimeType-involving cast falls into the defaultcase _ => unsupported(fromType, toType)and falls back to Spark.The fallback is correct, but it is implicit, and
getUnsupportedReasons()onCometCastdoes not enumerateTimeTypeas one of the unsupported branches, so the auto-generated compatibility guide doesn't mention it.Surfaced by the cast audit (collection PR queue).
Expected behavior
Either:
TimeTypearm toCometCast.isSupportedthat returnsUnsupported(Some("Cast to/from TimeType is not yet supported")), ORTimeTypecast (depends onTimeTypebeing supported broadly across Comet, tracked by the broader date/time epic [EPIC] Implement all Spark date/time expressions #4418).The minimum here is the explicit
Unsupportedarm so the compatibility guide reflects it.Additional context
Cast.scalain/tmp/spark-v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/.CometCast.scala.