sunchao commented on code in PR #5367:
URL: https://github.com/apache/datafusion-comet/pull/5367#discussion_r3816937259
##########
native/spark-expr/src/conversion_funcs/numeric.rs:
##########
@@ -852,15 +853,13 @@ pub(crate) fn spark_cast_int_to_int(
pub(crate) fn spark_cast_decimal_to_boolean(array: &dyn Array) ->
SparkResult<ArrayRef> {
let decimal_array = array.as_primitive::<Decimal128Type>();
- let mut result = BooleanBuilder::with_capacity(decimal_array.len());
- for i in 0..decimal_array.len() {
- if decimal_array.is_null(i) {
- result.append_null()
- } else {
- result.append_value(!decimal_array.value(i).is_zero());
- }
- }
- Ok(Arc::new(result.finish()))
+ // Arrow has no Decimal-to-Boolean cast. `neq` against a zero of the same
+ // precision/scale is exactly `!value.is_zero()`, including null handling.
+ let zero = Scalar::new(
+ Decimal128Array::from(vec![0i128])
+ .with_precision_and_scale(decimal_array.precision(),
decimal_array.scale())?,
Review Comment:
[P2] Preserve all-null `DECIMAL(0,0)` casts
Could we preserve the all-null case before constructing this scalar? Spark
accepts an RDD-backed nullable `DecimalType(0,0)` column, and `SELECT CAST(d AS
BOOLEAN)` returns null. With `spark.comet.sparkToColumnar.enabled=true` and
`spark.comet.sparkToColumnar.supportedOperatorList=RDDScan`, Comet's
row-to-Arrow and FFI paths can carry that schema for null rows.
I reproduced the Spark behavior on 3.5.2 and the native cast separately on
this head. Both an all-null `Decimal128(0,0)` array and
`ScalarValue::Decimal128(None, 0, 0)` now fail with `precision cannot be 0, has
to be between [1, 38]`. The previous loop returned null without validating the
precision. The input-column cast survives Spark optimization, so this is not
limited to a folded null literal.
An all-null fast path before `with_precision_and_scale`, with a regression
test, would preserve the previous behavior. This does not assume non-null
precision-zero values are supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
