mbutrovich commented on code in PR #350:
URL: https://github.com/apache/datafusion-spark/pull/350#discussion_r4615423261


##########
crates/spark/src/function/implicit_cast.rs:
##########
@@ -0,0 +1,774 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) 
under one or more contributor license agreements
+
+//! Casting a function's arguments to the input types it declares, as Spark 
does.
+//!
+//! Most Spark expressions mix in `ImplicitCastInputTypes` and declare
+//! `inputTypes: Seq[AbstractDataType]`. The analyzer then casts each argument 
with
+//! `implicitCast` from `TypeCoercion` (ANSI mode off) or `AnsiTypeCoercion` 
(ANSI mode on),
+//! and `ExpectsInputTypes.checkInputDataTypes` raises 
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`
+//! for an argument that cannot be cast.
+//!
+//! A function here declares the same types with [`SparkInputType`], holds 
them in a
+//! [`SparkInputTypes`], and takes its signature from it:
+//!
+//! ```ignore
+//! fn signature(&self) -> &Signature {
+//!     self.input_types.signature()
+//! }
+//!
+//! fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+//!     self.input_types.coerce(self.name(), arg_types)
+//! }
+//!
+//! fn with_updated_config(&self, config: &ConfigOptions) -> Option<ScalarUDF> 
{
+//!     let input_types = self.input_types.with_config(config)?;
+//!     Some(ScalarUDF::from(Self { input_types }))
+//! }
+//! ```
+//!
+//! This module only chooses the target types. The cast that runs is 
DataFusion's, so how a
+//! malformed string or an overflowing value is cast is part of the CAST work 
in #88.
+
+use arrow::datatypes::{DataType, IntervalUnit};
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::{Result, plan_err};
+use datafusion_expr::{Signature, Volatility};
+
+/// A type that a Spark function declares for one of its arguments: a concrete 
Spark type,
+/// or one of Spark's abstract types such as `NumericType`.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+#[allow(
+    dead_code,
+    reason = "declared for the functions that move onto these rules after sha2 
(#346)"
+)]
+pub(crate) enum SparkInputType {
+    /// `AnyDataType`: accepts every type and never casts.
+    Any,
+    /// `BooleanType`
+    Boolean,
+    /// `ByteType` (TINYINT)
+    Byte,
+    /// `ShortType` (SMALLINT)
+    Short,
+    /// `IntegerType` (INT)
+    Integer,
+    /// `LongType` (BIGINT)
+    Long,
+    /// `FloatType`
+    Float,
+    /// `DoubleType`
+    Double,
+    /// The abstract `NumericType`. Its default type is DOUBLE.
+    Numeric,
+    /// The abstract `IntegralType`. Its default type is INT.
+    Integral,
+    /// The abstract `DecimalType`: a decimal of any precision and scale. Its 
default type is
+    /// `DECIMAL(38,18)`.
+    Decimal,
+    /// `StringType`
+    String,
+    /// `BinaryType`
+    Binary,
+    /// `AbstractArrayType(element)`: an array whose elements are cast to 
`element`.
+    Array(Box<SparkInputType>),
+    /// `TypeCollection(types)`: the first type that the argument is accepted 
as, or else the
+    /// first that it can be cast to.
+    OneOf(Vec<SparkInputType>),
+}
+
+/// The input types a function declares, and whether ANSI mode is on, which 
changes the casts
+/// Spark allows.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct SparkInputTypes {
+    types: Vec<SparkInputType>,
+    ansi_mode: bool,
+    signature: Signature,
+}
+
+impl SparkInputTypes {
+    /// Declares one input type per argument, with ANSI mode off.
+    pub(crate) fn new(types: Vec<SparkInputType>, volatility: Volatility) -> 
Self {
+        Self {
+            types,
+            ansi_mode: false,
+            signature: Signature::user_defined(volatility),
+        }
+    }
+
+    /// The function's `Signature`, which leaves coercion to [`Self::coerce`].
+    pub(crate) fn signature(&self) -> &Signature {
+        &self.signature
+    }
+
+    /// Returns the input types for `config`, or `None` when ANSI mode is 
unchanged.
+    ///
+    /// Call it from `ScalarUDFImpl::with_updated_config`, which is how a 
function sees the
+    /// session's `datafusion.execution.enable_ansi_mode`.
+    pub(crate) fn with_config(&self, config: &ConfigOptions) -> Option<Self> {
+        let ansi_mode = config.execution.enable_ansi_mode;
+        (ansi_mode != self.ansi_mode).then(|| Self {
+            ansi_mode,
+            ..self.clone()
+        })
+    }
+
+    /// Casts each of `arg_types` to its declared type, for 
`ScalarUDFImpl::coerce_types`.
+    ///
+    /// Returns Spark's `WRONG_NUM_ARGS` error for the wrong number of 
arguments, and
+    /// `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` for an argument that cannot 
be cast.
+    pub(crate) fn coerce(
+        &self,
+        name: &str,
+        arg_types: &[DataType],
+    ) -> Result<Vec<DataType>> {
+        if arg_types.len() != self.types.len() {
+            return plan_err!(
+                "[WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `{name}` requires {} 
parameters but \
+                 the actual number is {}. Please, refer to \
+                 'https://spark.apache.org/docs/latest/sql-ref-functions.html' 
for a fix. \
+                 SQLSTATE: 42605",
+                self.types.len(),
+                arg_types.len()
+            );
+        }
+        arg_types
+            .iter()
+            .zip(&self.types)
+            .enumerate()
+            .map(|(idx, (arg_type, expected))| {
+                match implicit_cast(arg_type, expected, self.ansi_mode) {
+                    Some(target) => Ok(target),
+                    None => plan_err!(
+                        "[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The {} 
parameter of \
+                         `{name}` requires the {} type, however it has the 
type \"{}\". \
+                         SQLSTATE: 42K09",
+                        ordinal_number(idx),
+                        expected.sql(),
+                        spark_sql_type(arg_type)
+                    ),
+                }
+            })
+            .collect()
+    }
+}
+
+impl SparkInputType {
+    /// Spark's `AbstractDataType.acceptsType`: whether an argument of this 
type is used as it
+    /// is, without a cast.
+    fn accepts(&self, data_type: &DataType) -> bool {
+        use SparkInputType::*;
+        match self {
+            Any => true,
+            Boolean => matches!(data_type, DataType::Boolean),
+            Byte => matches!(data_type, DataType::Int8),
+            Short => matches!(data_type, DataType::Int16),
+            Integer => matches!(data_type, DataType::Int32),
+            Long => matches!(data_type, DataType::Int64),
+            Float => matches!(data_type, DataType::Float32),
+            Double => matches!(data_type, DataType::Float64),
+            Numeric => data_type.is_numeric(),
+            Integral => data_type.is_integer(),
+            Decimal => data_type.is_decimal(),
+            String => data_type.is_string(),
+            Binary => data_type.is_binary(),
+            Array(element) => {
+                list_element(data_type).is_some_and(|item| 
element.accepts(item))
+            }
+            OneOf(types) => types.iter().any(|t| t.accepts(data_type)),
+        }
+    }
+
+    /// Spark's `AbstractDataType.defaultConcreteType`: the type an argument 
is cast to when
+    /// the declared type is abstract. `None` for [`SparkInputType::Any`], 
which never casts.
+    fn default_type(&self) -> Option<DataType> {
+        use SparkInputType::*;
+        Some(match self {
+            Any => return None,
+            Boolean => DataType::Boolean,
+            Byte => DataType::Int8,
+            Short => DataType::Int16,
+            Integer | Integral => DataType::Int32,
+            Long => DataType::Int64,
+            Float => DataType::Float32,
+            Double | Numeric => DataType::Float64,
+            Decimal => DataType::Decimal128(38, 18),
+            String => DataType::Utf8,
+            Binary => DataType::Binary,
+            Array(element) => DataType::new_list(element.default_type()?, 
true),
+            OneOf(types) => return types.first()?.default_type(),
+        })
+    }
+
+    /// Whether this is a concrete Spark `AtomicType` (and not an abstract 
type).
+    fn is_concrete_atomic(&self) -> bool {
+        use SparkInputType::*;
+        self.is_concrete_numeric() || matches!(self, Boolean | String | Binary)
+    }
+
+    /// Whether this is a concrete Spark `NumericType` (and not `Numeric` or 
`Decimal`).
+    fn is_concrete_numeric(&self) -> bool {
+        use SparkInputType::*;
+        matches!(self, Byte | Short | Integer | Long | Float | Double)
+    }
+
+    /// The type as Spark's `toSQLType` writes it in an error message, with 
the quotes.
+    fn sql(&self) -> String {
+        use SparkInputType::*;
+        match self {
+            OneOf(types) => format!(
+                "({})",
+                types
+                    .iter()
+                    .map(|t| t.sql())
+                    .collect::<Vec<_>>()
+                    .join(" or ")
+            ),
+            other => format!("\"{}\"", other.simple_string()),
+        }
+    }
+
+    fn simple_string(&self) -> String {
+        use SparkInputType::*;
+        match self {
+            Any => "ANY",
+            Boolean => "BOOLEAN",
+            Byte => "TINYINT",
+            Short => "SMALLINT",
+            Integer => "INT",
+            Long => "BIGINT",
+            Float => "FLOAT",
+            Double => "DOUBLE",
+            Numeric => "NUMERIC",
+            Integral => "INTEGRAL",
+            Decimal => "DECIMAL",
+            String => "STRING",
+            Binary => "BINARY",
+            Array(element) => return format!("ARRAY<{}>", 
element.simple_string()),
+            // `toSQLType` upper-cases the whole `simpleString` of an abstract 
type, so a
+            // collection inside an array reads `ARRAY<(STRING OR BINARY)>`.
+            OneOf(types) => {
+                return format!(
+                    "({})",
+                    types
+                        .iter()
+                        .map(|t| t.simple_string())
+                        .collect::<Vec<_>>()
+                        .join(" OR ")
+                );
+            }
+        }
+        .to_string()
+    }
+}
+
+/// The type to cast an argument of type `from` to, for a declared `expected` 
type, or `None`
+/// if Spark does not cast it.
+///
+/// A port of the private `implicitCast(DataType, AbstractDataType)` in Spark's
+/// `TypeCoercion` (ANSI mode off) and `AnsiTypeCoercion` (ANSI mode on).
+fn implicit_cast(
+    from: &DataType,
+    expected: &SparkInputType,
+    ansi_mode: bool,
+) -> Option<DataType> {
+    let from = &spark_layout(from);
+    if expected.accepts(from) {
+        return Some(from.clone());
+    }
+    if ansi_mode {
+        ansi_implicit_cast(from, expected)
+    } else {
+        legacy_implicit_cast(from, expected)
+    }
+}
+
+/// The layout a function receives for an argument of a type Spark does not 
have.
+///
+/// A dictionary has the type of its values (see #349), and a fixed-size 
binary becomes
+/// BINARY, so that a function only has to handle the layouts of Spark's own 
types.
+fn spark_layout(data_type: &DataType) -> DataType {
+    match data_type {
+        DataType::Dictionary(_, value) => spark_layout(value),
+        DataType::FixedSizeBinary(_) => DataType::Binary,
+        other => other.clone(),
+    }
+}
+
+/// `TypeCoercion.implicitCast`, with ANSI mode off.
+fn legacy_implicit_cast(from: &DataType, expected: &SparkInputType) -> 
Option<DataType> {
+    use SparkInputType as T;
+    match (from, expected) {
+        (DataType::Null, target) => target.default_type(),
+        // Hive's convention: a string where any number is accepted becomes a 
DOUBLE.
+        (f, T::Numeric) if f.is_string() => Some(DataType::Float64),
+        (f, T::Decimal) if f.is_numeric() => decimal_for_type(f),
+        (f, target) if f.is_numeric() && target.is_concrete_numeric() => {
+            target.default_type()
+        }
+        (f, T::Decimal) if f.is_string() => T::Decimal.default_type(),
+        (f, target) if f.is_string() && target.is_concrete_numeric() => {
+            target.default_type()
+        }
+        (f, T::Binary) if f.is_string() => Some(binary_for_string(f)),
+        (f, T::String) if is_atomic(f) => Some(DataType::Utf8),
+        (f, T::OneOf(types)) => types.iter().find_map(|t| implicit_cast(f, t, 
false)),
+        (f, T::Array(element)) => {
+            let item = list_element(f)?;
+            implicit_cast(item, element, false).map(|t| DataType::new_list(t, 
true))
+        }
+        _ => None,
+    }
+}
+
+/// `AnsiTypeCoercion.implicitCast`, with ANSI mode on.
+fn ansi_implicit_cast(from: &DataType, expected: &SparkInputType) -> 
Option<DataType> {
+    use SparkInputType as T;
+    match (from, expected) {
+        (f, T::Decimal) if f.is_numeric() => decimal_for_type(f),
+        (f, T::Integral) if is_fractional(f) => None,
+        // Spark allows a string to be cast to any primitive type, to keep 
existing queries
+        // working, although `Cast.canANSIStoreAssign` does not.
+        (f, T::Binary) if f.is_string() => Some(binary_for_string(f)),
+        (f, target)
+            if f.is_string()
+                && (target.is_concrete_atomic()
+                    || matches!(target, T::Numeric | T::Decimal)) =>
+        {
+            target.default_type()
+        }
+        (f, T::Array(element)) if list_element(f).is_some() => {
+            let item = list_element(f)?;
+            implicit_cast(item, element, true).map(|t| DataType::new_list(t, 
true))
+        }
+        (f, T::OneOf(types)) => types.iter().find_map(|t| implicit_cast(f, t, 
true)),
+        (f, target) => {
+            let concrete = target.default_type()?;
+            can_ansi_store_assign(f, &concrete).then_some(concrete)
+        }
+    }
+}
+
+/// Spark's `Cast.canANSIStoreAssign`: whether a value of type `from` can be 
stored in a
+/// column of type `to` under ANSI rules.
+fn can_ansi_store_assign(from: &DataType, to: &DataType) -> bool {
+    if let (Some(from_item), Some(to_item)) = (list_element(from), 
list_element(to)) {
+        return can_ansi_store_assign(from_item, to_item);
+    }
+    match (from, to) {
+        _ if from == to => true,
+        (DataType::Null, _) => true,
+        (f, t) if f.is_numeric() && t.is_numeric() => true,
+        (f, t) if t.is_string() && (is_atomic(f) || is_calendar_interval(f)) 
=> true,
+        (f, t) if is_datetime(f) && is_datetime(t) => true,
+        (DataType::Map(from_entries, _), DataType::Map(to_entries, _)) => {
+            can_ansi_store_assign(from_entries.data_type(), 
to_entries.data_type())
+        }
+        (DataType::Struct(from_fields), DataType::Struct(to_fields)) => {
+            from_fields.len() == to_fields.len()
+                && from_fields.iter().zip(to_fields.iter()).all(|(f, t)| {
+                    (t.is_nullable() || !f.is_nullable())
+                        && can_ansi_store_assign(f.data_type(), t.data_type())
+                })
+        }
+        _ => false,
+    }
+}
+
+/// Spark's `DecimalType.forType`: the decimal that holds every value of a 
numeric type.
+fn decimal_for_type(data_type: &DataType) -> Option<DataType> {
+    use DataType::*;
+    let (precision, scale) = match data_type {
+        Int8 | UInt8 => (3, 0),
+        Int16 | UInt16 => (5, 0),
+        Int32 | UInt32 => (10, 0),
+        Int64 | UInt64 => (20, 0),
+        Float16 | Float32 => (14, 7),
+        Float64 => (30, 15),
+        _ => return None,
+    };
+    Some(Decimal128(precision, scale))
+}
+
+/// The binary layout that matches a string layout, so the cast keeps the 
buffers.
+fn binary_for_string(data_type: &DataType) -> DataType {
+    match data_type {
+        DataType::LargeUtf8 => DataType::LargeBinary,
+        DataType::Utf8View => DataType::BinaryView,
+        _ => DataType::Binary,
+    }
+}
+
+fn list_element(data_type: &DataType) -> Option<&DataType> {
+    match data_type {
+        DataType::List(field)
+        | DataType::LargeList(field)
+        | DataType::FixedSizeList(field, _) => Some(field.data_type()),
+        _ => None,
+    }
+}
+
+/// Spark's `FractionalType`: FLOAT, DOUBLE and DECIMAL.
+fn is_fractional(data_type: &DataType) -> bool {
+    data_type.is_floating() || data_type.is_decimal()
+}
+
+/// Spark's `DatetimeType`: DATE, TIMESTAMP, TIMESTAMP_NTZ and TIME.
+fn is_datetime(data_type: &DataType) -> bool {
+    matches!(
+        data_type,
+        DataType::Date32
+            | DataType::Date64
+            | DataType::Timestamp(_, _)
+            | DataType::Time32(_)
+            | DataType::Time64(_)
+    )
+}
+
+/// Spark's `CalendarIntervalType`, which is not an `AtomicType`.
+fn is_calendar_interval(data_type: &DataType) -> bool {
+    matches!(data_type, DataType::Interval(IntervalUnit::MonthDayNano))
+}
+
+/// Spark's `AtomicType`: every type except NULL, arrays, maps, structs and
+/// `CalendarIntervalType`. The year-month and day-time interval types are 
atomic.
+fn is_atomic(data_type: &DataType) -> bool {
+    matches!(
+        data_type,
+        DataType::Boolean
+            | DataType::Interval(IntervalUnit::YearMonth | 
IntervalUnit::DayTime)
+            | DataType::Duration(_)
+    ) || data_type.is_numeric()
+        || data_type.is_string()
+        || data_type.is_binary()
+        || is_datetime(data_type)
+}
+
+/// Spark's `ordinalNumber`, for "The first parameter ...". Spark spells out 
only the first
+/// three.
+fn ordinal_number(idx: usize) -> String {
+    match idx {
+        0 => "first".to_string(),
+        1 => "second".to_string(),
+        2 => "third".to_string(),
+        _ => format!("{}th", idx + 1),
+    }
+}
+
+/// The name Spark's `toSQLType` gives a type, for error messages.
+///
+/// The same function is in `type_coercion.rs` in #345; whichever lands second 
should use the
+/// other's copy.
+fn spark_sql_type(data_type: &DataType) -> String {
+    use DataType::*;
+    match data_type {
+        Null => "VOID".to_string(),
+        Boolean => "BOOLEAN".to_string(),
+        Int8 => "TINYINT".to_string(),
+        Int16 => "SMALLINT".to_string(),
+        Int32 => "INT".to_string(),
+        Int64 => "BIGINT".to_string(),
+        Float32 => "FLOAT".to_string(),
+        Float64 => "DOUBLE".to_string(),
+        Decimal32(p, s) | Decimal64(p, s) | Decimal128(p, s) | Decimal256(p, 
s) => {
+            format!("DECIMAL({p},{s})")
+        }
+        Utf8 | LargeUtf8 | Utf8View => "STRING".to_string(),
+        Binary | LargeBinary | BinaryView | FixedSizeBinary(_) => 
"BINARY".to_string(),
+        Date32 | Date64 => "DATE".to_string(),
+        Timestamp(_, Some(_)) => "TIMESTAMP".to_string(),
+        Timestamp(_, None) => "TIMESTAMP_NTZ".to_string(),
+        Interval(IntervalUnit::MonthDayNano) => "INTERVAL".to_string(),
+        List(field) | LargeList(field) | FixedSizeList(field, _) => {
+            format!("ARRAY<{}>", spark_sql_type(field.data_type()))
+        }
+        Struct(fields) => format!(
+            "STRUCT<{}>",
+            fields
+                .iter()
+                .map(|f| format!(
+                    "{}: {}{}",
+                    f.name(),

Review Comment:
   This copy of `spark_sql_type` doesn't have the names that #345 added in 
73b600c for [my comment 
there](https://github.com/apache/datafusion-spark/pull/345#discussion_r4605679264):
 `INTERVAL YEAR TO MONTH`, `INTERVAL DAY TO SECOND`, `TIME(6)`, and quoted 
struct field names ([#345's 
version](https://github.com/apache/datafusion-spark/blob/73b600c3b79fbbb6c14bf0ba8e4fa34b1a2ca909/crates/spark/src/function/type_coercion.rs#L219-L282)).
 So the two copies print different names for the same type, and `sha2`'s errors 
use the Arrow names:
   
   | Query | This PR | Spark 4.2.0 |
   |---|---|---|
   | `sha2(make_dt_interval(1), 256)` | Arrow's `Duration` name | `"INTERVAL 
DAY TO SECOND"` |
   | `sha2(arrow_cast(1, 'Interval(YearMonth)'), 256)` | 
`"Interval(YearMonth)"` | `"INTERVAL YEAR TO MONTH"` |
   | `sha2(make_time(1, 2, 3), 256)` | `"Time64(ns)"` | `"TIME(6)"` (with the 
TIME type enabled) |
   | `sha2(named_struct('a b', CAST(NULL AS INT)), 256)` | `"STRUCT<a b: INT>"` 
| ``"STRUCT<`a b`: INT>"`` |
   
   The Spark column for the year-month row is from `make_ym_interval(1, 0)`, 
since the crate has no `make_ym_interval`. Could this copy match #345's, 
including `quote_if_needed`? The copies also differ the other way: this one 
names `Decimal256`, `FixedSizeBinary`, and `Date64`, and #345's prints their 
Arrow names. An `.slt` error record for `sha2(make_dt_interval(1), 256)` would 
cover it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to