This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-12899-c8e7896157c6e003a5d7f7f5038de043a4de2ac5 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit daa8197bdd415dcd65d01eb2f65521d39b20fc25 Author: Sepuri Sai Krishna <[email protected]> AuthorDate: Wed Oct 28 12:59:00 2026 +0000 [Fix][Transform-V2] Fail CAST to BIGINT on out-of-range numeric input (#12899) --- .../introduction/concepts/incompatible-changes.md | 42 ++++- docs/en/transforms/sql-functions.md | 13 +- .../introduction/concepts/incompatible-changes.md | 34 +++- docs/zh/transforms/sql-functions.md | 12 +- .../sql/zeta/functions/SystemFunction.java | 118 +++++++++++--- .../transform/sql/zeta/ZetaSQLEngineTest.java | 159 ++++++++++++++++++ .../sql/zeta/functions/SystemFunctionTest.java | 177 +++++++++++++++++++-- 7 files changed, 515 insertions(+), 40 deletions(-) diff --git a/docs/en/introduction/concepts/incompatible-changes.md b/docs/en/introduction/concepts/incompatible-changes.md index a14cd53669..0433a494db 100644 --- a/docs/en/introduction/concepts/incompatible-changes.md +++ b/docs/en/introduction/concepts/incompatible-changes.md @@ -155,12 +155,50 @@ The Zeta SQL ARRAY function now declares TINYINT elements as `ARRAY<TINYINT>`, m `CAST` to an integral target is rejected while the statement is prepared, both before and after this change. Note the truncating path still differs from a string source, where `Integer.parseInt("5.7")` fails; aligning those is a separate decision. - - **Unchanged targets**: `BIGINT` | `LONG` keeps its existing conversion and is not range-checked - by this change; that gap is tracked separately in #12612. + - **`BIGINT` is covered by a separate change**: it kept its existing conversion when this entry + was written; the range check for it is described below and was tracked in #12612. - **Migration Guide**: Use `TRY_CAST` to get `NULL` instead of an error for values the target cannot hold, or widen the target type so the value fits. To keep a truncating conversion, compute it explicitly rather than relying on `CAST`. +### Zeta SQL Transform: CAST to BIGINT rejects out-of-range numeric values + +- **Behavior change: converting a numeric value to `BIGINT` now fails instead of silently wrapping** + - **Affected component**: `seatunnel-transforms-v2`, `SystemFunction.castAs` (Zeta SQL transform) + - **Description**: The `BIGINT` | `LONG` numeric branch converted with `Number.longValue()`, which + keeps only the low-order 64 bits of a `DECIMAL` or a `BigInteger`, maps `NaN` to `0`, and + saturates an out-of-range `DOUBLE` at `Long.MAX_VALUE`. Measured on `dev`: `18446744073709551621` + (2^64+5) arrived as `5`, `18446744073709551616` (2^64) as `0`, `NaN` as `0`, and `1e30` as + `9223372036854775807`. None of it was reported, so the row was written with a wrong number. The + conversion is now range-checked against `Long.MIN_VALUE`..`Long.MAX_VALUE`, reusing the same + per-family check added for `INT`. + - **Impact**: A job that relied on the wraparound, deliberately or not, now fails at the row that + overflows rather than writing a wrong number. In-range values, the exact integral wrappers, + widening casts and identity casts are unaffected. + - **The reachable route is `COALESCE` and `IFNULL`, not `CAST`**: an explicit `CAST(... AS BIGINT)` + could never produce this, because the planner admits only `TINYINT`, `SMALLINT`, `INT`, `BIGINT` + and `STRING` as sources for a `BIGINT` target; the four integral families widen to a long + exactly, and a `STRING` already reported its own overflow through `Long.parseLong`. A `DECIMAL` + or `DOUBLE` reaches the target through `COALESCE` and `IFNULL`, whose result type is inferred + from the first non-null argument rather than the widest, so `COALESCE(bigint_col, decimal_col)` + targets `BIGINT` and narrowed the decimal silently. + - **`TRY_CAST` cannot turn these into `NULL`**: for the same reason. `TRY_CAST(decimal_col AS + BIGINT)` and `TRY_CAST(double_col AS BIGINT)` are rejected while the statement is prepared, with + `Unsupported CAST FROM ... AS type: BIGINT`, so the per-row path where `TRY_CAST` catches a + failure is never reached. `TRY_CAST` does return `NULL` for an overflowing `STRING` source, which + is the one route that reaches the conversion as a cast. + - **Fractional sources are unchanged**: a value such as `5.7` reaching `BIGINT` through `COALESCE` + or `IFNULL` is still truncated towards zero rather than rejected. Only the range behaviour + changes. + - **Migration Guide**: Use `TRY_CAST` to get `NULL` instead of an error where the source is a + string, widen the target type so the value fits, or compute a truncating conversion explicitly + rather than relying on the inferred narrowing in `COALESCE` and `IFNULL`. To keep the widest + type rather than the type of the first argument, write the cast yourself: + `COALESCE(CAST(int_col AS BIGINT), bigint_col)`. + That works for an integral source only: a `DECIMAL` or `DOUBLE` has no cast to `BIGINT`, so + put it first instead, as in `COALESCE(decimal_col, bigint_col)`. Reordering is not a drop-in + change though: it also changes which argument wins when both are non-null. + ### DuckDB BIT and ENUM automatic DDL - Scalar `BIT` and `ENUM` columns with no catalog length now retain an unspecified STRING length diff --git a/docs/en/transforms/sql-functions.md b/docs/en/transforms/sql-functions.md index 025ea939cd..09e1d85b02 100644 --- a/docs/en/transforms/sql-functions.md +++ b/docs/en/transforms/sql-functions.md @@ -1255,6 +1255,13 @@ fractional value reaches an integral target only through `COALESCE` or `IFNULL`, type is inferred rather than written, and there it is truncated towards zero: when `int_col` is `NULL`, `COALESCE(int_col, double_col)` on `5.7` gives `5`. +`BIGINT` | `LONG` is range-checked the same way, but only an overflowing string source reaches it as +a cast, such as `CAST('9223372036854775808' AS BIGINT)`, where `TRY_CAST` returns `NULL`. A `DECIMAL` +or `DOUBLE` source is refused while the statement is prepared, so it reaches `BIGINT` only through +`COALESCE` or `IFNULL`: when `bigint_col` is `NULL`, `COALESCE(bigint_col, decimal_col)` on +`18446744073709551616` now fails instead of returning `0`. `TRY_CAST` cannot turn that into `NULL`, +because no cast is involved. + ### TRY_CAST ```TRY_CAST(value as dataType) -> dataType | NULL``` @@ -1273,6 +1280,8 @@ TRY_CAST(NAME AS INT) Returns the first value that is not null. If subsequent arguments have different data types from the first argument, they will be automatically converted to the type of the first argument. +Because the target type comes from the first argument rather than the widest, a later argument is narrowed to it, so an out-of-range value now fails instead of wrapping. To keep the widest type, write the cast yourself: `COALESCE(CAST(int_col AS BIGINT), bigint_col)`. A `DECIMAL` or `DOUBLE` argument cannot be cast to `BIGINT` at all, so for those put the wider argument first instead, as in `COALESCE(decimal_col, bigint_col)`. Reordering is not a drop-in change though: it also changes wh [...] + Example: COALESCE(A, B, C) @@ -1287,10 +1296,12 @@ SELECT COALESCE(A, B) as result FROM my_table ### IFNULL -```IFNULL(aValue, bValue) -> type(common of args)``` +```IFNULL(aValue, bValue) -> type(of first non-null arg)``` Returns the first value that is not null. If subsequent arguments have different data types from the first argument, they will be automatically converted to the type of the first argument. +Because the target type comes from the first argument rather than the widest, a later argument is narrowed to it, so an out-of-range value now fails instead of wrapping. To keep the widest type, write the cast yourself: `IFNULL(CAST(int_col AS BIGINT), bigint_col)`. A `DECIMAL` or `DOUBLE` argument cannot be cast to `BIGINT` at all, so for those put the wider argument first instead, as in `IFNULL(decimal_col, bigint_col)`. Reordering is not a drop-in change though: it also changes which [...] + Example: IFNULL(A, B) diff --git a/docs/zh/introduction/concepts/incompatible-changes.md b/docs/zh/introduction/concepts/incompatible-changes.md index 021601be5f..4ee22b301f 100644 --- a/docs/zh/introduction/concepts/incompatible-changes.md +++ b/docs/zh/introduction/concepts/incompatible-changes.md @@ -142,10 +142,42 @@ Zeta SQL ARRAY 函数现在将 TINYINT 元素声明为 `ARRAY<TINYINT>`，与实 这是该转换一直以来的行为，本次只改变范围相关的行为。而显式写成 `CAST` 到整型目标的浮点来源，在语句准备 阶段就会被拒绝，本次变更前后都是如此。注意截断这条路径与字符串来源仍然不同，`Integer.parseInt("5.7")` 会失败；是否统一两者是另一个决定。 - - **未改变的目标类型**：`BIGINT` | `LONG` 保持原有转换方式，本次变更不对其做范围检查；该问题单独记录在 #12612。 + - **`BIGINT` 由另一处变更覆盖**：撰写本条目时它仍保持原有转换方式；针对它的范围检查见下方条目，原记录于 #12612。 - **迁移指南**：若希望在目标类型无法容纳该值时得到 `NULL` 而不是报错，请使用 `TRY_CAST`；或将目标类型 扩宽到足以容纳该值。如果确实需要截断语义，请显式计算，而不要依赖 `CAST`。 +### Zeta SQL Transform：数值来源转换为 BIGINT 时拒绝超出范围的值 + +- **行为变更：数值来源转换为 `BIGINT` 时不再静默回绕，而是直接失败** + - **受影响组件**：`seatunnel-transforms-v2`，`SystemFunction.castAs`（Zeta SQL transform） + - **说明**：`BIGINT` | `LONG` 的数值分支使用 `Number.longValue()` 转换，它只保留 `DECIMAL` 或 + `BigInteger` 的低 64 位，把 `NaN` 映射为 `0`，并把超出范围的 `DOUBLE` 饱和到 `Long.MAX_VALUE`。 + 在 `dev` 上实测：`18446744073709551621`（2^64+5）得到 `5`，`18446744073709551616`（2^64）得到 + `0`，`NaN` 得到 `0`，`1e30` 得到 `9223372036854775807`。这些都不会被报告，因此该行会写入错误的 + 数字。现在该转换会按 `Long.MIN_VALUE`..`Long.MAX_VALUE` 做范围检查，并复用为 `INT` 添加的同一套 + 按数值族检查的逻辑。 + - **影响**：如果作业（有意或无意地）依赖了回绕行为，现在会在溢出的那一行失败，而不是写入错误的数字。 + 范围内的值、精确的整型包装类型、扩宽转换和同类型转换均不受影响。 + - **可达路径是 `COALESCE` 与 `IFNULL`，而不是 `CAST`**：显式的 `CAST(... AS BIGINT)` 不可能产生该问题， + 因为规划阶段对 `BIGINT` 目标只接受 `TINYINT`、`SMALLINT`、`INT`、`BIGINT` 与 `STRING` 作为来源； + 四种整型族都能无损加宽为 long，而 `STRING` 本来就会通过 `Long.parseLong` 报告自身的溢出。 + `DECIMAL` 或 `DOUBLE` 是通过 `COALESCE` 与 `IFNULL` 到达该目标的，这两者的结果类型按第一个非空参数 + 推断，而不是按最宽的类型推断，因此 `COALESCE(bigint_col, decimal_col)` 的目标类型是 `BIGINT`， + 并会静默收窄其中的 decimal。 + - **`TRY_CAST` 无法把这些情况转成 `NULL`**：原因同上。`TRY_CAST(decimal_col AS BIGINT)` 与 + `TRY_CAST(double_col AS BIGINT)` 在语句准备阶段就会以 + `Unsupported CAST FROM ... AS type: BIGINT` 被拒绝，因此永远不会走到 `TRY_CAST` 捕获失败的逐行路径。 + 对于超出范围的 `STRING` 来源，`TRY_CAST` 仍会返回 `NULL`，这是唯一以 cast 形式到达该转换的路径。 + - **小数来源的行为不变**：通过 `COALESCE` 或 `IFNULL` 到达 `BIGINT` 的 `5.7` 这类值仍然向零截断， + 而不是被拒绝。本次只改变范围相关的行为。 + - **迁移指南**：来源为字符串时，若希望得到 `NULL` 而不是报错，请使用 `TRY_CAST`；或将目标类型扩宽到 + 足以容纳该值；如果确实需要截断语义，请显式计算，而不要依赖 `COALESCE` 与 `IFNULL` 推断出的收窄。 + 如果希望结果使用最宽的类型，而不是第一个参数的类型，请自己写出转换： + `COALESCE(CAST(int_col AS BIGINT), bigint_col)`。 + 该写法仅适用于整型来源：`DECIMAL` 或 `DOUBLE` 没有到 `BIGINT` 的转换，请改为把它放在第一个参数， + 例如 `COALESCE(decimal_col, bigint_col)`。但重排并非等价替换：当两个参数都非空时， + 返回哪一个也会随之改变。 + ### DuckDB BIT 和 ENUM 自动建表 - Catalog 未提供长度时，标量 `BIT` 和 `ENUM` 列现在保留未指定的 STRING 长度，不再使用原来的 1/255 回退值。 diff --git a/docs/zh/transforms/sql-functions.md b/docs/zh/transforms/sql-functions.md index 262564dabb..460841af23 100644 --- a/docs/zh/transforms/sql-functions.md +++ b/docs/zh/transforms/sql-functions.md @@ -1253,6 +1253,12 @@ CAST(FLAG AS BOOLEAN) 小数值只能通过 COALESCE 或 IFNULL 到达整型目标（这两者的目标类型是推断出来的，而不是显式写出的）， 此时会向零截断：当 `int_col` 为 NULL 时，`COALESCE(int_col, double_col)` 对 `5.7` 得到 `5`。 +BIGINT | LONG 也会以同样方式做范围检查，但只有超出范围的字符串来源会以 cast 的形式到达，例如 +`CAST('9223372036854775808' AS BIGINT)`，此时 TRY_CAST 返回 NULL。DECIMAL 与 DOUBLE 来源在语句准备 +阶段就会被拒绝，因此它们只能通过 COALESCE 或 IFNULL 到达 BIGINT：当 `bigint_col` 为 NULL 时， +`COALESCE(bigint_col, decimal_col)` 对 `18446744073709551616` 现在会失败，而不再返回 `0`。 +由于其中不涉及 cast，TRY_CAST 无法把这种情况转成 NULL。 + ### TRY_CAST ```TRY_CAST(value as dataType) -> dataType | NULL``` @@ -1271,6 +1277,8 @@ TRY_CAST(NAME AS INT) 返回第一个非空值。如果后续参数与第一个参数的数据类型不同，则会自动转换为第一个参数的类型。 +由于目标类型取自第一个参数而不是最宽的类型，后续参数会被收窄到该类型，因此超出范围的值现在会直接失败，而不再回绕。如果希望结果使用最宽的类型，请自己写出转换：`COALESCE(CAST(int_col AS BIGINT), bigint_col)`。`DECIMAL` 或 `DOUBLE` 参数根本无法转换为 `BIGINT`，因此这类情况请改为把更宽的参数放在前面，例如 `COALESCE(decimal_col, bigint_col)`。但重排并非等价替换：当两个参数都非空时，返回哪一个也会随之改变。 + 示例: COALESCE(A, B, C) @@ -1285,10 +1293,12 @@ SELECT COALESCE(A, B) as result FROM my_table ### IFNULL -```IFNULL(aValue, bValue) -> type(common of args)``` +```IFNULL(aValue, bValue) -> type(of first non-null arg)``` 返回第一个非空值。如果后续参数与第一个参数的数据类型不同，则会自动转换为第一个参数的类型。 +由于目标类型取自第一个参数而不是最宽的类型，后续参数会被收窄到该类型，因此超出范围的值现在会直接失败，而不再回绕。如果希望结果使用最宽的类型，请自己写出转换：`IFNULL(CAST(int_col AS BIGINT), bigint_col)`。`DECIMAL` 或 `DOUBLE` 参数根本无法转换为 `BIGINT`，因此这类情况请改为把更宽的参数放在前面，例如 `IFNULL(decimal_col, bigint_col)`。但重排并非等价替换：当两个参数都非空时，返回哪一个也会随之改变。 + 示例: IFNULL(A, B) diff --git a/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/functions/SystemFunction.java b/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/functions/SystemFunction.java index 0219e03bf4..52e73cc76b 100644 --- a/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/functions/SystemFunction.java +++ b/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/functions/SystemFunction.java @@ -107,8 +107,33 @@ public class SystemFunction { return castAs(args); } - private static final BigInteger INT_MIN_BIG = BigInteger.valueOf(Integer.MIN_VALUE); - private static final BigInteger INT_MAX_BIG = BigInteger.valueOf(Integer.MAX_VALUE); + /** + * The bounds of an integral SQL target, carried in both forms the range check needs: as + * primitives for the common truncation path, and as {@link BigInteger} for a {@code DECIMAL} or + * {@link BigInteger} source. Enum constants so neither form is allocated per row. + */ + private enum IntegralBounds { + INT(Integer.MIN_VALUE, Integer.MAX_VALUE), + LONG(Long.MIN_VALUE, Long.MAX_VALUE); + + /** + * 2^63 as a double. Long.MAX_VALUE is 2^63-1, which no double can represent, so this is the + * smallest double magnitude that a narrowing cast to long saturates on. + */ + private static final double SATURATION_MAGNITUDE = 9.223372036854775808E18; + + private final long min; + private final long max; + private final BigInteger minBig; + private final BigInteger maxBig; + + IntegralBounds(long min, long max) { + this.min = min; + this.max = max; + this.minBig = BigInteger.valueOf(min); + this.maxBig = BigInteger.valueOf(max); + } + } /** * Narrows a numeric value to {@code int}, throwing when it cannot be represented. See <a @@ -128,40 +153,84 @@ public class SystemFunction { * @throws TransformException if the value cannot be represented as an int */ private static int numberToInt(Number value, String targetType) { + return (int) narrowToIntegral(value, targetType, IntegralBounds.INT); + } + + /** + * Narrows a numeric value to {@code long}, throwing when it cannot be represented. See <a + * href="https://github.com/apache/seatunnel/issues/12612";>#12612</a>. + * + * <p>The same shape as {@link #numberToInt}, against the long bounds and through the same + * helper rather than a long-specific copy of it. {@link Number#longValue()} keeps only the + * low-order 64 bits of a wide {@link BigDecimal} or {@link BigInteger}, maps {@code NaN} to + * zero, and saturates an out-of-range {@code double} at {@link Long#MAX_VALUE}, so each of + * those arrived as a wrong number rather than as a failure. + * + * <p>No explicit {@code CAST} reaches this with a value that cannot be represented: {@code + * CastFunction.LONG_CAST_TYPES} admits only TINYINT, SMALLINT, INT, BIGINT and STRING, the four + * integral families widen to long exactly, and a STRING goes through {@link Long#parseLong}, + * which reports its own overflow. The reachable route is {@code COALESCE} and {@code IFNULL}, + * whose result type is inferred from the first non-null argument, so a {@code DECIMAL} or + * {@code DOUBLE} argument is narrowed here against a BIGINT target. + * + * @param value the numeric value being converted, never null at the call site + * @param targetType the SQL type name, used in the error message + * @return the value as a long + * @throws TransformException if the value cannot be represented as a long + */ + private static long numberToLong(Number value, String targetType) { + return narrowToIntegral(value, targetType, IntegralBounds.LONG); + } + + /** + * Truncates a numeric value towards zero and range-checks the result against the target's + * bounds, reporting the original value in any failure. + */ + private static long narrowToIntegral(Number value, String targetType, IntegralBounds bounds) { if (value instanceof BigDecimal) { - return intFromExact(((BigDecimal) value).toBigInteger(), value, targetType); + return fromExact(((BigDecimal) value).toBigInteger(), value, targetType, bounds); } if (value instanceof BigInteger) { - return intFromExact((BigInteger) value, value, targetType); + return fromExact((BigInteger) value, value, targetType, bounds); } if (value instanceof Double || value instanceof Float) { double d = value.doubleValue(); if (Double.isNaN(d) || Double.isInfinite(d)) { - throw outOfIntRange(value, targetType); + throw outOfRange(value, targetType, bounds); } - // Truncate before range-checking, not after: narrowing a double to long rounds - // towards zero, so a fractional value whose truncation fits, such as 2147483647.5, - // must still convert. Comparing the raw double against the bounds would reject it. - long truncated = (long) d; - if (truncated < Integer.MIN_VALUE || truncated > Integer.MAX_VALUE) { - throw outOfIntRange(value, targetType); + // A narrowing cast to long saturates rather than overflowing, so a magnitude at or + // beyond 2^63 would arrive as Long.MAX_VALUE and pass a long range check: that is how + // 1e30 previously converted to Long.MAX_VALUE. Rejected here, before the cast. + // -2^63 is exactly Long.MIN_VALUE, so it is representable and is not rejected. + if (d >= IntegralBounds.SATURATION_MAGNITUDE + || d < -IntegralBounds.SATURATION_MAGNITUDE) { + throw outOfRange(value, targetType, bounds); } - return (int) truncated; + // Truncate before range-checking, not after: narrowing rounds towards zero, so a + // fractional value whose truncation fits, such as 2147483647.5 for INT, must still + // convert. Comparing the raw double against the bounds would reject it. + return checkRange((long) d, value, targetType, bounds); } // Byte, Short, Integer and Long widen to long exactly. - long widened = value.longValue(); - if (widened < Integer.MIN_VALUE || widened > Integer.MAX_VALUE) { - throw outOfIntRange(value, targetType); - } - return (int) widened; + return checkRange(value.longValue(), value, targetType, bounds); } /** Range-checks an already-integral value, reporting the original in any failure. */ - private static int intFromExact(BigInteger truncated, Number original, String targetType) { - if (truncated.compareTo(INT_MIN_BIG) < 0 || truncated.compareTo(INT_MAX_BIG) > 0) { - throw outOfIntRange(original, targetType); + private static long fromExact( + BigInteger truncated, Number original, String targetType, IntegralBounds bounds) { + if (truncated.compareTo(bounds.minBig) < 0 || truncated.compareTo(bounds.maxBig) > 0) { + throw outOfRange(original, targetType, bounds); + } + return truncated.longValue(); + } + + /** Range-checks a value already truncated to a long, reporting the original in any failure. */ + private static long checkRange( + long truncated, Number original, String targetType, IntegralBounds bounds) { + if (truncated < bounds.min || truncated > bounds.max) { + throw outOfRange(original, targetType, bounds); } - return truncated.intValue(); + return truncated; } /** @@ -172,12 +241,13 @@ public class SystemFunction { * target type was inferred. The surrounding expression is supplied by the engine's error * wrapper. */ - private static TransformException outOfIntRange(Number value, String targetType) { + private static TransformException outOfRange( + Number value, String targetType, IntegralBounds bounds) { return new TransformException( CommonErrorCodeDeprecated.UNSUPPORTED_OPERATION, String.format( "Value %s cannot be converted to %s: out of range [%d, %d]", - value, targetType, Integer.MIN_VALUE, Integer.MAX_VALUE)); + value, targetType, bounds.min, bounds.max)); } public static Object castAs(List<Object> args) { @@ -212,7 +282,7 @@ public class SystemFunction { } else if (v1 instanceof OffsetDateTime) { return ((OffsetDateTime) v1).toInstant().toEpochMilli(); } else if (v1 instanceof Number) { - return ((Number) v1).longValue(); + return numberToLong((Number) v1, v2); } else { throw new TransformException( CommonErrorCodeDeprecated.UNSUPPORTED_OPERATION, diff --git a/seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLEngineTest.java b/seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLEngineTest.java index 5252d1363a..99daeeb74f 100644 --- a/seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLEngineTest.java +++ b/seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLEngineTest.java @@ -678,4 +678,163 @@ public class ZetaSQLEngineTest { Locale.setDefault(original); } } + + @Test + public void testCoalesceRejectsWideAndNonFiniteSourcesForABigintTarget() { + // The reachable route for #12612. COALESCE and IFNULL infer their type from the first + // non-null argument, so COALESCE(bigint, decimal) targets BIGINT and the decimal is + // narrowed here. These are the four values measured on dev, which arrived as 5, 0, 0 and + // Long.MAX_VALUE respectively instead of failing. + SeaTunnelRowType decimalRowType = + new SeaTunnelRowType( + new String[] {"c_bigint", "c_decimal"}, + new SeaTunnelDataType[] {BasicType.LONG_TYPE, new DecimalType(38, 0)}); + SeaTunnelRowType doubleRowType = + new SeaTunnelRowType( + new String[] {"c_bigint", "c_double"}, + new SeaTunnelDataType[] {BasicType.LONG_TYPE, BasicType.DOUBLE_TYPE}); + + // Proves the test is about the magnitude and not about the conversion failing generally: + // longValue() silently produces these, which is the defect being fixed. + Assertions.assertEquals( + 5L, + new BigDecimal("18446744073709551621").longValue(), + "2^64+5 must wrap to 5 through longValue(), otherwise this test proves nothing"); + Assertions.assertEquals( + 0L, + new BigDecimal("18446744073709551616").longValue(), + "2^64 must wrap to 0 through longValue(), otherwise this test proves nothing"); + Assertions.assertEquals( + 0L, + Double.valueOf(Double.NaN).longValue(), + "NaN must map to 0 through longValue(), as the docs note for this change states"); + Assertions.assertEquals( + Long.MAX_VALUE, + Double.valueOf(1.0e30d).longValue(), + "1e30 must saturate at Long.MAX_VALUE through longValue(), per the docs note"); + + for (String fn : new String[] {"coalesce", "ifnull"}) { + for (Object v : + new Object[] { + new BigDecimal("18446744073709551621"), + new BigDecimal("18446744073709551616"), + new BigDecimal("-18446744073709551621") + }) { + ZetaSQLEngine engine = new ZetaSQLEngine(); + String sql = "select " + fn + "(c_bigint, c_decimal) as r from test"; + engine.init("test", "test", decimalRowType, sql); + SeaTunnelRowType outType = engine.typeMapping(new ArrayList<>()); + Assertions.assertThrows( + TransformException.class, + () -> + engine.transformBySQL( + new SeaTunnelRow(new Object[] {null, v}), outType), + sql + " should reject " + v); + } + + for (Object v : new Object[] {Double.NaN, 1.0e30d, -1.0e30d}) { + ZetaSQLEngine engine = new ZetaSQLEngine(); + String sql = "select " + fn + "(c_bigint, c_double) as r from test"; + engine.init("test", "test", doubleRowType, sql); + SeaTunnelRowType outType = engine.typeMapping(new ArrayList<>()); + Assertions.assertThrows( + TransformException.class, + () -> + engine.transformBySQL( + new SeaTunnelRow(new Object[] {null, v}), outType), + sql + " should reject " + v); + } + } + + // Controls: an in-range decimal and an in-range double still convert, and a fractional + // double is still truncated towards zero rather than rejected. + ZetaSQLEngine decimalEngine = new ZetaSQLEngine(); + String decimalSql = "select coalesce(c_bigint, c_decimal) as r from test"; + decimalEngine.init("test", "test", decimalRowType, decimalSql); + SeaTunnelRowType decimalOut = decimalEngine.typeMapping(new ArrayList<>()); + Assertions.assertEquals( + 7L, + decimalEngine + .transformBySQL( + new SeaTunnelRow(new Object[] {null, new BigDecimal("7")}), + decimalOut) + .get(0) + .getField(0), + "an in-range decimal must still convert to BIGINT"); + + ZetaSQLEngine doubleEngine = new ZetaSQLEngine(); + String doubleSql = "select coalesce(c_bigint, c_double) as r from test"; + doubleEngine.init("test", "test", doubleRowType, doubleSql); + SeaTunnelRowType doubleOut = doubleEngine.typeMapping(new ArrayList<>()); + Assertions.assertEquals( + 5L, + doubleEngine + .transformBySQL(new SeaTunnelRow(new Object[] {null, 5.7d}), doubleOut) + .get(0) + .getField(0), + "a fractional double must still truncate towards zero for BIGINT"); + } + + @Test + public void testStringSourceBoundariesForABigintTargetIncludingTryCast() { + // BIGINT was the one integral target missing from the string-source matrix above. It is + // also the only route by which TRY_CAST can return NULL for a BIGINT target, because + // Long.parseLong reports its own overflow per row; see the next test for why the numeric + // sources have no TRY_CAST form at all. + Assertions.assertEquals( + Long.MIN_VALUE, + castResult( + "select cast(c_str as BIGINT) as r from test", + rowWith(0L, String.valueOf(Long.MIN_VALUE))), + "BIGINT should accept the exact Long.MIN_VALUE string"); + Assertions.assertEquals( + Long.MAX_VALUE, + castResult( + "select cast(c_str as BIGINT) as r from test", + rowWith(0L, String.valueOf(Long.MAX_VALUE))), + "BIGINT should accept the exact Long.MAX_VALUE string"); + + // One step beyond each bound, written out because long arithmetic cannot express them. + for (String v : new String[] {"9223372036854775808", "-9223372036854775809"}) { + Assertions.assertThrows( + Exception.class, + () -> castResult("select cast(c_str as BIGINT) as r from test", rowWith(0L, v)), + "BIGINT should reject the string " + v); + Assertions.assertNull( + castResult("select try_cast(c_str as BIGINT) as r from test", rowWith(0L, v)), + "BIGINT TRY_CAST should be null for the string " + v); + } + } + + @Test + public void testBigintTargetCannotBeReachedByAnExplicitCastFromDecimalOrDouble() { + // Why the four numeric values in #12612 have no TRY_CAST form: CastFunction.LONG_CAST_TYPES + // admits only TINYINT, SMALLINT, INT, BIGINT and STRING, so a DECIMAL or DOUBLE source is + // refused while the statement is prepared, before any row exists. TRY_CAST's catch sits in + // the per-row path and is never reached, so it cannot turn these into NULL. Pinned so the + // PR's claim about TRY_CAST stays true if that type list is ever widened. + SeaTunnelRowType rowType = + new SeaTunnelRowType( + new String[] {"c_decimal", "c_double"}, + new SeaTunnelDataType[] {new DecimalType(38, 0), BasicType.DOUBLE_TYPE}); + + String[][] cases = { + {"cast(c_decimal as BIGINT)", "Unsupported CAST FROM DECIMAL AS type: BIGINT"}, + {"try_cast(c_decimal as BIGINT)", "Unsupported CAST FROM DECIMAL AS type: BIGINT"}, + {"cast(c_double as BIGINT)", "Unsupported CAST FROM DOUBLE AS type: BIGINT"}, + {"try_cast(c_double as BIGINT)", "Unsupported CAST FROM DOUBLE AS type: BIGINT"} + }; + for (String[] c : cases) { + ZetaSQLEngine engine = new ZetaSQLEngine(); + engine.init("test", "test", rowType, "select " + c[0] + " as r from test"); + TransformException thrown = + Assertions.assertThrows( + TransformException.class, + () -> engine.typeMapping(new ArrayList<>()), + c[0] + " should be rejected while the plan is prepared"); + Assertions.assertTrue( + thrown.getMessage().contains(c[1]), + c[0] + " should report \"" + c[1] + "\" but reported: " + thrown.getMessage()); + } + } } diff --git a/seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/sql/zeta/functions/SystemFunctionTest.java b/seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/sql/zeta/functions/SystemFunctionTest.java index 162437efd4..9361506300 100644 --- a/seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/sql/zeta/functions/SystemFunctionTest.java +++ b/seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/sql/zeta/functions/SystemFunctionTest.java @@ -25,6 +25,7 @@ import org.apache.seatunnel.api.table.type.SeaTunnelDataType; import org.apache.seatunnel.api.table.type.SeaTunnelRow; import org.apache.seatunnel.api.table.type.SeaTunnelRowType; import org.apache.seatunnel.api.table.type.SqlType; +import org.apache.seatunnel.transform.exception.TransformException; import org.apache.seatunnel.transform.sql.SQLTransform; import org.junit.jupiter.api.Assertions; @@ -375,7 +376,7 @@ public class SystemFunctionTest { Assertions.assertEquals("fallback", ifNullResult); Assertions.assertThrows( - org.apache.seatunnel.transform.exception.TransformException.class, + TransformException.class, () -> SystemFunction.ifnull( Collections.singletonList("onlyOneArg"), BasicType.STRING_TYPE)); @@ -549,16 +550,12 @@ public class SystemFunctionTest { args.clear(); args.add(2); args.add("BOOLEAN"); - Assertions.assertThrows( - org.apache.seatunnel.transform.exception.TransformException.class, - () -> SystemFunction.castAs(args)); + Assertions.assertThrows(TransformException.class, () -> SystemFunction.castAs(args)); args.clear(); args.add("notBool"); args.add("BOOLEAN"); - Assertions.assertThrows( - org.apache.seatunnel.transform.exception.TransformException.class, - () -> SystemFunction.castAs(args)); + Assertions.assertThrows(TransformException.class, () -> SystemFunction.castAs(args)); } @Test @@ -570,11 +567,11 @@ public class SystemFunctionTest { Integer.MIN_VALUE - 1L, Integer.MAX_VALUE + 1L, 3000000000L, Long.MAX_VALUE }) { Assertions.assertThrows( - org.apache.seatunnel.transform.exception.TransformException.class, + TransformException.class, () -> SystemFunction.castAs(Arrays.asList(out, "INT")), "INT should reject " + out); Assertions.assertThrows( - org.apache.seatunnel.transform.exception.TransformException.class, + TransformException.class, () -> SystemFunction.castAs(Arrays.asList(out, "INTEGER")), "INTEGER should reject " + out); } @@ -650,7 +647,7 @@ public class SystemFunctionTest { }; for (Object v : rejected) { Assertions.assertThrows( - org.apache.seatunnel.transform.exception.TransformException.class, + TransformException.class, () -> SystemFunction.castAs(Arrays.asList(v, "INT")), "INT should reject " + v); } @@ -704,9 +701,167 @@ public class SystemFunctionTest { new BigDecimal("-2147483649") }) { Assertions.assertThrows( - org.apache.seatunnel.transform.exception.TransformException.class, + TransformException.class, () -> SystemFunction.castAs(Arrays.asList(v, "INT")), "INT should reject " + v); } } + + @Test + public void testCastAsBigintRejectsWideAndNonFiniteNumericSources() { + // The four inputs measured on dev in #12612, each of which arrived as a wrong number + // rather than as a failure: a BigDecimal beyond 64 bits kept only its low-order bits + // (2^64+5 -> 5, 2^64 -> 0), NaN became 0, and 1e30 saturated at Long.MAX_VALUE. + Object[] rejected = { + new BigDecimal("18446744073709551621"), // 2^64+5, low 64 bits are 5 + new BigDecimal("18446744073709551616"), // 2^64, low 64 bits are 0 + new BigDecimal("-18446744073709551621"), + new BigInteger("18446744073709551621"), + new BigInteger("-18446744073709551621"), + Double.NaN, + Float.NaN, + Double.POSITIVE_INFINITY, + Double.NEGATIVE_INFINITY, + 1.0e30d, + -1.0e30d + }; + for (Object v : rejected) { + Assertions.assertThrows( + TransformException.class, + () -> SystemFunction.castAs(Arrays.asList(v, "BIGINT")), + "BIGINT should reject " + v); + Assertions.assertThrows( + TransformException.class, + () -> SystemFunction.castAs(Arrays.asList(v, "LONG")), + "LONG should reject " + v); + } + + // Control: an in-range BigDecimal still converts, so the fix cannot be mistaken for the + // conversion failing outright. Requested on the issue. + Assertions.assertEquals( + 5L, SystemFunction.castAs(Arrays.asList(new BigDecimal("5"), "BIGINT"))); + Assertions.assertEquals( + 5L, SystemFunction.castAs(Arrays.asList(new BigInteger("5"), "BIGINT"))); + Assertions.assertEquals( + -5L, SystemFunction.castAs(Arrays.asList(new BigDecimal("-5"), "BIGINT"))); + + // The exact integral wrappers are passed through unchanged, as agreed on the issue. + Assertions.assertEquals(5L, SystemFunction.castAs(Arrays.asList((byte) 5, "BIGINT"))); + Assertions.assertEquals(5L, SystemFunction.castAs(Arrays.asList((short) 5, "BIGINT"))); + Assertions.assertEquals(5L, SystemFunction.castAs(Arrays.asList(5, "BIGINT"))); + Assertions.assertEquals(5L, SystemFunction.castAs(Arrays.asList(5L, "BIGINT"))); + Assertions.assertEquals( + Long.MAX_VALUE, SystemFunction.castAs(Arrays.asList(Long.MAX_VALUE, "BIGINT"))); + Assertions.assertEquals( + Long.MIN_VALUE, SystemFunction.castAs(Arrays.asList(Long.MIN_VALUE, "BIGINT"))); + + // A fractional source is still truncated towards zero rather than rejected. Only the + // range behaviour changes, matching what #12605 did for INT. + Assertions.assertEquals(5L, SystemFunction.castAs(Arrays.asList(5.7d, "BIGINT"))); + Assertions.assertEquals(-5L, SystemFunction.castAs(Arrays.asList(-5.7d, "BIGINT"))); + Assertions.assertEquals(5L, SystemFunction.castAs(Arrays.asList(5.7f, "BIGINT"))); + Assertions.assertEquals( + 5L, SystemFunction.castAs(Arrays.asList(new BigDecimal("5.7"), "BIGINT"))); + } + + @Test + public void testCastAsBigintAcceptsTheExactBoundariesAndRejectsOneStepBeyond() { + // Requested on the issue: pin the bounds inclusively, not only at far-out magnitudes, so + // the comparison cannot be off by one at either end. + Assertions.assertEquals( + Long.MAX_VALUE, + SystemFunction.castAs( + Arrays.asList(new BigDecimal("9223372036854775807"), "BIGINT"))); + Assertions.assertEquals( + Long.MIN_VALUE, + SystemFunction.castAs( + Arrays.asList(new BigDecimal("-9223372036854775808"), "BIGINT"))); + Assertions.assertEquals( + Long.MAX_VALUE, + SystemFunction.castAs( + Arrays.asList(new BigInteger("9223372036854775807"), "BIGINT"))); + Assertions.assertEquals( + Long.MIN_VALUE, + SystemFunction.castAs( + Arrays.asList(new BigInteger("-9223372036854775808"), "BIGINT"))); + + // One step beyond each bound, which previously wrapped to the opposite sign. + for (Object v : + new Object[] { + new BigDecimal("9223372036854775808"), + new BigDecimal("-9223372036854775809"), + new BigInteger("9223372036854775808"), + new BigInteger("-9223372036854775809") + }) { + Assertions.assertThrows( + TransformException.class, + () -> SystemFunction.castAs(Arrays.asList(v, "BIGINT")), + "BIGINT should reject " + v); + } + + // A fractional value whose truncation is exactly the boundary still converts, so the + // range check runs on the truncated value rather than the raw one. + Assertions.assertEquals( + Long.MAX_VALUE, + SystemFunction.castAs( + Arrays.asList(new BigDecimal("9223372036854775807.5"), "BIGINT"))); + Assertions.assertEquals( + Long.MIN_VALUE, + SystemFunction.castAs( + Arrays.asList(new BigDecimal("-9223372036854775808.5"), "BIGINT"))); + } + + @Test + public void testCastAsBigintChecksTheDoubleItselfRatherThanItsNarrowedValue() { + // 2^63 is the asymmetric case, and the one that shows why the check cannot run after a + // narrowing cast. Long.MAX_VALUE is 2^63-1, which no double can represent, so the + // smallest double above the positive bound is 2^63 itself and must be rejected; (long) + // saturation would have turned it into Long.MAX_VALUE instead. Its negative, -2^63, is + // exactly Long.MIN_VALUE and must still convert. + Assertions.assertThrows( + TransformException.class, + () -> SystemFunction.castAs(Arrays.asList(9.223372036854776E18d, "BIGINT")), + "BIGINT should reject 2^63, which is one above Long.MAX_VALUE"); + Assertions.assertEquals( + Long.MIN_VALUE, + SystemFunction.castAs(Arrays.asList(-9.223372036854776E18d, "BIGINT")), + "-2^63 is exactly Long.MIN_VALUE and must convert"); + } + + @Test + public void testOutOfRangeMessageNamesTheTargetAndItsBounds() { + // The failure builder is shared between INT and BIGINT, so this pins both messages. The + // INT half guards against the shared refactor quietly changing a message that #12605 + // shipped; the BIGINT half pins that the bounds reported are the long ones and that the + // SQL type name the user wrote is echoed back, including the LONG alias. + Object[][] cases = { + { + new BigDecimal("18446744073709551621"), + "BIGINT", + "Value 18446744073709551621 cannot be converted to BIGINT: out of range [-9223372036854775808, 9223372036854775807]" + }, + { + 1.0e30d, + "LONG", + "Value 1.0E30 cannot be converted to LONG: out of range [-9223372036854775808, 9223372036854775807]" + }, + { + 3000000000L, + "INT", + "Value 3000000000 cannot be converted to INT: out of range [-2147483648, 2147483647]" + } + }; + for (Object[] c : cases) { + TransformException thrown = + Assertions.assertThrows( + TransformException.class, + () -> SystemFunction.castAs(Arrays.asList(c[0], c[1]))); + Assertions.assertTrue( + thrown.getMessage().contains((String) c[2]), + "expected the message to contain \"" + + c[2] + + "\" but it was: " + + thrown.getMessage()); + } + } }
