[
https://issues.apache.org/jira/browse/SPARK-58821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58821:
-----------------------------------
Labels: pull-request-available (was: )
> sequence() raises INTERNAL_ERROR for BIGINT endpoints whose difference
> overflows Long
> -------------------------------------------------------------------------------------
>
> Key: SPARK-58821
> URL: https://issues.apache.org/jira/browse/SPARK-58821
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: sepuri sai krishna
> Priority: Major
> Labels: pull-request-available
>
> {{sequence(start, stop, step)}} over BIGINT raises an {{INTERNAL_ERROR}}
> (SQLSTATE XX000) for argument combinations that have a small, well-defined
> result. Reachable from plain SQL with default configuration, on both the
> interpreted and the codegen path.
> {code:sql}
> SELECT sequence(-9223372036854775808L, 9223372036854775807L,
> 9223372036854775807L);
> -- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
> -- expected: [-9223372036854775808, -1, 9223372036854775806]
> SELECT sequence(-9223372036854775808L, 0L, 4611686018427387904L);
> -- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
> -- expected: [-9223372036854775808, -4611686018427387904, 0]
> {code}
> The same failure occurs when the arguments are not foldable, so it is not
> confined to constant folding:
> {code:sql}
> CREATE OR REPLACE TEMP VIEW t AS
> SELECT CAST(-9223372036854775808 AS BIGINT) AS a,
> CAST(9223372036854775807 AS BIGINT) AS b,
> CAST(9223372036854775807 AS BIGINT) AS c;
> SELECT sequence(a, b, c) FROM t;
> -- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
> {code}
> h3. Cause
> {{Sequence.sequenceLength}} computes the length in {{long}} and falls back to
> {{BigInt}} when that overflows:
> {code:scala}
> def sequenceLength(start: Long, stop: Long, step: Long): Int = {
> try {
> val delta = Math.subtractExact(stop, start) // (1) overflows
> ...
> val len = if (stop == start) 1L else Math.addExact(1L, (delta / step))
> if (len > ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH) {
> throw
> QueryExecutionErrors.createArrayWithElementsExceedLimitError(prettyName, len)
> }
> len.toInt
> } catch {
> case _: ArithmeticException =>
> val safeLen = BigInt(1) + (BigInt(stop) - BigInt(start)) / BigInt(step)
> // (2)
> if (safeLen > ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH) {
> throw
> QueryExecutionErrors.createArrayWithElementsExceedLimitError(prettyName,
> safeLen)
> }
> throw internalError("Unreachable code reached.")
> // (3)
> ...
> {code}
> The fallback assumes an overflow at (1) implies a result too large to
> allocate, so after recomputing the length exactly at (2) it only re-raises
> the array-limit error and otherwise treats (3) as dead code.
> That assumption does not hold. {{Math.subtractExact(stop, start)}} overflows
> whenever {{stop - start}} exceeds the {{long}} range, which says nothing
> about the length: the length also depends on {{step}}. When the step is
> large, the exact length computed at (2) is small, no limit error is raised,
> and control reaches the {{internalError}} at (3) -- with the correct answer
> sitting in {{safeLen}}, discarded.
> For the first example above, {{safeLen}} is exactly {{3}}.
> h3. Scope
> * BIGINT only. {{IntegralSequenceImpl}} reaches {{sequenceLength}} via
> {{getSequenceLength}}, which widens its arguments with {{toLong}}, so for
> INT/SMALLINT/TINYINT the subtraction cannot overflow. The DATE/TIMESTAMP
> callers pass microsecond values, whose range is far from the {{long}} limits.
> * Both execution paths. The interpreted path calls
> {{Sequence.sequenceLength}} directly and the generated code calls the same
> method, so they fail identically.
> * Not a recent regression. {{sequenceLength}} was introduced in its current
> shape by SPARK-43393 (commit {{afc4c49927cb}}, Nov 2023), which fixed a
> genuine correctness bug where the {{long}} length computation overflowed
> silently and {{sequence}} returned an empty array. That fix added the
> {{BigInt}} fallback and, reasonably at the time, treated the tail as
> unreachable -- the overflow cases it was written against all had huge
> lengths. The gap is that a large {{step}} makes the exact length small, so
> the tail is reachable after all. The method has not been modified since, so
> every release containing SPARK-43393 (3.3.4, 3.4.2, 3.5.1, 4.0.0 and later)
> is affected.
> h3. Suggested fix
> At (3), return {{safeLen.toInt}} instead of throwing. The value has already
> been proven to be at most {{ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}} by
> the check immediately above, so the narrowing conversion is safe, and the
> element-filling code already handles steps of this magnitude correctly --
> {{sequence(-4611686018427387904L, 4611686018427387903L,
> 4611686018427387903L)}}, whose endpoints are close enough together to avoid
> the overflow, returns {{[-4611686018427387904, -1, 4611686018427387902]}}
> today.
> Tests belong in {{CollectionExpressionsSuite}}, which exercises both the
> interpreted and the codegen path through {{checkEvaluation}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
