sepuri sai krishna created SPARK-58821:
------------------------------------------
Summary: sequence() raises INTERNAL_ERROR for BIGINT endpoints
whose difference overflows Long
Key: SPARK-58821
URL: https://issues.apache.org/jira/browse/SPARK-58821
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.3.0
Reporter: sepuri sai krishna
{{sequence(start, stop, step)}} over BIGINT raises an {{INTERNAL_ERROR}}
(SQLSTATE XX000) for argument combinations that have a small, well-defined
result. Reachable from plain SQL with default configuration, on both the
interpreted and the codegen path.
{code:sql}
SELECT sequence(-9223372036854775808L, 9223372036854775807L,
9223372036854775807L);
-- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
-- expected: [-9223372036854775808, -1, 9223372036854775806]
SELECT sequence(-9223372036854775808L, 0L, 4611686018427387904L);
-- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
-- expected: [-9223372036854775808, -4611686018427387904, 0]
{code}
The same failure occurs when the arguments are not foldable, so it is not
confined to constant folding:
{code:sql}
CREATE OR REPLACE TEMP VIEW t AS
SELECT CAST(-9223372036854775808 AS BIGINT) AS a,
CAST(9223372036854775807 AS BIGINT) AS b,
CAST(9223372036854775807 AS BIGINT) AS c;
SELECT sequence(a, b, c) FROM t;
-- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
{code}
h3. Cause
{{Sequence.sequenceLength}} computes the length in {{long}} and falls back to
{{BigInt}} when that overflows:
{code:scala}
def sequenceLength(start: Long, stop: Long, step: Long): Int = {
try {
val delta = Math.subtractExact(stop, start) // (1) overflows
...
val len = if (stop == start) 1L else Math.addExact(1L, (delta / step))
if (len > ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH) {
throw
QueryExecutionErrors.createArrayWithElementsExceedLimitError(prettyName, len)
}
len.toInt
} catch {
case _: ArithmeticException =>
val safeLen = BigInt(1) + (BigInt(stop) - BigInt(start)) / BigInt(step)
// (2)
if (safeLen > ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH) {
throw
QueryExecutionErrors.createArrayWithElementsExceedLimitError(prettyName,
safeLen)
}
throw internalError("Unreachable code reached.")
// (3)
...
{code}
The fallback assumes an overflow at (1) implies a result too large to allocate,
so after recomputing the length exactly at (2) it only re-raises the
array-limit error and otherwise treats (3) as dead code.
That assumption does not hold. {{Math.subtractExact(stop, start)}} overflows
whenever {{stop - start}} exceeds the {{long}} range, which says nothing about
the length: the length also depends on {{step}}. When the step is large, the
exact length computed at (2) is small, no limit error is raised, and control
reaches the {{internalError}} at (3) -- with the correct answer sitting in
{{safeLen}}, discarded.
For the first example above, {{safeLen}} is exactly {{3}}.
h3. Scope
* BIGINT only. {{IntegralSequenceImpl}} reaches {{sequenceLength}} via
{{getSequenceLength}}, which widens its arguments with {{toLong}}, so for
INT/SMALLINT/TINYINT the subtraction cannot overflow. The DATE/TIMESTAMP
callers pass microsecond values, whose range is far from the {{long}} limits.
* Both execution paths. The interpreted path calls {{Sequence.sequenceLength}}
directly and the generated code calls the same method, so they fail identically.
* Not a recent regression. {{sequenceLength}} was introduced in its current
shape by SPARK-43393 (commit {{afc4c49927cb}}, Nov 2023), which fixed a genuine
correctness bug where the {{long}} length computation overflowed silently and
{{sequence}} returned an empty array. That fix added the {{BigInt}} fallback
and, reasonably at the time, treated the tail as unreachable -- the overflow
cases it was written against all had huge lengths. The gap is that a large
{{step}} makes the exact length small, so the tail is reachable after all. The
method has not been modified since, so every release containing SPARK-43393
(3.3.4, 3.4.2, 3.5.1, 4.0.0 and later) is affected.
h3. Suggested fix
At (3), return {{safeLen.toInt}} instead of throwing. The value has already
been proven to be at most {{ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}} by the
check immediately above, so the narrowing conversion is safe, and the
element-filling code already handles steps of this magnitude correctly --
{{sequence(-4611686018427387904L, 4611686018427387903L,
4611686018427387903L)}}, whose endpoints are close enough together to avoid the
overflow, returns {{[-4611686018427387904, -1, 4611686018427387902]}} today.
Tests belong in {{CollectionExpressionsSuite}}, which exercises both the
interpreted and the codegen path through {{checkEvaluation}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
