Yaoxuan Wu created SPARK-60095:
----------------------------------
Summary: Interpreted MAX/MIN over collated strings return a wrong
value
Key: SPARK-60095
URL: https://issues.apache.org/jira/browse/SPARK-60095
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0, 4.0.0, 5.0.0
Reporter: Yaoxuan Wu
{code:java}
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[1]").getOrCreate() # all rows in
one partition
df = spark.createDataFrame([("abcd",), ("xy",), ("ab",)], "s STRING COLLATE
UTF8_LCASE")print(df.agg(F.max("s"), F.min("s")).collect())
# [Row(max(s)='xy', min(s)='ab')]: correct with
codegenspark.conf.set("spark.sql.codegen.wholeStage", "false")
spark.conf.set("spark.sql.codegen.factoryMode", "NO_CODEGEN")
print(df.agg(F.max("s"), F.min("s")).collect())
# expected: [Row(max(s)='xy', min(s)='ab')]
# Spark 4.2.0: [Row(max(s)='ab', min(s)='ab')] {code}
Same result on 4.1.1, 4.2.0 and master (4.0.0 fails the same way; use a CAST to
UTF8_BINARY in the output to see it, since collecting a collated column returns
no rows on 4.0.0). It happens for UTF8_LCASE, UTF8_LCASE_RTRIM and UNICODE_CI,
not for the default UTF8_BINARY. With more rows the result can also be
corrupted bytes (we saw 'Ssv\x00'), i.e. the buffer points into memory that was
overwritten. The rows must be in one partition.
*Cause:* InternalRow.getWriter copies strings with case StringType => ...
v.asInstanceOf[UTF8String].copy(). That pattern only matches the default
StringType object; a collated StringType(id) falls through to case _ =>
input.update(ordinal, v) and is stored without a copy. Fix: case _: StringType
=>.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
