Yibo Dong created SPARK-59043:
---------------------------------
Summary: SimplifyCaseConversionExpressions changes Unicode
case-conversion semantics
Key: SPARK-59043
URL: https://issues.apache.org/jira/browse/SPARK-59043
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Environment: Spark 5.0.0-SNAPSHOT, commit
9da9f8d673914d1648514f59d85e3adafb300d1a; macOS arm64; Java 17.0.17
Reporter: Yibo Dong
A legal nested Unicode case-conversion expression returns different results
depending on whether SimplifyCaseConversionExpressions is enabled.
h3. Reproduction
{code:sql}
SELECT lower(upper(s2)) AS result
FROM (VALUES ('ı')) AS t(s)
LATERAL VIEW explode(array(s)) e AS s2;
{code}
With the default optimizer configuration, the query returns:
{code}
ı
{code}
If only SimplifyCaseConversionExpressions is excluded:
{code}
--conf
spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.SimplifyCaseConversionExpressions
{code}
the same query returns:
{code}
i
{code}
h3. Expected behavior
Both configurations should preserve the semantics of the original expression and
return:
{code}
i
{code}
For the Unicode character U+0131 LATIN SMALL LETTER DOTLESS I:
{code}
upper('ı') = 'I'
lower('I') = 'i'
{code}
Therefore:
{code}
lower(upper('ı')) = 'i'
{code}
h3. Actual behavior
With SimplifyCaseConversionExpressions enabled, Spark returns `ı`.
The optimizer rule simplifies:
{code}
lower(upper(child))
{code}
to:
{code}
lower(child)
{code}
However, this transformation is not semantics-preserving for all Unicode
characters. For this input:
{code}
lower(upper('ı')) = 'i'
lower('ı') = 'ı'
{code}
h3. Additional information
The only configuration change needed to avoid the incorrect result is excluding
SimplifyCaseConversionExpressions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
