[
https://issues.apache.org/jira/browse/SPARK-52628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-52628.
----------------------------------
Resolution: Invalid
Resolving as Invalid — this is a usage/how-to question rather than a specific
Spark defect or actionable change. Usage questions are best directed to
[email protected] (https://spark.apache.org/community.html) or Stack
Overflow (tag apache-spark). Findings from triage: Verified the cited code.
Temp views live in an in-memory HashMap (SessionCatalog.scala:305) that never
persists SQL text; direct SQL likewise never persists. Only the Hive
permanent-view path persists the raw view string via
HiveClientImpl.scala:1225-1227 (setViewOriginalText/setViewExpandedText), which
the Hive Metastore writes into its backing RDBMS. The exact three-way signature
(direct SQL correct, temp view correct, Hive view = 0 rows for '女'/'男')
isolates the failure to the metastore's backing-DB persistence layer, not
Spark: Spark handles the strings as UTF-8 in the JVM and reaches the
Please reopen with a concrete reproducer or a specific proposed change if this
is actually a bug or an actionable improvement.
> Difference results from Hive View and Temp View based on the same SQL query
> ---------------------------------------------------------------------------
>
> Key: SPARK-52628
> URL: https://issues.apache.org/jira/browse/SPARK-52628
> Project: Spark
> Issue Type: Question
> Components: SQL
> Affects Versions: 3.5.3
> Reporter: Yu-Ting LIN
> Priority: Major
>
> *Describe the bug*
> I have a view {{demo.scr_opt}} which contains more than one records (1060767).
> If I query it directly using the following SQL command, then it returns more
> than one records (1060767).
> {code:java}
> spark.sql("""SELECT
> *,
> CASE
> WHEN SexName = '女' THEN 0
> WHEN SexName = '男' THEN 1
> END AS Sex
> FROM demo.scr_opt WHERE SexName IN ('女', '男')""").count() {code}
>
> However, if I created a HIVE permanent view based on the SQL, then this view
> returns no records.
> {code:java}
> spark.sql("""CREATE OR REPLACE VIEW demo.scr_opt1 AS (
> SELECT
> *,
> CASE
> WHEN SexName = '女' THEN 0
> WHEN SexName = '男' THEN 1
> END AS Sex
> FROM demo.scr_opt WHERE SexName IN ('女', '男'))""")
> spark.sql("SELECT * FROM demo.scr_opt1).count() {code}
>
> If I used the query with createOrReplaceTempView("demo.scr_opt1"), then it
> return records again.
> {code:java}
> spark.sql("""SELECT
> *,
> CASE
> WHEN SexName = '女' THEN 0
> WHEN SexName = '男' THEN 1
> END AS Sex
> FROM demo.scr_opt WHERE SexName IN ('女',
> '男')""").createOrReplaceTempView("scr_opt1")
> spark.sql("SELECT * FROM scr_opt1")
> {code}
>
> What could be the root cause of the strange phenomenon ?
> Does chinese character matter ?
> This can be reproduced in different Spark environment (Local Kubernetes or
> Google Dataproc.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
