[
https://issues.apache.org/jira/browse/SPARK-20534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Li updated SPARK-20534:
----------------------------
Fix Version/s: (was: 2.2.1)
(was: 2.3.0)
2.2.0
> Outer generators skip missing records if used alone
> ---------------------------------------------------
>
> Key: SPARK-20534
> URL: https://issues.apache.org/jira/browse/SPARK-20534
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Environment: master 814a61a867ded965433c944c90961df529ac83ab
> Reporter: Maciej Szymkiewicz
> Assignee: Maciej Szymkiewicz
> Fix For: 2.2.0
>
>
> Example data:
> {code}
> val df = Seq(
> (1, Some("a" :: "b" :: "c" :: Nil)),
> (2, None),
> (3, Some("a" :: Nil)
> )).toDF("k", "vs")
> {code}
> Correct behavior if there are other expressions:
> {code}
> df.select($"k", explode_outer($"vs")).show
> // +---+----+
> // | k| col|
> // +---+----+
> // | 1| a|
> // | 1| b|
> // | 1| c|
> // | 2|null|
> // | 3| a|
> // +---+----+
> df.select($"k", posexplode_outer($"vs")).show
> // +---+----+----+
> // | k| pos| col|
> // +---+----+----+
> // | 1| 0| a|
> // | 1| 1| b|
> // | 1| 2| c|
> // | 2|null|null|
> // | 3| 0| a|
> // +---+----+----+
> {code}
> Incorrect behavior if used alone:
> {code}
> df.select(explode_outer($"vs")).show
> // +---+
> // |col|
> // +---+
> // | a|
> // | b|
> // | c|
> // | a|
> // +---+
> df.select(posexplode_outer($"vs")).show
> // +---+---+
> // |pos|col|
> // +---+---+
> // | 0| a|
> // | 1| b|
> // | 2| c|
> // | 0| a|
> // +---+---+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
