[
https://issues.apache.org/jira/browse/SPARK-20451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-20451.
---------------------------------
Resolution: Fixed
Assignee: Sameer Agarwal
Fix Version/s: 2.3.0
2.2.0
2.1.1
2.0.3
> Filter out nested mapType datatypes from sort order in randomSplit
> ------------------------------------------------------------------
>
> Key: SPARK-20451
> URL: https://issues.apache.org/jira/browse/SPARK-20451
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0, 2.1.0, 2.2.0
> Reporter: Sameer Agarwal
> Assignee: Sameer Agarwal
> Fix For: 2.0.3, 2.1.1, 2.2.0, 2.3.0
>
>
> In {{randomSplit}}, It is possible that the underlying dataframe doesn't
> guarantee the ordering of rows in its constituent partitions each time a
> split is materialized which could result in overlapping splits.
> To prevent this, we explicitly sort each input partition to make the ordering
> deterministic. Given that MapTypes cannot be sorted they should be explicitly
> pruned out from the sort order. Additionally, if the resulting sort order is
> empty, we then materialize the dataset to guarantee determinism.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
