Ok, but at least we can have a separate binary comparison called
DeepEqualTo as an Expression which at least makes the performance issues
explicit and also have a way to achieve the equality on maps.
In our case, the maps are very small. And this restriction completely
reduces the expressibility of
yea, for all I know, there is no reasonable way to implement fast and
efficient equality checks on ArrayBasedMapData (See also:
https://github.com/apache/spark/pull/13847).
On Mon, Sep 26, 2016 at 9:04 PM, Lakshmi Rajagopalan
wrote:
> If optimization is the problem, can we use precomputed hash
If optimization is the problem, can we use precomputed hashes?
On Mon, Sep 26, 2016 at 4:50 PM, Lakshmi Rajagopalan
wrote:
> Can you please help me understand why the MapType shouldn't be part of
> equality tests? Practically, if we are using json line formats, the ideal
> equals is every key s
Can you please help me understand why the MapType shouldn't be part of
equality tests? Practically, if we are using json line formats, the ideal
equals is every key should map to exactly the same value in both the maps
Which also hold true in Aesthetic case where a MapType can be thought of as
a fu
Hi,
Have you check this jira?
https://issues.apache.org/jira/browse/SPARK-9415
// maropu
On Mon, Sep 26, 2016 at 7:09 PM, Lakshmi Rajagopalan
wrote:
> Hi,
>
> We wanted to extend the existing '===' on Column to support deep equals
> on Maps.
>
>
> Currently it checks for == which does referen
