If optimization is the problem, can we use precomputed hashes?
On Mon, Sep 26, 2016 at 4:50 PM, Lakshmi Rajagopalan <laks...@indix.com> wrote: > Can you please help me understand why the MapType shouldn't be part of > equality tests? Practically, if we are using json line formats, the ideal > equals is every key should map to exactly the same value in both the maps > Which also hold true in Aesthetic case where a MapType can be thought of as > a function (Extensional definition). > > On Mon, Sep 26, 2016 at 3:43 PM, Takeshi Yamamuro <linguin....@gmail.com> > wrote: > >> Hi, >> >> Have you check this jira? >> https://issues.apache.org/jira/browse/SPARK-9415 >> >> // maropu >> >> On Mon, Sep 26, 2016 at 7:09 PM, Lakshmi Rajagopalan <laks...@indix.com> >> wrote: >> >>> Hi, >>> >>> We wanted to extend the existing '===' on Column to support deep equals >>> on Maps. >>> >>> >>> Currently it checks for == which does referential checks for maps. >>> >>> https://github.com/apache/spark/blob/master/sql/catalyst/src >>> /main/scala/org/apache/spark/sql/catalyst/expressions/ >>> predicates.scala#L420 >>> >>> Is there any possible work around for this? I have a patch in my mind to >>> fix Equals for MapType also. Any thoughts? >>> >>> Thanks, >>> Lakshmi Rajagopalan >>> >> >> >> >> -- >> --- >> Takeshi Yamamuro >> > >
