yea, for all I know, there is no reasonable way to implement fast and efficient equality checks on ArrayBasedMapData (See also: https://github.com/apache/spark/pull/13847).
On Mon, Sep 26, 2016 at 9:04 PM, Lakshmi Rajagopalan <laks...@indix.com> wrote: > If optimization is the problem, can we use precomputed hashes? > > > On Mon, Sep 26, 2016 at 4:50 PM, Lakshmi Rajagopalan <laks...@indix.com> > wrote: > >> Can you please help me understand why the MapType shouldn't be part of >> equality tests? Practically, if we are using json line formats, the ideal >> equals is every key should map to exactly the same value in both the maps >> Which also hold true in Aesthetic case where a MapType can be thought of as >> a function (Extensional definition). >> >> On Mon, Sep 26, 2016 at 3:43 PM, Takeshi Yamamuro <linguin....@gmail.com> >> wrote: >> >>> Hi, >>> >>> Have you check this jira? >>> https://issues.apache.org/jira/browse/SPARK-9415 >>> >>> // maropu >>> >>> On Mon, Sep 26, 2016 at 7:09 PM, Lakshmi Rajagopalan <laks...@indix.com> >>> wrote: >>> >>>> Hi, >>>> >>>> We wanted to extend the existing '===' on Column to support deep >>>> equals on Maps. >>>> >>>> >>>> Currently it checks for == which does referential checks for maps. >>>> >>>> https://github.com/apache/spark/blob/master/sql/catalyst/src >>>> /main/scala/org/apache/spark/sql/catalyst/expressions/predic >>>> ates.scala#L420 >>>> >>>> Is there any possible work around for this? I have a patch in my mind >>>> to fix Equals for MapType also. Any thoughts? >>>> >>>> Thanks, >>>> Lakshmi Rajagopalan >>>> >>> >>> >>> >>> -- >>> --- >>> Takeshi Yamamuro >>> >> >> > -- --- Takeshi Yamamuro
