Can you please help me understand why the MapType shouldn't be part of equality tests? Practically, if we are using json line formats, the ideal equals is every key should map to exactly the same value in both the maps Which also hold true in Aesthetic case where a MapType can be thought of as a function (Extensional definition).
On Mon, Sep 26, 2016 at 3:43 PM, Takeshi Yamamuro <linguin....@gmail.com> wrote: > Hi, > > Have you check this jira? > https://issues.apache.org/jira/browse/SPARK-9415 > > // maropu > > On Mon, Sep 26, 2016 at 7:09 PM, Lakshmi Rajagopalan <laks...@indix.com> > wrote: > >> Hi, >> >> We wanted to extend the existing '===' on Column to support deep equals >> on Maps. >> >> >> Currently it checks for == which does referential checks for maps. >> >> https://github.com/apache/spark/blob/master/sql/catalyst/ >> src/main/scala/org/apache/spark/sql/catalyst/expressions >> /predicates.scala#L420 >> >> Is there any possible work around for this? I have a patch in my mind to >> fix Equals for MapType also. Any thoughts? >> >> Thanks, >> Lakshmi Rajagopalan >> > > > > -- > --- > Takeshi Yamamuro >
