It's not necessary, I will create a PR to remove them. For larger dict/list/tuple, the pickle approach may have less RPC calls, better performance.
Davies On Tue, Jan 13, 2015 at 4:53 AM, Meethu Mathew <meethu.mat...@flytxt.com> wrote: > Hi all, > > In the python object to java conversion done in the method _py2java in > spark/python/pyspark/mllib/common.py, why we are doing individual > conversion using MpaConverter,ListConverter? The same can be acheived using > > bytearray(PickleSerializer().dumps(obj)) > obj = sc._jvm.SerDe.loads(bytes) > > Is there any performance gain or something in using individual converters > rather than PickleSerializer? > > -- > > Regards, > > *Meethu* --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
