Re: toPandas very slow

2016-03-22 Thread Josh Levy-Kramer
or > Numpy > > Arrays using MapPartitions for each partition. Maybe a standard solution > > around this line of thought could be built. The integration is quite > tedious > > ;) > > > > I hope this helps. > > > > Regards, > > Mark > > >

toPandas very slow

2016-03-22 Thread Josh Levy-Kramer
/blob/a60f91284ceee64de13f04559ec19c13a820a133/python/pyspark/rdd.py#L123 Josh Levy-Kramer Data Scientist @ Starcount