You can bump up number of partitions while creating the rdd you are using for df On 17 Jul 2015 21:03, "Harit Vishwakarma" <harit.vishwaka...@gmail.com> wrote:
> Hi, > > I used createDataFrame API of SqlContext in python. and getting > OutOfMemoryException. I am wondering if it is creating whole dataFrame in > memory? > I did not find any documentation describing memory usage of Spark APIs. > Documentation given is nice but little more details (specially on memory > usage/ data distribution etc.) will really help. > > -- > Regards > Harit Vishwakarma > >
