Thanks everyone. I studied the source code and realized minPartitions is
passed over to Hadoop's InputFormat and its upto the InputFormat
implementation to use the parameter as a hint.
Thanks,
Aniket
On Fri, Jan 2, 2015, 7:13 AM Rishi Yadav wrote:
> Hi Ankit,
>
> Optional number of partitions v
Hi Ankit,
Optional number of partitions value is to increase number of partitions not
reduce it from default value.
On Thu, Jan 1, 2015 at 10:43 AM, Aniket Bhatnagar <
aniket.bhatna...@gmail.com> wrote:
> I am trying to read a file into a single partition but it seems like
> sparkContext.textFil
I am trying to read a file into a single partition but it seems like
sparkContext.textFile ignores the passed minPartitions value. I know I can
repartition the RDD but I was curious to know if this is expected or if
this is a bug that needs to be further investigated?
