>From what I understand, Spark code was written this way because you don't end up with very small partitions.
In your case, look at the size of the cluster. If 66 partitions can make good use of your cluster, it should be fine. On Tue, Apr 26, 2016 at 2:27 PM, Ulanov, Alexander <alexander.ula...@hpe.com > wrote: > Hi Ted, > > > > I have 36 files of size ~600KB and the rest 74 are about 400KB. > > > > Is there a workaround rather than changing Sparks code? > > > > Best regards, Alexander > > > > *From:* Ted Yu [mailto:yuzhih...@gmail.com] > *Sent:* Tuesday, April 26, 2016 1:22 PM > *To:* Ulanov, Alexander <alexander.ula...@hpe.com> > *Cc:* dev@spark.apache.org > *Subject:* Re: Number of partitions for binaryFiles > > > > Here is the body of StreamFileInputFormat#setMinPartitions : > > > > def setMinPartitions(context: JobContext, minPartitions: Int) { > > val totalLen = > listStatus(context).asScala.filterNot(_.isDirectory).map(_.getLen).sum > > val maxSplitSize = math.ceil(totalLen / math.max(minPartitions, > 1.0)).toLong > > super.setMaxSplitSize(maxSplitSize) > > > > I guess what happened was that among the 100 files you had, there were ~60 > files whose sizes were much bigger than the rest. > > According to the way max split size is computed above, you ended up with > fewer partitions. > > > > I just performed a test using local directory where 3 files were > significantly larger than the rest and reproduced what you observed. > > > > Cheers > > > > On Tue, Apr 26, 2016 at 11:10 AM, Ulanov, Alexander < > alexander.ula...@hpe.com> wrote: > > Dear Spark developers, > > > > I have 100 binary files in local file system that I want to load into > Spark RDD. I need the data from each file to be in a separate partition. > However, I cannot make it happen: > > > > scala> sc.binaryFiles("/data/subset").partitions.size > > res5: Int = 66 > > > > The “minPartitions” parameter does not seems to help: > > scala> sc.binaryFiles("/data/subset", minPartitions = 100).partitions.size > > res8: Int = 66 > > > > At the same time, Spark produces the required number of partitions with > sc.textFiles (though I cannot use it because my files are binary): > > scala> sc.textFile("/data/subset").partitions.size > > res9: Int = 100 > > > > Could you suggest how to force Spark to load binary files each in a > separate partition? > > > > Best regards, Alexander > > >
