[Problem Solved]Re: Spark partition size tuning

2016-01-27 Thread Jia Zou
yon (default is 512MB), but above method can't work >>> for Tachyon data. >>> >>> Do you have any suggestions? Thanks very much! >>> >>> Best Regards, >>> Jia >>> >>> >>> -- Forwarded message -- >>> F

Re: Spark partition size tuning

2016-01-27 Thread Jia Zou
gestions? Thanks very much! >> >> Best Regards, >> Jia >> >> >> -- Forwarded message -- >> From: Jia Zou >> Date: Thu, Jan 21, 2016 at 10:05 PM >> Subject: Spark partition size tuning >> To: "user @spark" >

Re: Spark partition size tuning

2016-01-26 Thread Gene Pang
a partition size for input data that > is stored in Tachyon (default is 512MB), but above method can't work for > Tachyon data. > > Do you have any suggestions? Thanks very much! > > Best Regards, > Jia > > > -- Forwarded message -- > From: Jia Zou

Re: Spark partition size tuning

2016-01-26 Thread Pavel Plotnikov
y much! > > Best Regards, > Jia > > > -- Forwarded message ------ > From: Jia Zou > Date: Thu, Jan 21, 2016 at 10:05 PM > Subject: Spark partition size tuning > To: "user @spark" > > > Dear all! > > When using Spark to read from local

Fwd: Spark partition size tuning

2016-01-25 Thread Jia Zou
thod can't work for Tachyon data. Do you have any suggestions? Thanks very much! Best Regards, Jia -- Forwarded message -- From: Jia Zou Date: Thu, Jan 21, 2016 at 10:05 PM Subject: Spark partition size tuning To: "user @spark" Dear all! When using Spark to re

Spark partition size tuning

2016-01-21 Thread Jia Zou
Dear all! When using Spark to read from local file system, the default partition size is 32MB, how can I increase the partition size to 128MB, to reduce the number of tasks? Thank you very much! Best Regards, Jia