file number is the same as basic hash-based shuffle.
>>>
>>>
>>>
>>> Thanks
>>>
>>> Jerry
>>>
>>>
>>>
>>> *From:* Jianshi Huang [mailto:jianshi.hu...@gmail.com]
>>> *Sent:* Thursday, July 31, 2014 10:3
mailto:jianshi.hu...@gmail.com]
>> *Sent:* Thursday, July 31, 2014 10:34 AM
>> *To:* user@spark.apache.org
>> *Cc:* xia...@sjtu.edu.cn
>> *Subject:* Re: spark.shuffle.consolidateFiles seems not working
>>
>>
>>
>> Ok... but my question is why spark.shuffle.cons
> *To:* user@spark.apache.org
> *Cc:* xia...@sjtu.edu.cn
> *Subject:* Re: spark.shuffle.consolidateFiles seems not working
>
>
>
> Ok... but my question is why spark.shuffle.consolidateFiles is working
> (or is it)? Is this a bug?
>
>
>
> On Wed, Jul 30, 2014 at 4:2
: user@spark.apache.org
Cc: xia...@sjtu.edu.cn
Subject: Re: spark.shuffle.consolidateFiles seems not working
Ok... but my question is why spark.shuffle.consolidateFiles is working (or is
it)? Is this a bug?
On Wed, Jul 30, 2014 at 4:29 PM, Larry Xiao
mailto:xia...@sjtu.edu.cn>> wrote:
Hi J
Ok... but my question is why spark.shuffle.consolidateFiles is working (or
is it)? Is this a bug?
On Wed, Jul 30, 2014 at 4:29 PM, Larry Xiao wrote:
> Hi Jianshi,
>
> I've met similar situation before.
> And my solution was 'ulimit', you can use
>
> -a to see your current settings
> -n to set o
Hi Jianshi,
I've met similar situation before.
And my solution was 'ulimit', you can use
-a to see your current settings
-n to set open files limit
(and other limits also)
And I set -n to 10240.
I see spark.shuffle.consolidateFiles helps by reusing open files.
(so I don't know to what extend d
I'm using Spark 1.0.1 on Yarn-Client mode.
SortByKey always reports a FileNotFoundExceptions with messages says "too
many open files".
I already set spark.shuffle.consolidateFiles to true:
conf.set("spark.shuffle.consolidateFiles", "true")
But it seems not working. What are the other possible
