Re: Sorting data large data- "too many open files" exception

set ulimit quite high in root mode & that should resolve it. Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi On Mon, May 26, 2014 at 7:48 PM, Matt Kielo wrote: > Hello, > > I currently have a task always failing with > "ja

Sorting data large data- "too many open files" exception

Hello, I currently have a task always failing with "java.io.FileNotFoundException: [...]/shuffle_0_257_2155 (Too many open files)" when I run sorting operations such as distinct, sortByKey, or reduceByKey on a large number of partitions. Im working with 365 GB of data which is being split into 59