Rajesh Balamohan created SPARK-21971:
----------------------------------------
Summary: Too many open files in Spark due to concurrent files
being opened
Key: SPARK-21971
URL: https://issues.apache.org/jira/browse/SPARK-21971
Project: Spark
Issue Type: Bug
Components: Spark Core, SQL
Affects Versions: 2.1.0
Reporter: Rajesh Balamohan
Priority: Minor
When running Q67 of TPC-DS at 1 TB dataset on multi node cluster, it
consistently fails with "too many open files" exception.
{noformat}
O scheduler.TaskSetManager: Finished task 25.0 in stage 844.0 (TID 243786) in
394 ms on machine111.xyz (executor 2) (189/200)
17/08/20 10:33:45 INFO scheduler.TaskSetManager: Finished task 172.0 in stage
844.0 (TID 243932) in 11996 ms on cn116-10.l42scl.hortonworks.com (executor 6)
(190/200)
17/08/20 10:37:40 WARN scheduler.TaskSetManager: Lost task 144.0 in stage 844.0
(TID 243904, machine1.xyz, executor 1): java.nio.file.FileSystemException:
/grid/3/hadoop/yarn/local/usercache/rbalamohan/appcache/application_1490656001509_7207/blockmgr-5180e3f0-f7ed-44bb-affc-8f99f09ba7bc/28/temp_local_690afbf7-172d-4fdb-8492-3e2ebd8d5183:
Too many open files
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at
org.apache.spark.io.NioBufferedFileInputStream.<init>(NioBufferedFileInputStream.java:43)
at
org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillReader.<init>(UnsafeSorterSpillReader.java:75)
at
org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillWriter.getReader(UnsafeSorterSpillWriter.java:150)
at
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.getIterator(UnsafeExternalSorter.java:607)
at
org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray.generateIterator(ExternalAppendOnlyUnsafeRowArray.scala:169)
at
org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray.generateIterator(ExternalAppendOnlyUnsafeRowArray.scala:173)
{noformat}
Cluster was configured with multiple cores per executor.
Window function uses "spark.sql.windowExec.buffer.spill.threshold=4096" which
causes large number of spills in larger dataset. With multiple cores per
executor, this reproduces easily.
{{UnsafeExternalSorter::getIterator()}} invokes {{spillWriter.getReader}} for
all the available spillWriters. {{UnsafeSorterSpillReader}} opens up the file
in its constructor and closes the file later as a part of its close() call.
This causes too many open files issue.
Note that this is not a file leak, but more of concurrent files being open at
any given time depending on the dataset being processed.
One option could be to increase "spark.sql.windowExec.buffer.spill.threshold"
so that fewer spill files are generated, but it is hard to determine the
sweetspot for all workload. Another option is to set ulimit to "unlimited" for
files, but that would not be a good production setting. It would be good to
consider reducing the number of concurrent "UnsafeExternalSorter::getIterator".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
