Vijay,
Are you closing the fileinputstream at the end of each loop ( in.close())? My
guess is those streams aren't close and thus the "too many open files"
exception.
On Tuesday, January 5, 2016 8:03 AM, Priya Ch
<[email protected]> wrote:
Can some one throw light on this ?
Regards,Padma Ch
On Mon, Dec 28, 2015 at 3:59 PM, Priya Ch <[email protected]> wrote:
Chris, we are using spark 1.3.0 version. we have not set
spark.streaming.concurrentJobs this parameter. It takes the default value.
Vijay,
From the tack trace it is evident that
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$1.apply$mcVI$sp(ExternalSorter.scala:730)
is throwing the exception. I opened the spark source code and visited the line
which is throwing this exception i.e
The lie which is marked in red is throwing the exception. The file is
ExternalSorter.scala in org.apache.spark.util.collection package.
i went through the following blog
http://blog.cloudera.com/blog/2015/01/improving-sort-performance-in-apache-spark-its-a-double/
and understood that there is merge factor which decide the number of on-disk
files that could be merged. Is it some way related to this ?
Regards,Padma CH
On Fri, Dec 25, 2015 at 7:51 PM, Chris Fregly <[email protected]> wrote:
and which version of Spark/Spark Streaming are you using?
are you explicitly setting the spark.streaming.concurrentJobs to something
larger than the default of 1?
if so, please try setting that back to 1 and see if the problem still exists.
this is a dangerous parameter to modify from the default - which is why it's
not well-documented.
On Wed, Dec 23, 2015 at 8:23 AM, Vijay Gharge <[email protected]> wrote:
Few indicators -
1) during execution time - check total number of open files using lsof command.
Need root permissions. If it is cluster not sure much !2) which exact line in
the code is triggering this error ? Can you paste that snippet ?
On Wednesday 23 December 2015, Priya Ch <[email protected]> wrote:
ulimit -n 65000
fs.file-max = 65000 ( in etc/sysctl.conf file)
Thanks,Padma Ch
On Tue, Dec 22, 2015 at 6:47 PM, Yash Sharma <[email protected]> wrote:
Could you share the ulimit for your setup please ? - Thanks, via mobile,
excuse brevity. On Dec 22, 2015 6:39 PM, "Priya Ch"
<[email protected]> wrote:
Jakob, Increased the settings like fs.file-max in /etc/sysctl.conf and also
increased user limit in /etc/security/limits.conf. But still see the same issue.
On Fri, Dec 18, 2015 at 12:54 AM, Jakob Odersky <[email protected]> wrote:
It might be a good idea to see how many files are open and try increasing the
open file limit (this is done on an os level). In some application use-cases it
is actually a legitimate need.
If that doesn't help, make sure you close any unused files and streams in your
code. It will also be easier to help diagnose the issue if you send an
error-reproducing snippet.
--
Regards,Vijay Gharge
--
Chris FreglyPrincipal Data Solutions EngineerIBM Spark Technology Center, San
Francisco, CAhttp://spark.tc | http://advancedspark.com
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
