If you are using systemd, you will need to specify the limit in the service
file. I had run into this problem and discovered the solution from the
following references:
* https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1
* http://serverfault.com/a/678861
On Fri, Feb 5, 2016 at 1:18 PM, Nirav
For centos there's also /etc/security/limits.d/90-nproc.conf that may need
modifications.
Services that you expect to use new limits needs to be restarted. Simple
thing to do is to reboot the machine.
On Fri, Feb 5, 2016 at 3:59 AM, Ted Yu wrote:
> bq. and *"session required pam_limits.so"*.
>
bq. and *"session required pam_limits.so"*.
What was the second file you modified ?
Did you make the change on all the nodes ?
Please see the verification step in
https://easyengine.io/tutorials/linux/increase-open-files-limit/
On Fri, Feb 5, 2016 at 1:42 AM, Mohamed Nadjib MAMI
wrote:
> Hell
It turns out the mesos can overwrite the OS ulimit -n setting. So we have
increased the mesos slave ulimit -n setting.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Too-many-open-files-exception-on-reduceByKey-tp2462p25019.html
Sent from the Apache Spark U
You are right, I did find that mesos overwrite this to a smaller number.So we
will modify that and try to run again. Thanks!
Tian
On Thursday, October 8, 2015 4:18 PM, DB Tsai wrote:
Try to run to see actual ulimit. We found that mesos overrides the ulimit
which causes the issue.
i
Try to run to see actual ulimit. We found that mesos overrides the ulimit
which causes the issue.
import sys.process._
val p = 1 to 100
val rdd = sc.parallelize(p, 100)
val a = rdd.map(x=> Seq("sh", "-c", "ulimit -n").!!.toDouble.toLong).collect
Sincerely,
DB Tsai
I hit this issue with spark 1.3.0 stateful application (with
updateStateByKey) function on mesos. It will
fail after running fine for about 24 hours.
The error stack trace as below, I checked ulimit -n and we have very large
numbers set on the machines.
What else can be wrong?
15/09/27 18:45:11 W
I don't think that is the issue. I have it setup to run in a thread pool
but I have set the pool size to 1 for this test until I get this resolved.
I am having some problems with using the Spark web portal since it is
picking a random port and with the way my environment is setup, by time I
have fi
Here is the code in which NewHadoopRDD register close handler and be called
when the task is completed (
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala#L136
).
>From my understanding, possibly the reason is that this `foreach` code in
your i
ah, now that does sound suspicious...
On 2 Sep 2015, at 14:09, Sigurd Knippenberg
mailto:sig...@knippenberg.com>> wrote:
Yep. I know. It's was set to 32K when I ran this test. If I bump it to 64K the
issue goes away. It still doesn't make sense to me that the Spark job doesn't
release its file
Yep. I know. It's was set to 32K when I ran this test. If I bump it to 64K
the issue goes away. It still doesn't make sense to me that the Spark job
doesn't release its file handles until the end of the job instead of doing
that while my loop iterates.
Sigurd
On Wed, Sep 2, 2015 at 4:33 AM, Steve
On 31 Aug 2015, at 19:49, Sigurd Knippenberg
mailto:sig...@knippenberg.com>> wrote:
I know I can adjust the max open files allowed by the OS but I'd rather fix the
underlaying issue.
bumping up the OS handle limits is step #1 of installing a hadoop cluster
https://wiki.apache.org/hadoop/TooM
there is no bad consequences.
>
>2. I will try to overlook admin access and see if I can get
>anything with only user rights
>
>From: Ted Yu [mailto:yuzhih...@gmail.com]
>Sent: Wednesday, July 29, 2015 12:59 PM
>To: Ellafi, Saif A.
>Cc:
>Subject: Re: Too many open fil
[mailto:yuzhih...@gmail.com]
Sent: Wednesday, July 29, 2015 12:59 PM
To: Ellafi, Saif A.
Cc:
Subject: Re: Too many open files
Please increase limit for open files:
http://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux
On Jul 29, 2015, at 8:39 AM
Please increase limit for open files:
http://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux
> On Jul 29, 2015, at 8:39 AM,
> wrote:
>
> Hello,
>
> I’ve seen a couple emails on this issue but could not find anything to solve
> my situation.
>
>
you probably should increase file handles limit for user that all processes
are running with(spark master & workers)
e.g.
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
On 29 July 2015 at 18:39, wrote:
> Hello,
>
> I’ve seen a couple emails on this issue but could
I terminated the old job and now start a new one. Currently, the Spark
streaming job has been running for 2 hours and when I use lsof, I do not
see many files related to the Spark job.
BTW, I am running Spark streaming using local[2] mode. The batch is 5
seconds and it has around 50 lines to read
Did you use lsof to see what files were opened during the job?
On Thu, Apr 30, 2015 at 1:05 PM, Bill Jay
wrote:
> The data ingestion is in outermost portion in foreachRDD block. Although
> now I close the statement of jdbc, the same exception happened again. It
> seems it is not related to the d
The data ingestion is in outermost portion in foreachRDD block. Although
now I close the statement of jdbc, the same exception happened again. It
seems it is not related to the data ingestion part.
On Wed, Apr 29, 2015 at 8:35 PM, Cody Koeninger wrote:
> Use lsof to see what files are actually b
Use lsof to see what files are actually being held open.
That stacktrace looks to me like it's from the driver, not executors.
Where in foreach is it being called? The outermost portion of foreachRDD
runs in the driver, the innermost portion runs in the executors. From the
docs:
https://spark.a
This function is called in foreachRDD. I think it should be running in the
executors. I add the statement.close() in the code and it is running. I
will let you know if this fixes the issue.
On Wed, Apr 29, 2015 at 4:06 PM, Tathagata Das wrote:
> Is the function ingestToMysql running on the dri
Also cc;ing Cody.
@Cody maybe there is a reason for doing connection pooling even if there is
not performance difference.
TD
On Wed, Apr 29, 2015 at 4:06 PM, Tathagata Das wrote:
> Is the function ingestToMysql running on the driver or on the executors?
> Accordingly you can try debugging whil
Is the function ingestToMysql running on the driver or on the executors?
Accordingly you can try debugging while running in a distributed manner,
with and without calling the function.
If you dont get "too many open files" without calling ingestToMysql(), the
problem is likely to be in ingestToMys
Maybe add statement.close() in finally block ?
Streaming / Kafka experts may have better insight.
On Wed, Apr 29, 2015 at 2:25 PM, Bill Jay
wrote:
> Thanks for the suggestion. I ran the command and the limit is 1024.
>
> Based on my understanding, the connector to Kafka should not open so many
Thanks for the suggestion. I ran the command and the limit is 1024.
Based on my understanding, the connector to Kafka should not open so many
files. Do you think there is possible socket leakage? BTW, in every batch
which is 5 seconds, I output some results to mysql:
def ingestToMysql(data: Arr
Can you run the command 'ulimit -n' to see the current limit ?
To configure ulimit settings on Ubuntu, edit */etc/security/limits.conf*
Cheers
On Wed, Apr 29, 2015 at 2:07 PM, Bill Jay
wrote:
> Hi all,
>
> I am using the direct approach to receive real-time data from Kafka in the
> following li
I'm executing my application in local mode (with --master local[*]).
I'm using ubuntu and I've put "session required pam_limits.so" into
/etc/pam.d/common-session
but it doesn't work
On Mon, Mar 30, 2015 at 4:08 PM, Ted Yu wrote:
> bq. In /etc/secucity/limits.conf set the next values:
>
> Have
bq. In /etc/secucity/limits.conf set the next values:
Have you done the above modification on all the machines in your Spark
cluster ?
If you use Ubuntu, be sure that the /etc/pam.d/common-session file contains
the following line:
session required pam_limits.so
On Mon, Mar 30, 2015 at 5:08 AM
Hi.
I've done relogin, in fact, I put 'uname -n' and returns 100, but it
crashs.
I'm doing reduceByKey and SparkSQL mixed over 17 files (250MB-500MB/file)
Regards.
Miguel Angel.
On Mon, Mar 30, 2015 at 1:52 PM, Akhil Das
wrote:
> Mostly, you will have to restart the machines to get the ul
Mostly, you will have to restart the machines to get the ulimit effect (or
relogin). What operation are you doing? Are you doing too many
repartitions?
Thanks
Best Regards
On Mon, Mar 30, 2015 at 4:52 PM, Masf wrote:
> Hi
>
> I have a problem with temp data in Spark. I have fixed
> spark.shuffl
Ops，the last reply didn't go to the user list. Mail app's fault.
Shuffling happens in the cluster, so you need change all the nodes in the
cluster.
Sent from my iPhone
> On 2014年8月30日, at 3:10, Sudha Krishna wrote:
>
> Hi,
>
> Thanks for your response. Do you know if I need to change this
Hi,
I am having the same problem reported by Michael. I am trying to open 30
files. ulimit -n shows the limit is 1024. So I am not sure why the program
is failing with "Too many open files" error. The total size of all the 30
files is 230 GB.
I am running the job on a cluster with 10 nodes, eac
Sorry, I also have some follow-up questions.
"In general if a node in your cluster has C assigned cores and you run a
job with X reducers then Spark will open C*X files in parallel and start
writing."
Some questions came to mind just now:
1) It would be nice to have a brief overview as to what th
Thanks. Just curious, is there a default number of reducers that are used?
-Matt Cheah
On Mon, Mar 10, 2014 at 7:22 PM, Patrick Wendell wrote:
> Hey Matt,
>
> The best way is definitely just to increase the ulimit if possible,
> this is sort of an assumption we make in Spark that clusters will
Hey Matt,
The best way is definitely just to increase the ulimit if possible,
this is sort of an assumption we make in Spark that clusters will be
able to move it around.
You might be able to hack around this by decreasing the number of
reducers but this could have some performance implications f
35 matches
Mail list logo
