Hello Neelesh,
Thank you for the checklist for determining the correct configuration of
Spark. I will go through these and let you know if I have further questions.
Regards,
Chris
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/spark-defaults-conf-optim
Hi Chris,
Thank you for posting the question.
Tuning spark configurations is a tricky task since there are a lot factors
to consider.
The configurations that you listed cover the most them.
To understand the situation that can guide you in making a decision about
tuning:
1) What kind of spark app
ion:
BDA v3 server : SUN SERVER X4-2L
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
CPU cores : 32
GB of memory (>=63): 63
number of disks : 12 spark-defaults.conf
spark.driver.memory 20g
spark.executor.memory 40g
spark.executor.extraJavaOptions -XX:+PrintGCDetai
t; > Thanks,
> >
> > -Axel
> >
> >
> >
> > On Wed, Sep 2, 2015 at 10:34 PM, Davies Liu
> wrote:
> >>
> >> This should be a bug, could you create a JIRA for it?
> >>
> >> On Wed, Sep 2, 2015 at 4:38 PM, Axel Dahl
> wrote:
I think it's a missing feature.
On Wed, Sep 2, 2015 at 10:58 PM, Axel Dahl wrote:
> So a bit more investigation, shows that:
>
> if I have configured spark-defaults.conf with:
>
> "spark.files library.py"
>
> then if I call
>
> "spark-submi
So a bit more investigation, shows that:
if I have configured spark-defaults.conf with:
"spark.files library.py"
then if I call
"spark-submit.py -v test.py"
I see that my "spark.files" default option has been replaced with
"spark.files t
This should be a bug, could you create a JIRA for it?
On Wed, Sep 2, 2015 at 4:38 PM, Axel Dahl wrote:
> in my spark-defaults.conf I have:
> spark.files file1.zip, file2.py
> spark.master spark://master.domain.com:7077
>
> If I execute:
> bin/pyspar
in my spark-defaults.conf I have:
spark.files file1.zip, file2.py
spark.master spark://master.domain.com:7077
If I execute:
bin/pyspark
I can see it adding the files correctly.
However if I execute
bin/spark-submit test.py
where test.py relies on the file1.zip, I get
: Friday, July 31, 2015 11:49 AM
To: Wang, Ningjun (LNG-NPV)
Cc: user@spark.apache.org
Subject: Re: How to register array class with Kyro in spark-defaults.conf
For the second exception, was there anything following SparkException which
would give us more clue ?
Can you tell us how EsDoc is
olve this problem?
>
>
>
> Ningjun
>
>
>
> *From:* Wang, Ningjun (LNG-NPV)
> *Sent:* Thursday, July 30, 2015 11:06 AM
> *To:* user@spark.apache.org
> *Subject:* How to register array class with Kyro in spark-defaults.conf
>
>
>
> I regist
Does anybody have any idea how to solve this problem?
Ningjun
From: Wang, Ningjun (LNG-NPV)
Sent: Thursday, July 30, 2015 11:06 AM
To: user@spark.apache.org
Subject: How to register array class with Kyro in spark-defaults.conf
I register my class with Kyro in spark-defaults.conf as follow
I register my class with Kyro in spark-defaults.conf as follow
spark.serializer
org.apache.spark.serializer.KryoSerializer
spark.kryo.registrationRequired true
spark.kryo.classesToRegister ltn.analytics.es.EsDoc
But I got the following
ser@spark.apache.org"
> Subject: Re: Unable to use dynamicAllocation if spark.executor.instances
> is set in spark-defaults.conf
>
> Hi Jonathan,
>
> This is a problem that has come up for us as well, because we'd like
> dynamic allocation to be turned on by de
mailto:user@spark.apache.org>>
Subject: Re: Unable to use dynamicAllocation if spark.executor.instances is set
in spark-defaults.conf
Hi Jonathan,
This is a problem that has come up for us as well, because we'd like dynamic
allocation to be turned on by default in some setups, but not brea
I can see how the exception would be helpful if, say, you tried to pass
> both "-c spark.executor.instances" (or --num-executors) *and* "-c
> spark.dynamicAllocation.enabled=true" to spark-submit on the command line
> (as opposed to having one of them in spark-defau
pass both
"-c spark.executor.instances" (or --num-executors) *and* "-c
spark.dynamicAllocation.enabled=true" to spark-submit on the command line (as
opposed to having one of them in spark-defaults.conf and one of them in the
spark-submit args), but currently there doesn
bump
From: Jonathan Kelly mailto:jonat...@amazon.com>>
Date: Tuesday, July 14, 2015 at 4:23 PM
To: "user@spark.apache.org<mailto:user@spark.apache.org>"
mailto:user@spark.apache.org>>
Subject: Unable to use dynamicAllocation if spark.executor.instances is set in
spar
I've set up my cluster with a pre-calcualted value for spark.executor.instances
in spark-defaults.conf such that I can run a job and have it maximize the
utilization of the cluster resources by default. However, if I want to run a
job with dynamicAllocation (by passi
r replying
Original message From: Akhil Das
Date:07/01/2015 2:27 AM (GMT-05:00)
To: Yana Kadiyska Cc:
user@spark.apache.org Subject: Re: Difference between
spark-defaults.conf and SparkConf.set
.addJar works for me when i run it as a stand-alone application (without
using sp
river.extraClassPath
>
> to point to some external JARs. If I set them in spark-defaults.conf
> everything works perfectly.
> However, if I remove spark-defaults.conf and just create a SparkConf and
> call
> .set("spark.executor.extraClassPath","...)
> .set(&q
Hi folks, running into a pretty strange issue:
I'm setting
spark.executor.extraClassPath
spark.driver.extraClassPath
to point to some external JARs. If I set them in spark-defaults.conf
everything works perfectly.
However, if I remove spark-defaults.conf and just create a SparkConf and
call
So no takers regarding why spark-defaults.conf is not being picked up.
Here is another one:
If Zookeeper is configured in Spark why do we need to start a slave like
this:
spark-1.3.0-bin-hadoop2.4/sbin/start-slave.sh 1 spark://somemaster:7077
i.e. why do we need to specify the master url
Thanks.
I've set SPARK_HOME and SPARK_CONF_DIR appropriately in .bash_profile
But when I start worker like this
spark-1.3.0-bin-hadoop2.4/sbin/start-slave.sh
I still get
failed to launch org.apache.spark.deploy.worker.Worker:
Default is conf/spark-defaults
You should distribute your configuration file to workers and set the
appropriate environment variables, like HADOOP_HOME, SPARK_HOME,
HADOOP_CONF_DIR, SPARK_CONF_DIR.
On Mon, Apr 27, 2015 at 12:56 PM James King wrote:
> I renamed spark-defaults.conf.template to spark-defaults.conf
> and i
I renamed spark-defaults.conf.template to spark-defaults.conf
and invoked
spark-1.3.0-bin-hadoop2.4/sbin/start-slave.sh
But I still get
failed to launch org.apache.spark.deploy.worker.Worker:
--properties-file FILE Path to a custom Spark properties file
Here is related problem:
http://apache-spark-user-list.1001560.n3.nabble.com/Launching-history-server-problem-td12574.html
but no answer.
What I'm trying to do: wrap spark-history with /etc/init.d script
Problems I have: can't make it read spark-defaults.conf
I've put this file h
Thank you.
Zhanfeng Huo
From: Andrew Or
Date: 2014-09-02 08:21
To: Zhanfeng Huo
CC: user
Subject: Re: Can value in spark-defaults.conf support system variables?
No, not currently.
2014-09-01 2:53 GMT-07:00 Zhanfeng Huo :
Hi，all：
Can value in spark-defaults.conf support system variables
No, not currently.
2014-09-01 2:53 GMT-07:00 Zhanfeng Huo :
> Hi，all：
>
> Can value in spark-defaults.conf support system variables?
>
> Such as "mess = ${user.home}/${user.name}".
>
> Best Regards
>
> --
> Zhanfeng Huo
>
Hi，all：
Can value in spark-defaults.conf support system variables?
Such as "mess = ${user.home}/${user.name}".
Best Regards
Zhanfeng Huo
y ask them to create their own
spark-defaults.conf since this is set to read-only. A workaround is to set it
to a shared folder e.g. /user/spark/logs and user permission 1777. This isn't
really ideal since other people can see what are the other jobs running on the
shared cluster.
It will be ni
Hi Andrew,
It's definitely not bad practice to use spark-shell with HistoryServer. The
issue here is not with spark-shell, but the way we pass Spark configs to
the application. spark-defaults.conf does not currently support embedding
environment variables, but instead interprets everything
Hi All,
Not sure if anyone has ran into this problem, but this exist in spark 1.0.0
when you specify the location in conf/spark-defaults.conf for
spark.eventLog.dir hdfs:///user/$USER/spark/logs
to use the $USER env variable.
For example, I'm running the command with user 'test'.
in an aws ec2 cluster that i launched using the spark-ec2
> script that comes with spark
> and I use the "-v master" option to run the head version.
>
> If I then log into master and make changes
> to spark/conf/spark-defaults.conf
> How do I make the changes take effe
I am running in an aws ec2 cluster that i launched using the spark-ec2
script that comes with spark
and I use the "-v master" option to run the head version.
If I then log into master and make changes to spark/conf/spark-defaults.conf
How do I make the changes take effect across the cl
34 matches
Mail list logo
