I found the problem. I was manually constructing the CLASSPATH and
SPARK_CLASSPATH because I needed jars for running the cassandra lib.
For some reason that I cannot explain, it was this that was causing the
issue. Maybe one of the jars had a log4j.properties rolled up in it?
I removed almost all
Out of curiosity, how do you actually launch pyspark in your set-up?
On Wed, Oct 1, 2014 at 3:44 PM, Rick Richardson
wrote:
> Here is the other relevant bit of my set-up:
> MASTER=spark://sparkmaster:7077
> IPYTHON_OPTS="notebook --pylab inline --ip=0.0.0.0"
> CASSANDRA_NODES="cassandra1|cassand
Here is the other relevant bit of my set-up:
MASTER=spark://sparkmaster:7077
IPYTHON_OPTS="notebook --pylab inline --ip=0.0.0.0"
CASSANDRA_NODES="cassandra1|cassandra2|cassandra3"
PYSPARK_SUBMIT_ARGS="--master $MASTER --deploy-mode client --num-executors
6 --executor-memory 1g --executor-cores 1" i
I was starting PySpark as a profile within IPython Notebook as per:
http://blog.cloudera.com/blog/2014/08/how-to-use-ipython-notebook-with-apache-spark/
My setup looks like:
import os
import sys
spark_home = os.environ.get('SPARK_HOME', None)
if not spark_home:
raise ValueError('SPARK_HOME e
How do you setup IPython to access pyspark in notebook?
I did as following, it worked for me:
$ export SPARK_HOME=/opt/spark-1.1.0/
$ export
PYTHONPATH=/opt/spark-1.1.0/python:/opt/spark-1.1.0/python/lib/py4j-0.8.2.1-src.zip
$ ipython notebook
All the logging will go into console (not in notebo
Thanks for your reply. Unfortunately changing the log4j.properties within
SPARK_HOME/conf has no effect on pyspark for me. When I change it in the
master or workers the log changes have the desired effect, but pyspark
seems to ignore them. I have changed the levels to WARN, changed the
appender
On Tue, Sep 30, 2014 at 10:14 PM, Rick Richardson
wrote:
> I am experiencing significant logging spam when running PySpark in IPython
> Notebok
>
> Exhibit A: http://i.imgur.com/BDP0R2U.png
>
> I have taken into consideration advice from:
> http://apache-spark-user-list.1001560.n3.nabble.com/Disa
I am experiencing significant logging spam when running PySpark in IPython
Notebok
Exhibit A: http://i.imgur.com/BDP0R2U.png
I have taken into consideration advice from:
http://apache-spark-user-list.1001560.n3.nabble.com/Disable-all-spark-logging-td1960.html
also
http://stackoverflow.com/ques
