Re: Down-scaling Spark on EC2 cluster

What about down-scaling when I use Mesos, does that really deteriorate the performance ? Otherwise we would probably go for spark on mesos on ec2 :) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Down-scaling-Spark-on-EC2-cluster-tp10494p12109.html Sent fro

Re: Down-scaling Spark on EC2 cluster

Any idea about the probable dates for this implementation. I believe it would be a wonderful (and essential) functionality to gain more acceptance in the community. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Down-scaling-Spark-on-EC2-cluster-tp10494p106

Down-scaling Spark on EC2 cluster

it be if it is in the middle of a task). Thanks in advance. Shubhabrata -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Down-scaling-Spark-on-EC2-cluster-tp10494.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Log analysis

I am new to spark and we are developing a data science pipeline based on spark on ec2. So far we have been using python on spark standalone cluster. However, being a newbie I would like to know more about how can I do debugging (program level) from spark logs (is it stderr ?). I find it a bit diffi

unsubscribe

unsubscribe

Can a job running on a cluster read from a local file path ?

1) Can a job (python script), running on a standalone cluster read from local file path ? 2) Does sc.addPyFile(path) create a directory or only copies the file ? 3) If the path contains a zip file, does it automatically gets unzipped ? -- View this message in context: http://apache-spark-user

Re: Deploying a python code on a spark EC2 cluster

In order to check if there is any issue with python API I ran a scala application provided in the examples. Still the same error ./bin/run-example org.apache.spark.examples.SparkPi spark://[Master-URL]:7077 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/mn

Re: Deploying a python code on a spark EC2 cluster

This is the error from stderr: Spark Executor Command: "java" "-cp" ":/root/ephemeral-hdfs/conf:/root/ephemeral-hdfs/conf:/root/ephemeral-hdfs/conf:/root/spark/conf:/root/spark/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop1.0.4.jar" "-Djava.library.path=/root/ephemeral-hdfs/lib/nati

Re: Deploying a python code on a spark EC2 cluster

Well, we used the script that comes with spark I think v0.9.1. But I am gonna try the newer version (1.0rvc2 script). I shall keep you posted about my findings. Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Deploying-a-python-code-on-a-spark-EC2-cl

Re: Deploying a python code on a spark EC2 cluster

Moreover it seems all the workers are registered and have sufficient memory (2.7GB where as I have asked for 512 MB). The UI also shows the jobs are running on the slaves. But on the termial it is still the same error "Initial job has not accepted any resources; check your cluster UI to ensure that

Re: Deploying a python code on a spark EC2 cluster

Spark Command: /usr/lib/jvm/java-1.7.0/bin/java -cp :/root/ephemeral-hdfs/conf:/root/ephemeral-hdfs/conf:/root/ephemeral-hdfs/conf:/root/ephemeral-hdfs/conf:/root/spark/conf:/root/spark/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop1.0.4.jar -Dspark.akka.logLifecycleEvents=true -Djava.

Deploying a python code on a spark EC2 cluster

I am stuck with an issue for last two days and did not find any solution after several hours of googling. Here is the details. The following is a simple python code (Temp.py): import sys from random import random from operator import add from pyspark import SparkContext from pyspark import Spar