Upgrade Spark-1.3.0 to Spark-1.4.0 in CDH5.4

Hello all, The configuration of my cluster is as follows; # 4 noded cluster running on Centos OS 6.4 # spark-1.3.0 installed on all I would like to use SparkR shipped with spark-1.4.0. I checked Cloudera and find that the latest release CDH5.4 still does not have the spark-1.4.0. Forums like Stack

Re: SparkR Error in sparkR.init(master=“local”) in RStudio

Hi, Try this > Sys.setenv(SPARK_HOME="C:\\spark-1.4.0") # The path to your spark > installation > .libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths())) > library("SparkR", lib.loc="C:\\spark-1.4.0\\lib") # The path to the lib > folder in the spark location > library(SparkR) >

Re: sparkR

I had been facing this problem for a long time and this practically forced me to move to pyspark. This is what I tried after reading the posts here > Sys.setenv(SPARK_HOME="C:\\spark-1.4.0") > .libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths())) > library("SparkR", lib.loc="C:

Re: sparkR

Please can you explain how did you set this second step in windows environment? .libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths())) I mean to ask where do I type this command, at R prompt or in command prompt? Thanks for your time. Regards, Ashish -- View this message i

Re: Is it possible to change the default port number 7077 for spark?

Many thanks for your response. Regards, Ashish -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Is-it-possible-to-change-the-default-port-number-7077-for-spark-tp23774p23797.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --

Is it possible to change the default port number 7077 for spark?

Hello all, In my lab a colleague installed and configured spark 1.3.0 on a 4 noded cluster on CDH5.4 environment. The default port number for our spark configuration is 7456. I have been trying to SSH to spark-master from using this port number but it fails every time giving error JVM is timed out

Re: DLL load failed: %1 is not a valid win32 application on invoking pyspark

Not really a clean solution but I solved the problem by reinstalling Anaconda -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/DLL-load-failed-1-is-not-a-valid-win32-application-on-invoking-pyspark-tp23733p23743.html Sent from the Apache Spark User List mail

WindowsError: [Error 2] The system cannot find the file specified

Hi, My environment: OS= Windows 7 64 bit, Python version=2.7, Spark=1.3.0 Hadoop=winutils Q1) When i invoke the command from pyspark import SparkContext i get an error message DLL load failed: %1 is not a valid Win32 application. However when I invoke the command import pyspark there is no

DLL load failed: %1 is not a valid win32 application on invoking pyspark

Hi, I get the error, "DLL load failed: %1 is not a valid win32 application" whenever I invoke pyspark. Attached is the screenshot of the same. Is there any way I can get rid of it. Still being new to PySpark and have had, a not so pleasant experience so far most probably because I am on a windows

JVM is not ready after 10 seconds

Hi, I am trying to connect a worker to the master. The spark master is on cloudera manager and I know the master IP address and port number. I downloaded the spark binary for CDH4 on the worker machine and then when I try to invoke the command > sc = sparkR.init("master="ip address:port number")