Calling Python code from Scala

Hi, I have a Spark project in Scala and I would like to call some Python functions from within the program. Both parts are quite big, so re-coding everything in one language is not really an option. The workflow would be: - Creating a RDD with Scala code - Mapping a Python function over this RDD

Re: Managing permissions when saving as text file

Ok, I solved this problem by : - changing the primary group of launcher to spark - adding "umask 002" in launcher's .bashrc and spark's init.d script -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Managing-permissions-when-saving-as-text-file-tp21928p21943.

Managing permissions when saving as text file

Hi, I'm having a problem involving file permissions on the local filesystem. On a first machine, I have two different users : - launcher, which launches my job from an uber jar file - spark, which runs the master On a second machine, I have a user spark (same uid/gid as the other) which runs the

Re: OutOfMemoryError with ramdom forest and small training dataset

Ok, I would suggest adding SPARK_DRIVER_MEMORY in spark-env.sh, with a larger amount of memory than the default 512m -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/OutOfMemoryError-with-ramdom-forest-and-small-training-dataset-tp21598p21618.html Sent from