Spark 1.1 (slow, working), Spark 1.2 (fast, freezing)
Hi, I just recently tried to migrate from Spark 1.1 to Spark 1.2 - using PySpark. Initially, I was super glad, noticing that Spark 1.2 is way faster than Spark 1.1. However, the initial joy faded quickly when I noticed that all my stuff didn't successfully terminate operations anymore. Using Spark 1.1 it still works perfectly fine, though. Specifically, the execution just freezes without any error output at one point, when calling a joint map() and collect() statement (after having it called many times successfully before in a loop). Any clue? Or do I have to wait for the next version? Best, Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-1-slow-working-Spark-1-2-fast-freezing-tp21278.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark 1.1 (slow, working), Spark 1.2 (fast, freezing)
Seems like it is a bug rather than a feature. I filed a bug report: https://issues.apache.org/jira/browse/SPARK-5363 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-1-slow-working-Spark-1-2-fast-freezing-tp21278p21317.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
SPARK_LOCAL_DIRS Issue
Hi, Using Spark 1.2 I ran into issued setting SPARK_LOCAL_DIRS to a different path then local directory. On our cluster we have a folder for temporary files (in a central file system), which is called /scratch. When setting SPARK_LOCAL_DIRS=/scratch/ I get: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.newAPIHadoopFile. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, XXX): java.io.IOException: Function not implemented at sun.nio.ch.FileDispatcherImpl.lock0(Native Method) at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:91) at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1022) at java.nio.channels.FileChannel.lock(FileChannel.java:1052) at org.apache.spark.util.Utils$.fetchFile(Utils.scala:379) Using SPARK_LOCAL_DIRS=/tmp, however, works perfectly. Any idea? Best, Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SPARK-LOCAL-DIRS-Issue-tp21602.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Broadcasting Large Objects Fails?
Hi, I am trying to broadcast large objects (order of a couple of 100 MBs). However, I keep getting errors when trying to do so: Traceback (most recent call last): File "/LORM_experiment.py", line 510, in broadcast_gradient_function = sc.broadcast(gradient_function) File "/scratch/users/213444/spark/python/pyspark/context.py", line 643, in broadcast return Broadcast(self, value, self._pickled_broadcast_vars) File "/scratch/users/213444/spark/python/pyspark/broadcast.py", line 65, in __init__ self._path = self.dump(value, f) File "/scratch/users/213444/spark/python/pyspark/broadcast.py", line 82, in dump cPickle.dump(value, f, 2) SystemError: error return without exception set 15/02/22 04:52:14 ERROR Utils: Uncaught exception in thread delete Spark local dirs java.lang.IllegalStateException: Shutdown in progress Any idea how to prevent that? I got plenty of RAM, so there shouldn't be any problem with that. Thanks, Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Broadcasting-Large-Objects-Fails-tp21752.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Using Hadoop InputFormat in Python
Yes, thanks great. This seems to be the issue. At least running with spark-submit works as well. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-Hadoop-InputFormat-in-Python-tp12067p12126.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
RDD Grouping
Hi, is there a way such that I can group items in an RDD together such that I can process them using parallelize/map Let's say I have data items with keys 1...1000 e.g. loading RDD = sc. newAPIHadoopFile(...).cache() Now, I would like them to be processed in chunks of e.g. tens chunk1=[0..9],chunk2=[10..19],...,chunk100=[991..999] sc.parallelize([chunk1,,chunk100]).map(process my chunk) I thought I could use groupBy() or something like that but the return-type is PipelinedRDD, which is not iterable. Anybody an idea? Thanks in advance, Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Grouping-tp12407.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: RDD Grouping
Thanks a lot. Yes, this mapPartitions seems a better way of dealing with this problem as for groupBy() I need to collect() data before applying parallelize(), which is expensive. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Grouping-tp12407p12424.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
RDD Row Index
Hi, I wonder if there is something like an (row) index to of the elements in the RDD. Specifically, my RDD is generated from a series of files, where the value corresponds the file contents. Ideally, I would like to have the keys to be an enumeration of the file number e.g. (0,),(1,). Any idea? Thanks, Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Row-Index-tp12457.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Mapping with extra arguments
Hi, I am using Spark in Python. I wonder if there is a possibility for passing extra arguments to the mapping function. In my scenario, after each map I update parameters, which I want to use in the folllowning new iteration of mapping. Any idea? Thanks in advance. -Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Mapping-with-extra-arguments-tp12541.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Mapping with extra arguments
Thanks. That's pretty much what I need. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Mapping-with-extra-arguments-tp12541p12548.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Mapping with extra arguments
Thanks for the nice example. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Mapping-with-extra-arguments-tp12541p12549.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: RDD Row Index
Thanks. As my files are defined to be non-splittable, I eventually I ended up using mapPartitionsWithIndex() taking the split ID as index def g(splitIndex, iterator): yield (splitIndex, iterator.next()) myRDD.mapPartitionsWithIndex(g) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RDD-Row-Index-tp12457p12550.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Spark-Submit Python along with JAR
Hi, I'd like to run my python script using "spark-submit" together with a JAR file containing Java specifications for a Hadoop file system. How can I do that? It seems I can either provide a JAR file or a PYthon file to spark-submit. So far I have been running my code in ipython with IPYTHON_OPTS="notebook --pylab inline" /usr/local/spark/bin/pyspark --jars /usr/local/spark/HadoopFileFormat.jar Best, Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Submit-Python-along-with-JAR-tp16938.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Yarn-Client Python
Hi there, I am trying to run Spark on YARN managed cluster using Python (which requires yarn-client mode). However, I cannot get it running (same with example apps). Using spark-submit to launch the script I get the following warning: WARN cluster.YarnClientClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory The log file states: ERROR yarn.ExecutorLauncher: Failed to connect to driver at localhost:52006, retrying ... Any idea? Would greatly appreciate that. Thanks, Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Yarn-Client-Python-tp17551.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Yarn-Client Python
Hi Andrew, thanks for trying to help. However, I am a bit confused now. I'm not setting any 'spark.driver.host', particularly spark-defaults.conf is empty/non-exisiting. I thought this is only required when running Spark standalone mode. Isn't it the case, when using YARN all the configuration needs to be done within Hadoop? As I am not launching any spark service I don't have any SparkUI, but the Hadoop UI. Cheers, Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Yarn-Client-Python-tp17551p17556.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: cannot run spark shell in yarn-client mode
Hi Marco, I have the same issue. Did you fix it by chance? How? Best, Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/cannot-run-spark-shell-in-yarn-client-mode-tp4013p17603.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Spark Standalone on cluster stops
Hi, I have an issue with running Spark in standalone mode on a cluster. Everything seems to run fine for a couple of minutes until Spark stops executing the tasks. Any idea? Would appreciate some help. Thanks in advance, Tassilo I get errors like that at the end: 14/10/31 16:16:59 INFO client.AppClient$ClientActor: Executor updated: app-20141031161538-0003/7 is now EXITED (Command exited with code 1) 14/10/31 16:16:59 INFO cluster.SparkDeploySchedulerBackend: Executor app-20141031161538-0003/7 removed: Command exited with code 1 14/10/31 16:16:59 INFO client.AppClient$ClientActor: Executor added: app-20141031161538-0003/11 on worker-20141031142207-localhost-36911 (localhost:36911) with 12 cores 14/10/31 16:16:59 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20141031161538-0003/11 on hostPort localhost:36911 with 12 cores, 16.0 GB RAM 14/10/31 16:16:59 INFO client.AppClient$ClientActor: Executor updated: app-20141031161538-0003/11 is now RUNNING 14/10/31 16:16:59 INFO client.AppClient$ClientActor: Executor updated: app-20141031161538-0003/8 is now EXITED (Command exited with code 1) 14/10/31 16:16:59 INFO cluster.SparkDeploySchedulerBackend: Executor app-20141031161538-0003/8 removed: Command exited with code 1 14/10/31 16:16:59 INFO client.AppClient$ClientActor: Executor added: app-20141031161538-0003/12 on worker-20141031142207-localhost-41750 (localhost:41750) with 12 cores -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Standalone-on-cluster-stops-tp17869.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: stage failure: java.lang.IllegalStateException: unread block data
Hi, I get exactly the same error. It runs on my local machine but not on the cluster. I am running the example pi.py example. Best, Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/stage-failure-java-lang-IllegalStateException-unread-block-data-tp17751p17889.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
org.apache.hadoop.security.UserGroupInformation.doAs Issue
Hi there, I am trying to run the example code pi.py on a cluster, however, I only got it working on localhost. When trying to run in standalone mode, ./bin/spark-submit \ --master spark://[mymaster]:7077 \ examples/src/main/python/pi.py \ I get warnings about resources and memory (the workstation actually has 192GByte Memory and 32 cores). 14/11/01 21:37:05 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory 14/11/01 21:37:05 INFO client.AppClient$ClientActor: Executor updated: app-20141101213420-/4 is now EXITED (Command exited with code 1) 14/11/01 21:37:05 INFO cluster.SparkDeploySchedulerBackend: Executor app-20141101213420-/4 removed: Command exited with code 1 14/11/01 21:37:05 INFO client.AppClient$ClientActor: Executor added: app-20141101213420-/5 on worker-20141101213345-localhost-33525 (localhost:33525) with 32 cores 14/11/01 21:37:05 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20141101213420-/5 on hostPort localhost:33525 with 32 cores, 1024.0 MB RAM 14/11/01 21:37:05 INFO client.AppClient$ClientActor: Executor updated: app-20141101213420-/5 is now RUNNING 14/11/01 21:37:20 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory 14/11/01 21:37:35 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory 14/11/01 21:37:38 INFO client.AppClient$ClientActor: Executor updated: app-20141101213420-/5 is now EXITED (Command exited with code 1) 14/11/01 21:37:38 INFO cluster.SparkDeploySchedulerBackend: Executor app-20141101213420-/5 removed: Command exited with code 1 14/11/01 21:37:38 INFO client.AppClient$ClientActor: Executor added: app-20141101213420-/6 on worker-20141101213345-localhost-33525 (localhost:33525) with 32 cores 14/11/01 21:37:38 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20141101213420-/6 on hostPort localhost:33525 with 32 cores, 1024.0 MB RAM 14/11/01 21:37:38 INFO client.AppClient$ClientActor: Executor updated: app-20141101213420-/6 is now RUNNING 14/11/01 21:37:50 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory 14/11/01 21:38:05 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory 14/11/01 21:38:11 INFO client.AppClient$ClientActor: Executor updated: app-20141101213420-/6 is now EXITED (Command exited with code 1) 14/11/01 21:38:11 INFO cluster.SparkDeploySchedulerBackend: Executor app-20141101213420-/6 removed: Command exited with code 1 14/11/01 21:38:11 INFO client.AppClient$ClientActor: Executor added: app-20141101213420-/7 on worker-20141101213345-localhost-33525 (localhost:33525) with 32 cores 14/11/01 21:38:11 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20141101213420-/7 on hostPort localhost:33525 with 32 cores, 1024.0 MB RAM 14/11/01 21:38:11 INFO client.AppClient$ClientActor: Executor updated: app-20141101213420-/7 is now RUNNING [..] The worker is connected successfully to the master and tries to run the code: 14/11/01 21:39:17 INFO worker.Worker: Asked to launch executor app-20141101213420-/9 for PythonPi 14/11/01 21:39:17 WARN worker.CommandUtils: SPARK_JAVA_OPTS was set on the worker. It is deprecated in Spark 1.0. 14/11/01 21:39:17 WARN worker.CommandUtils: Set SPARK_LOCAL_DIRS for node-specific storage locations. 14/11/01 21:39:17 INFO worker.ExecutorRunner: Launch command: "/usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java" "-cp" "::/etc/hadoop/spark/spark-1.1.0/conf:/etc/hadoop/spark/spark-1.1.0/assembly/target/scala-2.10/spark-assembly-1.1.0-hadoop2.5.1.jar:/etc/hadoop/conf" "-XX:MaxPermSize=128m" "-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCTimeStamps" "-Dspark.akka.frameSize=32" "-Dspark.driver.port=47509" "-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCTimeStamps" "-Xms1024M" "-Xmx1024M" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "akka.tcp://sparkDriver@localhost:47509/user/CoarseGrainedScheduler" "9" "localhost" "32" "akka.tcp://sparkWorker@localhost:33525/user/Worker" "app-20141101213420-" 14/11/01 21:39:50 INFO worker.Worker: Executor app-20141101213420-/9 finished with state EXITED message Command exited with code 1 exitStatus 1 Looking at the working thread log file in /spark-1.1.0/work/app-20141101213420-/[..]/stderr 14/11/01 21:38:46 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://driverPropsFetcher@localhost:52163] 14/11/01 21:38:46 INFO Remoting: Remoting now listens on addresses: [akka.tcp://driverPropsFetche
Spark Memory Hungry?
Hi, I am using PySpark (1.1) and I am using it for some image processing tasks. The images (RDD) are of in the order of several MB to low/mid two digit MB. However, when using the data and running operations on it using Spark, I experience blowing up memory. Is there anything I can do about it? I played around with serialization and RDD compression, but that didn't really help. Any other idea what I can do or what I should particularly aware of? Best, Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Memory-Hungry-tp18923.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Spark Standalone Scheduling
Hi, I am running some Spark code on my cluster in standalone mode. However, I have noticed that the most powerful machines (32 cores, 192 Gb mem) hardly get any tasks, whereas my small machines (8 cores, 128 Gb mem) all get plenty of tasks. The resources are all displayed correctly in the WebUI and machines all have the same configuration. When 'slaves' is to only contain the powerful machines they work well, though. However, I would like to make use of 'all' machines. Any idea what could be the reason? Or how the scheduler decides on which machine the task is assigned to? Would appreciate some help, Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Standalone-Scheduling-tp19323.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Issues running spark on cluster
Hi, I am running PySpark on a cluster. Generally it runs. However, frequently I get the warning message (and consequently, the task not being executed): WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory It is weird because all my nodes have the same specifications and the same data. Why does it work sometimes and sometimes not? Looking at the log file I see stuff like this: 15/01/14 11:42:11 INFO Worker: Disassociated [akka.tcp://sparkWorker@node001.cluster:50198] -> [akka.tcp://sparkMaster@node001:7077] Disassociated ! 15/01/14 11:42:11 ERROR Worker: Connection to master failed! Waiting for master to reconnect... 15/01/14 11:42:11 ERROR EndpointWriter: AssociationError [akka.tcp://sparkWorker@node001.cluster:50198] -> [akka.tcp://sparkExecutor@node001.cluster:35231]: Error [Association failed with [akka.tcp://sparkExecutor@node001.cluster:35231]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor@node001.cluster:35231] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: node001.cluster/172.16.6.101:35231 ] 15/01/14 11:42:11 ERROR EndpointWriter: AssociationError [akka.tcp://sparkWorker@node001.cluster:50198] -> [akka.tcp://sparkMaster@node001:7077]: Error [Association failed with [akka.tcp://sparkMaster@node001:7077]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@node001:7077] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: node001/172.16.6.101:7077 ] 15/01/14 11:42:11 INFO Worker: Disassociated [akka.tcp://sparkWorker@node001.cluster:50198] -> [akka.tcp://sparkMaster@node001:7077] Disassociated ! 15/01/14 11:42:11 ERROR Worker: Connection to master failed! Waiting for master to reconnect... 15/01/14 11:42:11 INFO RemoteActorRefProvider$RemoteDeadLetterActorRef: Message [org.apache.spark.deploy.DeployMessages$ExecutorStateChanged] from Actor[akka://sparkWorker/user/Worker#-1661660308] to Actor[akka://sparkWorker/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown' Maybe somebody has an idea? Would greatly appreciate that. -Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Issues-running-spark-on-cluster-tp21138.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Issues running spark on cluster
I got it working. It was a bug in Spark 1.1. After upgrading to 1.2 it worked. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Issues-running-spark-on-cluster-tp21138p21140.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Performance issue
Hi, I observed some weird performance issue using Spark in combination with Theano, and I have no real explanation for that. To exemplify the issue I am using the pi.py example of spark that computes pi: When I modify the function from the example: #unmodified code def f(_): x = random() * 2 - 1 y = random() * 2 - 1 return 1 if x ** 2 + y ** 2 < 1 else 0 count = sc.parallelize(xrange(1, n + 1), partitions).map(f).reduce(add) # by adding a very simple dummy function that just computes the product of two floats, the execution slows down massively (about 100x slower). Here is the slow code: # define simple function in theano that computes the product x = T.dscalar() y = T.dscalar() dummyFun = theano.function([x,y],y * x) broadcast_dummyFun = sc.broadcast(dummyFun) def f(_): x = random() * 2 - 1 y = random() * 2 - 1 # compute product tmp = broadcast_dummyFun.value(x,y) return 1 if x ** 2 + y ** 2 < 1 else 0 Any idea why it slows down so much? Using a python function that computes the product (or lambda function) again gives full-speed. I would appreciate some help on that. -Tassilo -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Performance-issue-tp21194.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Performance issue
I suspect that putting a function into shared variable incurs additional overhead? Any suggestion how to avoid that? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Performance-issue-tp21194p21210.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
