Hi,
My code was running properly but then it suddenly gave this error. Can you
just put some light on it.
#######################
0 KB, free: 38.7 MB)
14/07/09 01:46:12 INFO BlockManagerMaster: Updated info of block rdd_2212_4
14/07/09 01:46:13 INFO PythonRDD: Times: total = 1486, boot = 698, init =
626, finish = 162
Exception in thread "stdin writer for python" 14/07/09 01:46:14 INFO
MemoryStore: ensureFreeSpace(61480) called with cur
Mem=270794224, maxMem=311387750
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.<init>(Unknown Source)
at
org.apache.spark.api.python.PythonRDD$$anon$2.run(PythonRDD.scala:62)
14/07/09 01:46:15 INFO MemoryStore: Block rdd_2212_0 stored as values to
memory (estimated size 60.0 KB, free 38.7 MB)
Exception in thread "stdin writer for python" java.lang.OutOfMemoryError:
Java heap space
at java.io.BufferedOutputStream.<init>(Unknown Source)
at
org.apache.spark.api.python.PythonRDD$$anon$2.run(PythonRDD.scala:62)
14/07/09 01:46:18 INFO BlockManagerMasterActor$BlockManagerInfo: Added
rdd_2212_0 in memory on shawn-PC:51451 (size: 60.
0 KB, free: 38.7 MB)
PySpark worker failed with exception:
Traceback (most recent call last):
File "F:\spark-0.9.1\spark-0.9.1\bin\..\/python/pyspark/worker.py", line
50, in main
split_index = read_int(infile)
File "F:\spark-0.9.1\spark-0.9.1\python\pyspark\serializers.py", line
328, in read_int
raise EOFError
EOFError
14/07/09 01:46:25 INFO BlockManagerMaster: Updated info of block rdd_2212_0
Exception in thread "stdin writer for python" java.lang.OutOfMemoryError:
Java heap space
PySpark worker failed with exception:
Traceback (most recent call last):
File "F:\spark-0.9.1\spark-0.9.1\bin\..\/python/pyspark/worker.py", line
50, in main
split_index = read_int(infile)
File "F:\spark-0.9.1\spark-0.9.1\python\pyspark\serializers.py", line
328, in read_int
raise EOFError
EOFError
Exception in thread "Executor task launch worker-3"
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "spark-akka.actor.default-dispa
tcher-15"
Exception in thread "Executor task launch worker-1" Exception in thread
"Executor task launch worker-2" java.lang.OutOfM
emoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "Executor task launch worker-0" Exception in thread
"Executor task launch worker-5" java.lang.OutOfM
emoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
14/07/09 01:46:52 WARN BlockManagerMaster: Error sending message to
BlockManagerMaster in 1 attempts
akka.pattern.AskTimeoutException:
Recipient[Actor[akka://spark/user/BlockManagerMaster#920823400]] had
already been term
inated.
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
at
org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
at
org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
at org.apache.spark.storage.BlockManager.org
$apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
at
org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
at
akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
14/07/09 01:46:56 WARN BlockManagerMaster: Error sending message to
BlockManagerMaster in 2 attempts
akka.pattern.AskTimeoutException:
Recipient[Actor[akka://spark/user/BlockManagerMaster#920823400]] had
already been term
inated.
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
at
org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
at
org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
at org.apache.spark.storage.BlockManager.org
$apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
at
org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
at
akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
14/07/09 01:46:59 WARN BlockManagerMaster: Error sending message to
BlockManagerMaster in 3 attempts
akka.pattern.AskTimeoutException:
Recipient[Actor[akka://spark/user/BlockManagerMaster#920823400]] had
already been term
inated.
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
at
org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
at
org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
at org.apache.spark.storage.BlockManager.org
$apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
at
org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
at
akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Exception in thread "Connection manager future execution context-0"
java.lang.Error: org.apache.spark.SparkException: Er
ror sending message to BlockManagerMaster [message =
HeartBeat(BlockManagerId(<driver>, shawn-PC, 51451, 0))]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.spark.SparkException: Error sending message to
BlockManagerMaster [message = HeartBeat(BlockManage
rId(<driver>, shawn-PC, 51451, 0))]
at
org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:176)
at
org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
at org.apache.spark.storage.BlockManager.org
$apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
at
org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
at
akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
... 3 more
Caused by: akka.pattern.AskTimeoutException:
Recipient[Actor[akka://spark/user/BlockManagerMaster#920823400]] had alread
y been terminated.
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
at
org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
... 8 more
Traceback (most recent call last):
File ".\naive_bayes_analyser.py", line 110, in <module>
model = NaiveBayes.train(sc.parallelize(array(train_data)))
File "F:\spark-0.9.1\spark-0.9.1\python\pyspark\mllib\classification.py",
line 122, in train
ans = sc._jvm.PythonMLLibAPI().trainNaiveBayes(dataBytes._jrdd, lambda_)
File
"F:\spark-0.9.1\spark-0.9.1\bin\..\python\lib\py4j-0.8.1-src.zip\py4j\java_gateway.py",
line 535, in __call__
File
"F:\spark-0.9.1\spark-0.9.1\bin\..\python\lib\py4j-0.8.1-src.zip\py4j\java_gateway.py",
line 363, in send_command
File
"F:\spark-0.9.1\spark-0.9.1\bin\..\python\lib\py4j-0.8.1-src.zip\py4j\java_gateway.py",
line 472, in send_command
File "C:\Python27\lib\socket.py", line 430, in readline
data = recv(1)
KeyboardInterrupt
^CThe process tried to write to a nonexistent pipe.
^CTerminate batch job (Y/N)? The process tried to write to a nonexistent
pipe.
##################
Thanks
--
Rahul K Bhojwani
3rd Year B.Tech
Computer Science and Engineering
National Institute of Technology, Karnataka
