Hi Team,
My Spark job fails with below error :
Could you please advice me what is the problem with my job.
Below is my error stack:
16/08/04 05:11:06 ERROR ActorSystemImpl: Uncaught fatal error from thread
[sparkDriver-akka.actor.default-dispatcher-14] shutting down ActorSystem
[sparkDriver]
java.lang.OutOfMemoryError: GC overhead limit exceeded
at sun.reflect.ByteVectorImpl.trim(ByteVectorImpl.java:70)
at
sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:388)
at
sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:77)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:46)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at
akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
at
akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
16/08/04 05:11:06 INFO RemoteActorRefProvider$RemotingTerminator: Shutting
down remote daemon.
16/08/04 05:11:07 INFO RemoteActorRefProvider$RemotingTerminator: Remote
daemon shut down; proceeding with flushing remote transports.
16/08/04 05:11:07 INFO TaskSetManager: Finished task 18540.0 in stage 148.0
(TID 153058) in 190291 ms on lhrrhegapq005.enterprisenet.org (18536/32768)
16/08/04 05:11:07 INFO TaskSetManager: Finished task 18529.0 in stage 148.0
(TID 153044) in 190300 ms on lhrrhegapq008.enterprisenet.org (18537/32768)
16/08/04 05:11:07 INFO TaskSetManager: Finished task 18530.0 in stage 148.0
(TID 153049) in 190297 ms on lhrrhegapq005.enterprisenet.org (18538/32768)
16/08/04 05:11:07 INFO TaskSetManager: Finished task 18541.0 in stage 148.0
(TID 153062) in 190291 ms on lhrrhegapq006.enterprisenet.org (18539/32768)
16/08/04 05:11:09 INFO TaskSetManager: Finished task 18537.0 in stage 148.0
(TID 153057) in 191648 ms on lhrrhegapq003.enterprisenet.org (18540/32768)
16/08/04 05:11:10 INFO TaskSetManager: Finished task 18557.0 in stage 148.0
(TID 153073) in 193193 ms on lhrrhegapq003.enterprisenet.org (18541/32768)
16/08/04 05:11:10 INFO TaskSetManager: Finished task 18528.0 in stage 148.0
(TID 153045) in 193206 ms on lhrrhegapq007.enterprisenet.org (18542/32768)
16/08/04 05:11:10 INFO TaskSetManager: Finished task 18555.0 in stage 148.0
(TID 153072) in 193195 ms on lhrrhegapq002.enterprisenet.org (18543/32768)
16/08/04 05:11:10 ERROR YarnClientSchedulerBackend: Yarn application has
already exited with state FINISHED!
16/08/04 05:11:13 WARN QueuedThreadPool: 9 threads could not be stopped
16/08/04 05:11:13 INFO SparkUI: Stopped Spark web UI at
http://10.90.50.64:4043
16/08/04 05:11:15 INFO DAGScheduler: Stopping DAGScheduler
16/08/04 05:11:16 INFO DAGScheduler: Job 94 failed: save at
ndx_scala_util.scala:1264, took 232.788303 s
16/08/04 05:11:16 ERROR InsertIntoHadoopFsRelation: Aborting job.
org.apache.spark.SparkException: Job cancelled because SparkContext was
shut down
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:703)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:702)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at
org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:702)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:1525)
at org.apache.spark.util.EventLoop.stop(EventLoop.scala:84)
at
org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1449)
at
org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724)
at
org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1184)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1723)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:146)
at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1914)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:150)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108)
at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57)
at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57)
at
org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:69)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:140)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:138)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at
org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:138)
at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:933)
at
org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:933)
at
org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:197)
at
org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:146)
at
org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:137)
at org.apache.spark.sql.DataFrame.save(DataFrame.scala:1808)
at
com.acnielsen.madras.utils.ndx_scala_util$.newHiveTableData(ndx_scala_util.scala:1264)
at
com.acnielsen.madras.utils.ndx_scala_util$.UPDATE(ndx_scala_util.scala:238)
at
com.acnielsen.madras.pkgews_panel_extract$$anonfun$p_signed_rank_yago$1.apply(pkgews_panel_extract.scala:658)
at
com.acnielsen.madras.pkgews_panel_extract$$anonfun$p_signed_rank_yago$1.apply(pkgews_panel_extract.scala:652)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at
com.acnielsen.madras.pkgews_panel_extract$.p_signed_rank_yago(pkgews_panel_extract.scala:652)
at
com.acnielsen.madras.pkgews_panel_extract$.p_main(pkgews_panel_extract.scala:4844)
at
com.acnielsen.madras.pkgews_panel_extract$.main(pkgews_panel_extract.scala:4655)
at
com.acnielsen.madras.pkgews_panel_extract.main(pkgews_panel_extract.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/04 05:11:17 INFO TaskSetManager: Finished task 18544.0 in stage 148.0
(TID 153064) in 200319 ms on lhrrhegapq005.enterprisenet.org (18544/32768)
16/08/04 05:11:17 INFO DAGScheduler: ShuffleMapStage 148 (save at
ndx_scala_util.scala:1264) failed in 233.531 s
16/08/04 05:11:17 INFO TaskSetManager: Finished task 18552.0 in stage 148.0
(TID 153067) in 200319 ms on lhrrhegapq008.enterprisenet.org (18545/32768)
16/08/04 05:11:17 INFO DAGScheduler: ShuffleMapStage 149 (save at
ndx_scala_util.scala:1264) failed in 233.277 s
16/08/04 05:11:17 INFO TaskSetManager: Finished task 18549.0 in stage 148.0
(TID 153056) in 200323 ms on lhrrhegapq002.enterprisenet.org (18546/32768)
16/08/04 05:11:17 INFO TaskSetManager: Finished task 18539.0 in stage 148.0
(TID 153061) in 200321 ms on lhrrhegapq006.enterprisenet.org (18547/32768)
16/08/04 05:11:17 INFO YarnClientSchedulerBackend: Shutting down all
executors
16/08/04 05:11:17 INFO Remoting: Remoting shut down
16/08/04 05:11:17 INFO RemoteActorRefProvider$RemotingTerminator: Remoting
shut down.
16/08/04 05:11:17 WARN AkkaRpcEndpointRef: Error sending message [message =
StopExecutors] in 1 attempts
org.apache.spark.rpc.RpcTimeoutException:
Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]]
had already been terminated.. This timeout is controlled by
spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org
$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214)
at
org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229)
at
org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)
at scala.util.Try$.apply(Try.scala:161)
at scala.util.Failure.recover(Try.scala:185)
at
scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at
scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
at
scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)
at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at
scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280)
at
scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270)
at scala.concurrent.Future$class.recover(Future.scala:324)
at
scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153)
at
org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319)
at
org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100)
at
org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:274)
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stop(CoarseGrainedSchedulerBackend.scala:283)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:180)
at
org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:439)
at
org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1450)
at
org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724)
at
org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1184)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1723)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:146)
Caused by: akka.pattern.AskTimeoutException:
Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]]
had already been terminated.
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
at
org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307)
... 11 more
16/08/04 05:11:17 ERROR DefaultWriterContainer: Job job_201608040506_0000
aborted.
Failed to create dataset in HDFS location :
/user/hive/warehouse//TEWS_MS_TYPE_BASKET/1470301585850
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18533.0 in stage 148.0
(TID 153053) in 202457 ms on lhrrhegapq004.enterprisenet.org (18548/32768)
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18551.0 in stage 148.0
(TID 153070) in 202448 ms on lhrrhegapq002.enterprisenet.org (18549/32768)
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18553.0 in stage 148.0
(TID 153069) in 202449 ms on lhrrhegapq003.enterprisenet.org (18550/32768)
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18534.0 in stage 148.0
(TID 153047) in 202459 ms on lhrrhegapq007.enterprisenet.org (18551/32768)
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18554.0 in stage 148.0
(TID 153071) in 202958 ms on lhrrhegapq003.enterprisenet.org (18552/32768)
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18543.0 in stage 148.0
(TID 153065) in 202961 ms on lhrrhegapq004.enterprisenet.org (18553/32768)
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18538.0 in stage 148.0
(TID 153060) in 202963 ms on lhrrhegapq007.enterprisenet.org (18554/32768)
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18524.0 in stage 148.0
(TID 153048) in 202970 ms on lhrrhegapq005.enterprisenet.org (18555/32768)
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18584.0 in stage 148.0
(TID 153104) in 202166 ms on lhrrhegapq008.enterprisenet.org (18556/32768)
16/08/04 05:11:20 WARN AkkaRpcEndpointRef: Error sending message [message =
StopExecutors] in 2 attempts
org.apache.spark.rpc.RpcTimeoutException:
Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]]
had already been terminated.. This timeout is controlled by
spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org
$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214)
at
org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229)
at
org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)
at scala.util.Try$.apply(Try.scala:161)
at scala.util.Failure.recover(Try.scala:185)
at
scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at
scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
at
scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)
at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at
scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280)
at
scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270)
at scala.concurrent.Future$class.recover(Future.scala:324)
at
scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153)
at
org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319)
at
org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100)
at
org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:274)
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stop(CoarseGrainedSchedulerBackend.scala:283)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:180)
at
org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:439)
at
org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1450)
at
org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724)
at
org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1184)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1723)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:146)
Caused by: akka.pattern.AskTimeoutException:
Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]]
had already been terminated.
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
at
org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307)
... 11 more
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18556.0 in stage 148.0
(TID 153074) in 203328 ms on lhrrhegapq008.enterprisenet.org (18557/32768)
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18571.0 in stage 148.0
(TID 153087) in 202531 ms on lhrrhegapq003.enterprisenet.org (18558/32768)
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18545.0 in stage 148.0
(TID 153066) in 203332 ms on lhrrhegapq006.enterprisenet.org (18559/32768)
16/08/04 05:11:20 INFO TaskSetManager: Finished task 18581.0 in stage 148.0
(TID 153100) in 202531 ms on lhrrhegapq004.enterprisenet.org (18560/32768)
Failed to fetch affected records
16/08/04 05:11:20 INFO pkgews_panel_extract$Log$: LOG,
PKGEWS_PANEL_EXTRACT, P_MAIN, CURRENT SCHEMA - mdr_devi1_supt_test, No of
records merged into TEWS_MS_TYPE_BASKET for msc_yago_trend > 03, LOG,
PKGEWS_PANEL_EXTRACT, p_signed_rank_yago, Signed rank assignment for YAGO
Trend
16/08/04 05:11:21 INFO TaskSetManager: Finished task 18562.0 in stage 148.0
(TID 153081) in 203403 ms on lhrrhegapq004.enterprisenet.org (18561/32768)
16/08/04 05:11:21 INFO TaskSetManager: Finished task 18579.0 in stage 148.0
(TID 153084) in 203403 ms on lhrrhegapq008.enterprisenet.org (18562/32768)
16/08/04 05:11:21 INFO TaskSetManager: Finished task 18547.0 in stage 148.0
(TID 153077) in 203405 ms on lhrrhegapq007.enterprisenet.org (18563/32768)
16/08/04 05:11:21 INFO TaskSetManager: Finished task 18575.0 in stage 148.0
(TID 153098) in 203401 ms on lhrrhegapq002.enterprisenet.org (18564/32768)
16/08/04 05:11:22 INFO TaskSetManager: Finished task 18559.0 in stage 148.0
(TID 153078) in 204359 ms on lhrrhegapq004.enterprisenet.org (18565/32768)
16/08/04 05:11:22 INFO TaskSetManager: Finished task 18560.0 in stage 148.0
(TID 153082) in 204359 ms on lhrrhegapq006.enterprisenet.org (18566/32768)
16/08/04 05:11:22 INFO TaskSetManager: Finished task 18583.0 in stage 148.0
(TID 153091) in 204357 ms on lhrrhegapq008.enterprisenet.org (18567/32768)
16/08/04 05:11:22 INFO TaskSetManager: Finished task 18558.0 in stage 148.0
(TID 153075) in 204369 ms on lhrrhegapq008.enterprisenet.org (18568/32768)
16/08/04 05:11:22 INFO TaskSetManager: Finished task 18591.0 in stage 148.0
(TID 153103) in 204358 ms on lhrrhegapq006.enterprisenet.org (18569/32768)
16/08/04 05:11:22 INFO TaskSetManager: Finished task 18569.0 in stage 148.0
(TID 153089) in 204362 ms on lhrrhegapq002.enterprisenet.org (18570/32768)
16/08/04 05:11:22 INFO TaskSetManager: Finished task 18561.0 in stage 148.0
(TID 153076) in 204364 ms on lhrrhegapq002.enterprisenet.org (18571/32768)
16/08/04 05:11:22 INFO TaskSetManager: Finished task 18568.0 in stage 148.0
(TID 153088) in 204363 ms on lhrrhegapq007.enterprisenet.org (18572/32768)
16/08/04 05:11:22 INFO TaskSetManager: Finished task 18572.0 in stage 148.0
(TID 153096) in 204362 ms on lhrrhegapq004.enterprisenet.org (18573/32768)
16/08/04 05:11:23 INFO TaskSetManager: Finished task 18574.0 in stage 148.0
(TID 153094) in 205321 ms on lhrrhegapq006.enterprisenet.org (18574/32768)
16/08/04 05:11:23 INFO TaskSetManager: Finished task 18576.0 in stage 148.0
(TID 153090) in 205323 ms on lhrrhegapq003.enterprisenet.org (18575/32768)
16/08/04 05:11:23 INFO TaskSetManager: Finished task 18567.0 in stage 148.0
(TID 153086) in 205324 ms on lhrrhegapq006.enterprisenet.org (18576/32768)
16/08/04 05:11:23 INFO TaskSetManager: Finished task 18570.0 in stage 148.0
(TID 153092) in 205323 ms on lhrrhegapq002.enterprisenet.org (18577/32768)
16/08/04 05:11:23 INFO TaskSetManager: Finished task 18582.0 in stage 148.0
(TID 153102) in 205324 ms on lhrrhegapq007.enterprisenet.org (18578/32768)
16/08/04 05:11:23 INFO TaskSetManager: Finished task 18563.0 in stage 148.0
(TID 153079) in 205328 ms on lhrrhegapq002.enterprisenet.org (18579/32768)
16/08/04 05:11:23 INFO TaskSetManager: Finished task 18578.0 in stage 148.0
(TID 153097) in 205325 ms on lhrrhegapq007.enterprisenet.org (18580/32768)
16/08/04 05:11:23 INFO TaskSetManager: Finished task 18589.0 in stage 148.0
(TID 153101) in 205326 ms on lhrrhegapq005.enterprisenet.org (18581/32768)
16/08/04 05:11:23 INFO TaskSetManager: Finished task 18566.0 in stage 148.0
(TID 153080) in 205330 ms on lhrrhegapq002.enterprisenet.org (18582/32768)
16/08/04 05:11:23 INFO TaskSetManager: Finished task 18577.0 in stage 148.0
(TID 153095) in 205328 ms on lhrrhegapq003.enterprisenet.org (18583/32768)
16/08/04 05:11:23 INFO TaskSetManager: Finished task 18565.0 in stage 148.0
(TID 153085) in 205332 ms on lhrrhegapq005.enterprisenet.org (18584/32768)
16/08/04 05:11:24 WARN AkkaRpcEndpointRef: Error sending message [message =
StopExecutors] in 3 attempts
org.apache.spark.rpc.RpcTimeoutException:
Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]]
had already been terminated.. This timeout is controlled by
spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org
$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214)
at
org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229)
at
org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)
at scala.util.Try$.apply(Try.scala:161)
at scala.util.Failure.recover(Try.scala:185)
at
scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at
scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
at
scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)
at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at
scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280)
at
scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270)
at scala.concurrent.Future$class.recover(Future.scala:324)
at
scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153)
at
org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319)
at
org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100)
at
org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:274)
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stop(CoarseGrainedSchedulerBackend.scala:283)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:180)
at
org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:439)
at
org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1450)
at
org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724)
at
org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1184)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1723)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:146)
Caused by: akka.pattern.AskTimeoutException:
Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]]
had already been terminated.
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
at
org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307)
... 11 more
16/08/04 05:11:24 INFO TaskSetManager: Finished task 18564.0 in stage 148.0
(TID 153083) in 206426 ms on lhrrhegapq005.enterprisenet.org (18585/32768)
16/08/04 05:11:24 ERROR Utils: Uncaught exception in thread Yarn
application state monitor
org.apache.spark.SparkException: Error asking standalone scheduler to shut
down executors
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:278)
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stop(CoarseGrainedSchedulerBackend.scala:283)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:180)
at
org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:439)
at
org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1450)
at
org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724)
at
org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1184)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1723)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:146)
Caused by: org.apache.spark.SparkException: Error sending message [message
= StopExecutors]
at
org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:118)
at
org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
at
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:274)
... 8 more
Caused by: org.apache.spark.rpc.RpcTimeoutException:
Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]]
had already been terminated.. This timeout is controlled by
spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org
$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214)
at
org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229)
at
org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)
at scala.util.Try$.apply(Try.scala:161)
at scala.util.Failure.recover(Try.scala:185)
at
scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at
scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
at
scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)
at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at
scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280)
at
scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270)
at scala.concurrent.Future$class.recover(Future.scala:324)
at
scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153)
at
org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319)
at
org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100)
... 10 more
Caused by: akka.pattern.AskTimeoutException:
Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#-1206596405]]
had already been terminated.
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
at
org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307)
... 11 more
16/08/04 05:11:24 INFO TaskSetManager: Finished task 18573.0 in stage 148.0
(TID 153093) in 206426 ms on lhrrhegapq005.enterprisenet.org (18586/32768)
16/08/04 05:11:24 INFO TaskSetManager: Finished task 18580.0 in stage 148.0
(TID 153099) in 206425 ms on lhrrhegapq003.enterprisenet.org (18587/32768)
16/08/04 05:11:24 INFO TaskSetManager: Finished task 18585.0 in stage 148.0
(TID 153105) in 206418 ms on lhrrhegapq003.enterprisenet.org (18588/32768)
16/08/04 05:11:24 INFO TaskSetManager: Finished task 18586.0 in stage 148.0
(TID 153107) in 206431 ms on lhrrhegapq004.enterprisenet.org (18589/32768)
16/08/04 05:11:25 INFO TaskSetManager: Finished task 18596.0 in stage 148.0
(TID 153112) in 207515 ms on lhrrhegapq008.enterprisenet.org (18590/32768)
16/08/04 05:11:25 INFO TaskSetManager: Finished task 18587.0 in stage 148.0
(TID 153108) in 207517 ms on lhrrhegapq002.enterprisenet.org (18591/32768)
16/08/04 05:11:25 INFO TaskSetManager: Finished task 18588.0 in stage 148.0
(TID 153106) in 207519 ms on lhrrhegapq008.enterprisenet.org (18592/32768)
16/08/04 05:11:25 INFO TaskSetManager: Finished task 18590.0 in stage 148.0
(TID 153109) in 207519 ms on lhrrhegapq007.enterprisenet.org (18593/32768)
16/08/04 05:11:25 INFO TaskSetManager: Finished task 18594.0 in stage 148.0
(TID 153114) in 207524 ms on lhrrhegapq004.enterprisenet.org (18594/32768)
16/08/04 05:11:25 INFO TaskSetManager: Finished task 18593.0 in stage 148.0
(TID 153111) in 207526 ms on lhrrhegapq004.enterprisenet.org (18595/32768)
16/08/04 05:11:25 ERROR pkgews_panel_extract$Log$: PKGEWS_PANEL_EXTRACT,
p_signed_rank_yago, 2, Merge into TEWS_MS_TYPE_BASKET < 0 - NOT OK
java.lang.IllegalStateException: Cannot call methods on a stopped
SparkContext
at org.apache.spark.SparkContext.org
$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104)
at
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:2063)
at
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:2076)
at
org.apache.spark.sql.hive.HadoopTableReader.<init>(TableReader.scala:70)
at
org.apache.spark.sql.hive.execution.HiveTableScan.<init>(HiveTableScan.scala:77)
at
org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$$anonfun$3.apply(HiveStrategies.scala:77)
at
org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$$anonfun$3.apply(HiveStrategies.scala:77)
at
org.apache.spark.sql.SQLContext$SparkPlanner.pruneFilterProject(SQLContext.scala:853)
at
org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$.apply(HiveStrategies.scala:73)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54)
at
org.apache.spark.sql.execution.SparkStrategies$EquiJoinSelection$.makeBroadcastHashJoin(SparkStrategies.scala:92)
at
org.apache.spark.sql.execution.SparkStrategies$EquiJoinSelection$.apply(SparkStrategies.scala:101)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54)
at
org.apache.spark.sql.execution.SparkStrategies$BasicOperators$.apply(SparkStrategies.scala:346)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54)
at
org.apache.spark.sql.execution.SparkStrategies$EquiJoinSelection$.makeBroadcastHashJoin(SparkStrategies.scala:92)
at
org.apache.spark.sql.execution.SparkStrategies$EquiJoinSelection$.apply(SparkStrategies.scala:101)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54)
at
org.apache.spark.sql.execution.SparkStrategies$BasicOperators$.apply(SparkStrategies.scala:346)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:54)
at
org.apache.spark.sql.execution.SparkStrategies$Aggregation$.apply(SparkStrategies.scala:235)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at
org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59)
at
org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:926)
at
org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:924)
at
org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:930)
at
org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:930)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:53)
at
org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:1904)
at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1385)
at
com.acnielsen.madras.pkgews_panel_extract$.p_signed_rank_yago(pkgews_panel_extract.scala:685)
at
com.acnielsen.madras.pkgews_panel_extract$.p_main(pkgews_panel_extract.scala:4844)
at
com.acnielsen.madras.pkgews_panel_extract$.main(pkgews_panel_extract.scala:4655)
at
com.acnielsen.madras.pkgews_panel_extract.main(pkgews_panel_extract.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/04 05:11:25 INFO TaskSetManager: Finished task 18592.0 in stage 148.0
(TID 153110) in 207527 ms on lhrrhegapq008.enterprisenet.org (18596/32768)
16/08/04 05:11:25 INFO TaskSetManager: Finished task 18603.0 in stage 148.0
(TID 153116) in 207525 ms on lhrrhegapq006.enterprisenet.org (18597/32768)
16/08/04 05:11:25 ERROR pkgews_panel_extract$Log$: PKGEWS_PANEL_EXTRACT,
p_signed_rank_yago, 2, Merge into TEWS_MS_TYPE_BASKET < 0 - NOT OK ,
PKGEWS_PANEL_EXTRACT, p_signed_rank_yago, 2, Error in p_signed_rank_yago -
NOT OK
Error >>>>> : null
Exception in thread "main" java.lang.Exception
at
com.acnielsen.madras.pkgews_panel_extract$.p_main(pkgews_panel_extract.scala:4875)
at
com.acnielsen.madras.pkgews_panel_extract$.main(pkgews_panel_extract.scala:4655)
at
com.acnielsen.madras.pkgews_panel_extract.main(pkgews_panel_extract.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/04 05:11:25 INFO TaskSetManager: Finished task 18602.0 in stage 148.0
(TID 153115) in 207526 ms on lhrrhegapq006.enterprisenet.org (18598/32768)
16/08/04 05:11:27 WARN QueuedThreadPool: 6 threads could not be stopped
16/08/04 05:11:27 INFO TaskSetManager: Finished task 18597.0 in stage 148.0
(TID 153113) in 208704 ms on lhrrhegapq008.enterprisenet.org (18599/32768)
16/08/04 05:11:27 INFO DiskBlockManager: Shutdown hook called
16/08/04 05:11:27 WARN AkkaRpcEndpointRef: Error sending message [message =
StopMapOutputTracker] in 1 attempts
org.apache.spark.rpc.RpcTimeoutException:
Recipient[Actor[akka://sparkDriver/user/MapOutputTracker#1304693884]] had
already been terminated.. This timeout is controlled by spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org
$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214)
at
org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229)
at
org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)
at scala.util.Try$.apply(Try.scala:161)
at scala.util.Failure.recover(Try.scala:185)
at
scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at
scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
at
scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)
at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at
scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280)
at
scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270)
at scala.concurrent.Future$class.recover(Future.scala:324)
at
scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153)
at
org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319)
at
org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100)
at
org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
at
org.apache.spark.MapOutputTracker.askTracker(MapOutputTracker.scala:109)
at
org.apache.spark.MapOutputTracker.sendTracker(MapOutputTracker.scala:119)
at
org.apache.spark.MapOutputTrackerMaster.stop(MapOutputTracker.scala:382)
at org.apache.spark.SparkEnv.stop(SparkEnv.scala:97)
at
org.apache.spark.SparkContext$$anonfun$stop$12.apply$mcV$sp(SparkContext.scala:1749)
at
org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1184)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1748)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:146)
Caused by: akka.pattern.AskTimeoutException:
Recipient[Actor[akka://sparkDriver/user/MapOutputTracker#1304693884]] had
already been terminated.
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
at
org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307)
... 10 more
16/08/04 05:11:27 INFO TaskSetManager: Finished task 18604.0 in stage 148.0
(TID 153127) in 208703 ms on lhrrhegapq003.enterprisenet.org (18600/32768)
16/08/04 05:11:27 INFO ZooKeeper: Session: 0x15590c0c2202d11 closed
16/08/04 05:11:27 INFO TaskSetManager: Finished task 18598.0 in stage 148.0
(TID 153117) in 208707 ms on lhrrhegapq002.enterprisenet.org (18601/32768)
16/08/04 05:11:27 INFO ClientCnxn: EventThread shut down
16/08/04 05:11:27 INFO CuratorFrameworkSingleton: Closing ZooKeeper client.
16/08/04 05:11:27 INFO TaskSetManager: Finished task 18599.0 in stage 148.0
(TID 153119) in 208708 ms on lhrrhegapq004.enterprisenet.org (18602/32768)
16/08/04 05:11:27 INFO ShutdownHookManager: Shutdown hook called
16/08/04 05:11:27 INFO ShutdownHookManager: Deleting directory
/tmp/spark-419c3d6c-a6d4-49c4-9f77-b79c5d3c8f37
16/08/04 05:11:27 INFO ShutdownHookManager: Deleting directory
/tmp/spark-579fc117-08a4-45a6-a2a1-2a6e90c1adcd/userFiles-f739d328-716d-4641-821f-a1b2af709182
16/08/04 05:11:27 INFO ShutdownHookManager: Deleting directory
/tmp/spark-579fc117-08a4-45a6-a2a1-2a6e90c1adcd
16/08/04 05:11:27 INFO TaskSetManager: Finished task 18608.0 in stage 148.0
(TID 153121) in 208715 ms on lhrrhegapq006.enterprisenet.org (18603/32768)
Best regards,
Vasu
