Is it a python app?
On Mon, Aug 8, 2016 at 2:44 PM, Hemalatha A <
hemalatha.amru...@googlemail.com> wrote:
> Hello,
>
> I am seeing multiple exceptions shown in logs during an action, but none
> of them fails the Spark streaming batch in yarn-client mode, whereas the
> same exception is thrown in Yarn-cluster mode and the application ends.
>
> I am trying to save a Dataframe To cassandra, which results in error due
> to wrong password lets say. The job goes to failed state throwing the below
> exception in Jobs tab in Spark UI but in the streaming tab, the
> corresponding batch remains in active state forever.It doesn't fail the
> streaming batch in yarn-client mode.. Whereas, the same works fine in
> Yarn-cluster mode, it throws the same error and ends the application.
>
> Why is this difference in behaviour in the 2 modes? Why does yarn-client
> mode behaves in this way?
>
> *Exception seen in both modes:*
>
> 16/08/04 08:04:43 ERROR org.apache.spark.streaming.scheduler.JobScheduler:
> Error running job streaming job 1470297880000 ms.0
> java.io.IOException: Failed to open native connection to Cassandra at
> {172.x.x.x}:9042
> at
> com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:162)
> at
> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
> at
> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
> at
> com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31)
> at
> com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56)
> at
> com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81)
> at
> com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:109)
> at
> com.datastax.spark.connector.rdd.partitioner.CassandraRDDPartitioner$.getTokenFactory(CassandraRDDPartitioner.scala:184)
> at
> org.apache.spark.sql.cassandra.CassandraSourceRelation$.apply(CassandraSourceRelation.scala:267)
> at
> org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.scala:84)
> at
> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:170)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:146)
> at
> boccstreamingall$$anonfun$process_kv_text_stream$1.apply(bocc_spark_all.scala:249)
> at
> boccstreamingall$$anonfun$process_kv_text_stream$1.apply(bocc_spark_all.scala:233)
> at
> org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:631)
> at
> org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:631)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:42)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:40)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:40)
> at
> org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:399)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:40)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
> at scala.util.Try$.apply(Try.scala:161)
> at org.apache.spark.streaming.scheduler.Job.run(Job.scala:34)
> at
> org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:207)
> at
> org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:207)
> at
> org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:207)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>
> at
> org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:206)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.datastax.driver.core.exceptions.AuthenticationException:
> Authentication error on host /172.x.x.x:9042: Username and/or password are
> incorrect
> at com.datastax.driver.core.Connection$8.apply(Connection.java:376)
> at com.datastax.driver.core.Connection$8.apply(Connection.java:346)
> at
> shadeio.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861)
> at
> shadeio.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> at
> shadeio.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> shadeio.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> shadeio.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)
> at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1174)
> at
> com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1005)
> at
> com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:928)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> at
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1084)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:959)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:327)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:230)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
> at
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:831)
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:346)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>
> --
>
>
> Regards
> Hemalatha
>
--
Best Regards,
Ayan Guha
