Yes, I use YARN. I could run the job an delete the Kerberos credentials and check if it's because there aren't credetianls. I don't know if it's the same destroy the credential that the credentials expire. But I can try that.
2015-07-31 7:09 GMT+02:00 Tathagata Das <[email protected]>: > Are you running it on YARN? Might be a known YARN credential expiring > issue. > > Hari, any insights? > > On Thu, Jul 30, 2015 at 4:04 AM, Guillermo Ortiz <[email protected]> > wrote: > >> I'm executing a job with Spark Streaming and got this error all times >> when the job has been executing for a while (usually hours of days). >> >> I have no idea why it's happening. >> >> 15/07/30 13:02:14 ERROR LiveListenerBus: Listener EventLoggingListener >> threw an exception >> java.lang.reflect.InvocationTargetException >> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) >> at >> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) >> at scala.Option.foreach(Option.scala:236) >> at >> org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144) >> at >> org.apache.spark.scheduler.EventLoggingListener.onJobEnd(EventLoggingListener.scala:169) >> at >> org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:36) >> at >> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) >> at >> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) >> at >> org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53) >> at >> org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36) >> at >> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76) >> at >> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61) >> at >> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61) >> at >> org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617) >> at >> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60) >> Caused by: java.io.IOException: Lease timeout of 0 seconds expired. >> at >> org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:2192) >> at >> org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:935) >> at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:889) >> at >> org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) >> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) >> at >> org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) >> at >> org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) >> at java.lang.Thread.run(Thread.java:745) >> 15/07/30 13:02:14 INFO SparkContext: Starting job: foreachRDD at >> MetricsSpark.scala:67 >> 15/07/30 13:02:14 INFO DAGScheduler: Got job 5050 (foreachRDD at >> MetricsSpark.scala:67) with 3 output partitions (allowLocal=false) >> 15/07/30 13:02:14 INFO DAGScheduler: Final stage: Stage 5050(foreachRDD >> at MetricsSpark.scala:67) >> >> Sometimes this error happens, but it doesn't mean that Spark stops >> working forever. Because it looks like after this error Spark works >> correctly some iterations, but most of time just fails after this error >> producing this error all the time. >> >> The code just filters some records and index the record in >> ElasticSearch.after adding some new fields. >> > >
