Any insights on this ? On Fri, Feb 26, 2016 at 1:21 PM, Abhishek Anand <abhis.anan...@gmail.com> wrote:
> On changing the default compression codec which is snappy to lzf the > errors are gone !! > > How can I fix this using snappy as the codec ? > > Is there any downside of using lzf as snappy is the default codec that > ships with spark. > > > Thanks !!! > Abhi > > On Mon, Feb 22, 2016 at 7:42 PM, Abhishek Anand <abhis.anan...@gmail.com> > wrote: > >> Hi , >> >> I am getting the following exception on running my spark streaming job. >> >> The same job has been running fine since long and when I added two new >> machines to my cluster I see the job failing with the following exception. >> >> >> >> 16/02/22 19:23:01 ERROR Executor: Exception in task 2.0 in stage 4229.0 >> (TID 22594) >> java.io.IOException: java.lang.reflect.InvocationTargetException >> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1257) >> at >> org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165) >> at >> org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64) >> at >> org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64) >> at >> org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88) >> at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:59) >> at org.apache.spark.scheduler.Task.run(Task.scala:70) >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >> at >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:408) >> at >> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:68) >> at >> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60) >> at org.apache.spark.broadcast.TorrentBroadcast.org >> $apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) >> at >> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167) >> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1254) >> ... 11 more >> Caused by: java.lang.IllegalArgumentException >> at >> org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:152) >> ... 20 more >> >> >> >> Thanks !!! >> Abhi >> > >
