This is because the Snappy library cannot load the native library. Did you forget to install the snappy native library in your new machines?
On Fri, Feb 26, 2016 at 11:05 PM, Abhishek Anand <abhis.anan...@gmail.com> wrote: > Any insights on this ? > > On Fri, Feb 26, 2016 at 1:21 PM, Abhishek Anand <abhis.anan...@gmail.com> > wrote: > >> On changing the default compression codec which is snappy to lzf the >> errors are gone !! >> >> How can I fix this using snappy as the codec ? >> >> Is there any downside of using lzf as snappy is the default codec that >> ships with spark. >> >> >> Thanks !!! >> Abhi >> >> On Mon, Feb 22, 2016 at 7:42 PM, Abhishek Anand <abhis.anan...@gmail.com> >> wrote: >> >>> Hi , >>> >>> I am getting the following exception on running my spark streaming job. >>> >>> The same job has been running fine since long and when I added two new >>> machines to my cluster I see the job failing with the following exception. >>> >>> >>> >>> 16/02/22 19:23:01 ERROR Executor: Exception in task 2.0 in stage 4229.0 >>> (TID 22594) >>> java.io.IOException: java.lang.reflect.InvocationTargetException >>> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1257) >>> at >>> org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165) >>> at >>> org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64) >>> at >>> org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64) >>> at >>> org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88) >>> at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) >>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:59) >>> at org.apache.spark.scheduler.Task.run(Task.scala:70) >>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:744) >>> Caused by: java.lang.reflect.InvocationTargetException >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >>> at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:408) >>> at >>> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:68) >>> at >>> org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60) >>> at org.apache.spark.broadcast.TorrentBroadcast.org >>> $apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) >>> at >>> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167) >>> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1254) >>> ... 11 more >>> Caused by: java.lang.IllegalArgumentException >>> at >>> org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:152) >>> ... 20 more >>> >>> >>> >>> Thanks !!! >>> Abhi >>> >> >> >
