Github user darose commented on the pull request:
https://github.com/apache/spark/pull/468#issuecomment-42027309
Man oh man, I cannot get this to work no way no how. I tried rebuilding
spark using the jets3t 0.9 jar, then tried rebuilding shark doing the same. I
keep getting a verify error - presumably because something in the call stack
isn't compatible with the new jets3t version. Anyone have any
ideas/suggestions? I'm at my wits' end on this. Spent days, and still unable
to get a working version of spark/shark running with CDH5. Output below.
```
14/05/02 06:34:14 WARN scheduler.TaskSetManager: Loss was due to
java.lang.VerifyError
java.lang.VerifyError: Bad type on operand stack
Exception Details:
Location:
org/apache/hadoop/fs/s3native/Jets3tNativeFileSystemStore.initialize(Ljava/net/URI;Lorg/apache/hadoop/conf/Configuration;)V
@38: invokespecial
Reason:
Type 'org/jets3t/service/security/AWSCredentials' (current frame,
stack[3]) is not assignable to 'org/jets3t/service/security/ProviderCredentials
'
Current Frame:
bci: @38
flags: { }
locals: { 'org/apache/hadoop/fs/s3native/Jets3tNativeFileSystemStore',
'java/net/URI', 'org/apache/hadoop/conf/Configuration', 'org/apache/hadoop
/fs/s3/S3Credentials', 'org/jets3t/service/security/AWSCredentials' }
stack: { 'org/apache/hadoop/fs/s3native/Jets3tNativeFileSystemStore',
uninitialized 32, uninitialized 32, 'org/jets3t/service/security/AWSCredent
ials' }
Bytecode:
0000000: bb00 0259 b700 034e 2d2b 2cb6 0004 bb00
0000010: 0559 2db6 0006 2db6 0007 b700 083a 042a
0000020: bb00 0959 1904 b700 0ab5 000b a700 0b3a
0000030: 042a 1904 b700 0d2a 2c12 0e03 b600 0fb5
0000040: 0010 2a2c 1211 1400 12b6 0014 1400 15b8
0000050: 0017 b500 182a 2c12 1914 0015 b600 1414
0000060: 0015 b800 17b5 001a 2abb 001b 592b b600
0000070: 1cb7 001d b500 1eb1
Exception Handler Table:
bci [14, 44] => handler: 47
Stackmap Table:
[344/1956]
full_frame(@47,{Object[#176],Object[#177],Object[#178],Object[#179]},{Object[#180]})
same_frame(@55)
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.createDefaultStore(NativeS3FileSystem.java:280)
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:270)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at
org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:107)
at
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at
org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:156)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:149)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:64)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:34)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:34)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:34)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
at org.apache.spark.scheduler.Task.run(Task.scala:53)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
