I got my answer.
The way to access S3 has changed.
val hadoopConf = sc.hadoopConfiguration
hadoopConf.set("fs.s3a.access.key", accessKey)
hadoopConf.set("fs.s3a.secret.key", secretKey)
val lines = ssc.textFileStream("s3a://amg-events-out/")
This worked.
Cheers,
Ben
> On May 21, 2016, at 4:18
Ted,
I only see 1 jets3t-0.9.0 jar in the classpath after running this to list the
jars.
val cl = ClassLoader.getSystemClassLoader
cl.asInstanceOf[java.net.URLClassLoader].getURLs.foreach(println)
/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/jars/jets3t-0.9.0.jar
I don’t know what else cou
Maybe more than one version of jets3t-xx.jar was on the classpath.
FYI
On Fri, May 20, 2016 at 8:31 PM, Benjamin Kim wrote:
> I am trying to stream files from an S3 bucket using CDH 5.7.0’s version of
> Spark 1.6.0. It seems not to work. I keep getting this error.
>
> Exception in thread "JobGe
I am trying to stream files from an S3 bucket using CDH 5.7.0’s version of
Spark 1.6.0. It seems not to work. I keep getting this error.
Exception in thread "JobGenerator" java.lang.VerifyError: Bad type on operand
stack
Exception Details:
Location:
org/apache/hadoop/fs/s3native/Jets3tNat
