I am not familiar with CDH distribution, we built spark ourselves.
The error means running code generated with Protocol-Buffers 2.5.0 with a
protocol-buffers-2.4.1 (or earlier) jar.
So there is a protocol-buffer 2.4.1 version somewhere, either in the jar you
built, or in the cluster runtime.
This shows a trick to identify which jar file the class is loaded from:
http://stackoverflow.com/questions/1983839/determine-which-jar-file-a-class-is-from
You may want to add the log in the first line of your code to check class
"com.google.protobuf.GeneratedMessage" to see which jar file it is loaded from,
and verify if it is in 2.5 version or below.
Yong
Date: Tue, 25 Aug 2015 13:44:17 -0700
Subject: Re: Protobuf error when streaming from Kafka
From: lcas...@gmail.com
To: java8...@hotmail.com
CC: yuzhih...@gmail.com; user@spark.apache.org
Do you think this binary would have issue? Do I need to build spark from source
code?
On Tue, Aug 25, 2015 at 1:06 PM, Cassa L <lcas...@gmail.com> wrote:
I downloaded below binary version of spark.
spark-1.4.1-bin-cdh4
On Tue, Aug 25, 2015 at 1:03 PM, java8964 <java8...@hotmail.com> wrote:
Did your spark build with Hive?
I met the same problem before because the hive-exec jar in the maven itself
include "protobuf" class, which will be included in the Spark jar.
Yong
Date: Tue, 25 Aug 2015 12:39:46 -0700
Subject: Re: Protobuf error when streaming from Kafka
From: lcas...@gmail.com
To: yuzhih...@gmail.com
CC: user@spark.apache.org
Hi,
I am using Spark-1.4 and Kafka-0.8.2.1
As per google suggestions, I rebuilt all the classes with protobuff-2.5
dependencies. My new protobuf is compiled using 2.5. However now, my spark job
does not start. Its throwing different error. Does Spark or any other its
dependencies uses old protobuff-2.4?
Exception in thread "main" java.lang.VerifyError: class
com.apple.ist.retail.xcardmq.serializers.SampleProtobufMessage$ProtoBuff
overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
com.apple.ist.retail.spark.kafka.consumer.SparkMQProcessor.start(SparkProcessor.java:68)
at
com.apple.ist.retail.spark.kafka.consumer.SparkMQConsumer.main(SparkConsumer.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
On Mon, Aug 24, 2015 at 6:53 PM, Ted Yu <yuzhih...@gmail.com> wrote:
Can you show the complete stack trace ?
Which Spark / Kafka release are you using ?
Thanks
On Mon, Aug 24, 2015 at 4:58 PM, Cassa L <lcas...@gmail.com> wrote:
Hi,
I am storing messages in Kafka using protobuf and reading them into Spark. I
upgraded protobuf version from 2.4.1 to 2.5.0. I got
"java.lang.UnsupportedOperationException" for older messages. However, even for
new messages I get the same error. Spark does convert it though. I see my
messages. How do I get rid of this error?
java.lang.UnsupportedOperationException: This is supposed to be overridden by
subclasses.
at
com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
at
org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$FsPermissionProto.getSerializedSize(HdfsProtos.java:5407)
at
com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:749)
