w.r.t. protobuf-java version mismatch, I wonder if you can rebuild Spark with the following change (using maven):
http://pastebin.com/fVQAYWHM Cheers On Sat, Jan 30, 2016 at 12:49 AM, Yash Sharma <yash...@gmail.com> wrote: > Hi All, > I have a quick question if anyone has experienced this here. > > I have been trying to get Spark read events from Kinesis recently but am > having problem in receiving the events. While Spark is able to connect to > Kinesis and is able to get metadata from Kinesis, Its not able to get > events from it. It always fetches zero elements back. > > There are no errors, just empty results back. Spark is able to get > metadata (Eg. number of shards in kinesis etc). > > I have used these [1 & 2] guides for getting it working but have not got > much luck yet. I have also tried couple of suggestions from SO [3]. The > cluster has sufficient resources/cores available. > > We have seen a version conflict in Protobuf Version between Spark and > Kinesis which could also be a cause for this behavior. Spark uses > protobuf-java > version 2.5.0 and kinesis probably uses protobuf-java-2.6.1.jar. > > Just wondered if anyone has come across this behavior or, has got spark > working with kinesis. > > Have tried with Spark 1.5.0, Spark 1.6.0. > > Appreciate any pointers. > > Best Regards, > Yash > > 1. http://spark.apache.org/docs/latest/streaming-kinesis-integration.html > 2. > https://github.com/apache/spark/blob/master/extras/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala > > 3. > http://stackoverflow.com/questions/26941844/apache-spark-kinesis-sample-not-working > > >
