Hi Yash, I've run into multiple problems due to version incompatibilities, either due to protobuf or jackson. That may be your culprit. The problem is that all failures by the Kinesis Client Lib is silent, therefore don't show up on the logs. It's very hard to debug those buggers.
Best, Burak On Sat, Jan 30, 2016 at 5:36 AM, Yash Sharma <yash...@gmail.com> wrote: > Thanks Ted, Rebuilding would not be possible for the setup unfortunately > so just wanted to check if the version mismatch is the primary issue here. > Wanted to know if anyone has hit across similar issue and how they have > solved this. > > Thanks > > On Sat, Jan 30, 2016 at 10:23 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> w.r.t. protobuf-java version mismatch, I wonder if you can rebuild Spark >> with the following change (using maven): >> >> http://pastebin.com/fVQAYWHM >> >> Cheers >> >> On Sat, Jan 30, 2016 at 12:49 AM, Yash Sharma <yash...@gmail.com> wrote: >> >>> Hi All, >>> I have a quick question if anyone has experienced this here. >>> >>> I have been trying to get Spark read events from Kinesis recently but am >>> having problem in receiving the events. While Spark is able to connect to >>> Kinesis and is able to get metadata from Kinesis, Its not able to get >>> events from it. It always fetches zero elements back. >>> >>> There are no errors, just empty results back. Spark is able to get >>> metadata (Eg. number of shards in kinesis etc). >>> >>> I have used these [1 & 2] guides for getting it working but have not got >>> much luck yet. I have also tried couple of suggestions from SO [3]. The >>> cluster has sufficient resources/cores available. >>> >>> We have seen a version conflict in Protobuf Version between Spark and >>> Kinesis which could also be a cause for this behavior. Spark uses >>> protobuf-java >>> version 2.5.0 and kinesis probably uses protobuf-java-2.6.1.jar. >>> >>> Just wondered if anyone has come across this behavior or, has got spark >>> working with kinesis. >>> >>> Have tried with Spark 1.5.0, Spark 1.6.0. >>> >>> Appreciate any pointers. >>> >>> Best Regards, >>> Yash >>> >>> 1. >>> http://spark.apache.org/docs/latest/streaming-kinesis-integration.html >>> 2. >>> https://github.com/apache/spark/blob/master/extras/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala >>> >>> 3. >>> http://stackoverflow.com/questions/26941844/apache-spark-kinesis-sample-not-working >>> >>> >>> >> >
