Hi All, I have a quick question if anyone has experienced this here. I have been trying to get Spark read events from Kinesis recently but am having problem in receiving the events. While Spark is able to connect to Kinesis and is able to get metadata from Kinesis, Its not able to get events from it. It always fetches zero elements back.
There are no errors, just empty results back. Spark is able to get metadata (Eg. number of shards in kinesis etc). I have used these [1 & 2] guides for getting it working but have not got much luck yet. I have also tried couple of suggestions from SO [3]. The cluster has sufficient resources/cores available. We have seen a version conflict in Protobuf Version between Spark and Kinesis which could also be a cause for this behavior. Spark uses protobuf-java version 2.5.0 and kinesis probably uses protobuf-java-2.6.1.jar. Just wondered if anyone has come across this behavior or, has got spark working with kinesis. Have tried with Spark 1.5.0, Spark 1.6.0. Appreciate any pointers. Best Regards, Yash 1. http://spark.apache.org/docs/latest/streaming-kinesis-integration.html 2. https://github.com/apache/spark/blob/master/extras/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala 3. http://stackoverflow.com/questions/26941844/apache-spark-kinesis-sample-not-working
