Answering my own Question -
I have got some success with Spark Kinesis integration, and the key being
the unionStreams.foreachRDD.
There are 2 versions of the foreachRDD available
- unionStreams.foreachRDD
- unionStreams.foreachRDD ((rdd: RDD[Array[Byte]], time: Time)
For some reason the first o
Thanks Burak,
By any chance were you able to work around these errors or get the setup
working ? Is there anything else that you might have tried ?
Regards
On Sun, Jan 31, 2016 at 4:41 AM, Burak Yavuz wrote:
> Hi Yash,
>
> I've run into multiple problems due to version incompatibilities, either
Hi Yash,
I've run into multiple problems due to version incompatibilities, either
due to protobuf or jackson. That may be your culprit. The problem is that
all failures by the Kinesis Client Lib is silent, therefore don't show up
on the logs. It's very hard to debug those buggers.
Best,
Burak
On
Thanks Ted, Rebuilding would not be possible for the setup unfortunately so
just wanted to check if the version mismatch is the primary issue here.
Wanted to know if anyone has hit across similar issue and how they have
solved this.
Thanks
On Sat, Jan 30, 2016 at 10:23 PM, Ted Yu wrote:
> w.r.t
w.r.t. protobuf-java version mismatch, I wonder if you can rebuild Spark
with the following change (using maven):
http://pastebin.com/fVQAYWHM
Cheers
On Sat, Jan 30, 2016 at 12:49 AM, Yash Sharma wrote:
> Hi All,
> I have a quick question if anyone has experienced this here.
>
> I have been tr
