Re: Spark 1.5 Streaming and Kinesis

2015-10-19 Thread Phil Kallos
couple of > other Apache projects ;)). I will try to do it today. I keep you posted. > > Regards > JB > > On 10/16/2015 07:21 AM, Phil Kallos wrote: > >> JB, >> >> To clarify, you are able to run the Amazon Kinesis example provided in >> the spark examples dir?

Re: Spark 1.5 Streaming and Kinesis

2015-10-15 Thread Phil Kallos
:00 Jean-Baptiste Onofré : >> >>> Thanks for the update Phil. >>> >>> I'm preparing a environment to reproduce it. >>> >>> I keep you posted. >>> >>> Thanks again, >>> Regards >>> JB >>> >>>

Re: Spark 1.5 Streaming and Kinesis

2015-10-14 Thread Phil Kallos
Map[StreamBlockId, SequenceNumberRanges] > So, it doesn't look fully correct to me. > Let me investigate a bit this morning. > Regards > JB > On 10/15/2015 07:49 AM, Phil Kallos wrote: > We are trying to migrate from Spark1.4 to Spark1.5 for our Kinesis > streaming app

Spark 1.5 Streaming and Kinesis

2015-10-14 Thread Phil Kallos
Hi, We are trying to migrate from Spark1.4 to Spark1.5 for our Kinesis streaming applications, to take advantage of the new Kinesis checkpointing improvements in 1.5. However after upgrading, we are consistently seeing the following error: java.lang.ClassCastException: scala.collection.mutable.H

Spark Kinesis Checkpointing/Processing Delay

2015-08-10 Thread Phil Kallos
Hi! Sorry if this is a repost. I'm using Spark + Kinesis ASL to process and persist stream data to ElasticSearch. For the most part it works nicely. There is a subtle issue I'm running into about how failures are handled. For example's sake, let's say I am processing a Kinesis stream that produc

subscribe

2015-08-10 Thread Phil Kallos
please