Re: Spark 1.5 Streaming and Kinesis

2015-10-20 Thread Jean-Baptiste Onofré
<mailto:user@spark.apache.org>> <mailto:user@spark.apache.org <mailto:user@spark.apache.org> <mailto:user@spark.apache.org <mailto:user@spark.apache.org>&g

Re: Spark 1.5 Streaming and Kinesis

2015-10-20 Thread Jean-Baptiste Onofré
he.org <mailto:user@spark.apache.org>> <mailto:user@spark.apache.org <mailto:user@spark.apache.org> <mailto:user@spark.apache.org <mailto:user@spark.apache.org>>>

Re: Spark 1.5 Streaming and Kinesis

2015-10-19 Thread Jean-Baptiste Onofré
pache.org>> <mailto:user@spark.apache.org <mailto:user@spark.apache.org> <mailto:user@spark.apache.org <mailto:user@spark.apache.org>>> Cc: Date: Thu,

Re: Spark 1.5 Streaming and Kinesis

2015-10-19 Thread Phil Kallos
omes in >> spark/examples >> >> Tried running the streaming job locally, and also in EMR >> with release >> 4.1.0 that includes Spark 1.5 >> >> Very strange! >> >>

Re: Spark 1.5 Streaming and Kinesis

2015-10-15 Thread Jean-Baptiste Onofré
<mailto:j...@nanthrax.net>>> To: user@spark.apache.org <mailto:user@spark.apache.org> <mailto:user@spark.apache.org <mailto:user@spark.apache.org>> Cc:

Re: Spark 1.5 Streaming and Kinesis

2015-10-15 Thread Phil Kallos
at comes in spark/examples >>>> >>>> Tried running the streaming job locally, and also in EMR with release >>>> 4.1.0 that includes Spark 1.5 >>>> >>>> Very strange! >>>> >>>> -- Forwarded message ---

Re: Spark 1.5 Streaming and Kinesis

2015-10-15 Thread Eugen Cepoi
and also in EMR with release >>> 4.1.0 that includes Spark 1.5 >>> >>> Very strange! >>> >>> -- Forwarded message -- >>> >>> From: "Jean-Baptiste Onofré" >> j...@nanthrax.net>> >>>

Re: Spark 1.5 Streaming and Kinesis

2015-10-15 Thread Eugen Cepoi
ith release >> 4.1.0 that includes Spark 1.5 >> >> Very strange! >> >> -- Forwarded message -- >> >> From: "Jean-Baptiste Onofré" mailto:j...@nanthrax.net >> >> >> To: user@spark.apache.org <ma

Re: Spark 1.5 Streaming and Kinesis

2015-10-14 Thread Jean-Baptiste Onofré
; Cc: Date: Thu, 15 Oct 2015 08:03:55 +0200 Subject: Re: Spark 1.5 Streaming and Kinesis Hi Phil, KinesisReceiver is part of extra. Just a dumb question: did you update all, including the Spark Kinesis extra containing the KinesisReceiver ? I ch

Re: Spark 1.5 Streaming and Kinesis

2015-10-14 Thread Phil Kallos
an-Baptiste Onofré" > To: user@spark.apache.org > Cc: > Date: Thu, 15 Oct 2015 08:03:55 +0200 > Subject: Re: Spark 1.5 Streaming and Kinesis > Hi Phil, > KinesisReceiver is part of extra. Just a dumb question: did you update > all, including the Spark Kinesis extra cont

Re: Spark 1.5 Streaming and Kinesis

2015-10-14 Thread Jean-Baptiste Onofré
By correct, I mean: the map declaration looks good to me, so the ClassCastException is weird ;) I'm trying to reproduce the issue in order to investigate. Regards JB On 10/15/2015 08:03 AM, Jean-Baptiste Onofré wrote: Hi Phil, KinesisReceiver is part of extra. Just a dumb question: did you u

Re: Spark 1.5 Streaming and Kinesis

2015-10-14 Thread Jean-Baptiste Onofré
Hi Phil, KinesisReceiver is part of extra. Just a dumb question: did you update all, including the Spark Kinesis extra containing the KinesisReceiver ? I checked on tag v1.5.0, and at line 175 of the KinesisReceiver, we see: blockIdToSeqNumRanges.clear() which is a: private val blockIdToSeq

Spark 1.5 Streaming and Kinesis

2015-10-14 Thread Phil Kallos
Hi, We are trying to migrate from Spark1.4 to Spark1.5 for our Kinesis streaming applications, to take advantage of the new Kinesis checkpointing improvements in 1.5. However after upgrading, we are consistently seeing the following error: java.lang.ClassCastException: scala.collection.mutable.H