Re: Spark 1.5 Streaming and Kinesis

2015-10-20 Thread Jean-Baptiste Onofré
<mailto:user@spark.apache.org>> <mailto:user@spark.apache.org <mailto:user@spark.apache.org> <mailto:user@spark.apache.org <mailto:user@spark.apache.org>&g

Re: Spark 1.5 Streaming and Kinesis

2015-10-20 Thread Jean-Baptiste Onofré
he.org <mailto:user@spark.apache.org>> <mailto:user@spark.apache.org <mailto:user@spark.apache.org> <mailto:user@spark.apache.org <mailto:user@spark.apache.org>>>

Re: Spark 1.5 Streaming and Kinesis

2015-10-19 Thread Jean-Baptiste Onofré
pache.org>> <mailto:user@spark.apache.org <mailto:user@spark.apache.org> <mailto:user@spark.apache.org <mailto:user@spark.apache.org>>> Cc: Date: Thu,

Re: Spark 1.5 Streaming and Kinesis

2015-10-19 Thread Phil Kallos
omes in >> spark/examples >> >> Tried running the streaming job locally, and also in EMR >> with release >> 4.1.0 that includes Spark 1.5 >> >> Very strange! >> >>

Re: Spark 1.5 Streaming and Kinesis

2015-10-15 Thread Jean-Baptiste Onofré
<mailto:j...@nanthrax.net>>> To: user@spark.apache.org <mailto:user@spark.apache.org> <mailto:user@spark.apache.org <mailto:user@spark.apache.org>> Cc:

Re: Spark 1.5 Streaming and Kinesis

2015-10-15 Thread Phil Kallos
;>> "org.apache.spark" %% "spark-streaming-kinesis-asl" % Versions.spark % >>>> "provided" >>>> >>>> I am experiencing the issue in my own spark project, but also when I try >>>> to run the spark streaming kinesis example th

Re: Spark 1.5 Streaming and Kinesis

2015-10-15 Thread Eugen Cepoi
and also in EMR with release >>> 4.1.0 that includes Spark 1.5 >>> >>> Very strange! >>> >>> -- Forwarded message -- >>> >>> From: "Jean-Baptiste Onofré" >> j...@nanthrax.net>> >>>

Re: Spark 1.5 Streaming and Kinesis

2015-10-15 Thread Eugen Cepoi
ith release >> 4.1.0 that includes Spark 1.5 >> >> Very strange! >> >> -- Forwarded message -- >> >> From: "Jean-Baptiste Onofré" mailto:j...@nanthrax.net >> >> >> To: user@spark.apache.org <ma

Re: Spark 1.5 Streaming and Kinesis

2015-10-14 Thread Jean-Baptiste Onofré
; Cc: Date: Thu, 15 Oct 2015 08:03:55 +0200 Subject: Re: Spark 1.5 Streaming and Kinesis Hi Phil, KinesisReceiver is part of extra. Just a dumb question: did you update all, including the Spark Kinesis extra containing the KinesisReceiver ? I ch

Re: Spark 1.5 Streaming and Kinesis

2015-10-14 Thread Phil Kallos
an-Baptiste Onofré" > To: user@spark.apache.org > Cc: > Date: Thu, 15 Oct 2015 08:03:55 +0200 > Subject: Re: Spark 1.5 Streaming and Kinesis > Hi Phil, > KinesisReceiver is part of extra. Just a dumb question: did you update > all, including the Spark Kinesis extra cont

Re: Spark 1.5 Streaming and Kinesis

2015-10-14 Thread Jean-Baptiste Onofré
By correct, I mean: the map declaration looks good to me, so the ClassCastException is weird ;) I'm trying to reproduce the issue in order to investigate. Regards JB On 10/15/2015 08:03 AM, Jean-Baptiste Onofré wrote: Hi Phil, KinesisReceiver is part of extra. Just a dumb question: did you u

Re: Spark 1.5 Streaming and Kinesis

2015-10-14 Thread Jean-Baptiste Onofré
Hi Phil, KinesisReceiver is part of extra. Just a dumb question: did you update all, including the Spark Kinesis extra containing the KinesisReceiver ? I checked on tag v1.5.0, and at line 175 of the KinesisReceiver, we see: blockIdToSeqNumRanges.clear() which is a: private val blockIdToSeq