Re: Breaking lineage and reducing stages in Spark Streaming

hat are in myRDD, so >>>>> the reduceByKey after union keeps the overall tuple count in myRDD >>>>> fixed. Or even with fixed tuple count, it will keep consuming more >>>>> resources? >>>>> >>>>> On 9 July 201

Re: Breaking lineage and reducing stages in Spark Streaming

batch. Obviously this is going to not last for very long. You >>>>> fundamentally cannot keep processing ever increasing amount of data with >>>>> finite resources, isnt it? >>>>> >>>>> On Thu, Jul 9, 2015 at 3:17 A

Re: Breaking lineage and reducing stages in Spark Streaming

;> wrote: >>>> >>>>> Thats from the Streaming tab for Spark 1.4 WebUI. >>>>> >>>>> On 9 July 2015 at 15:35, Michel Hubert wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> >>>>>

Re: Breaking lineage and reducing stages in Spark Streaming

;>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>> I was just wondering how you generated to second image with the charts. >>>>> >>>>> What product? >>>>> >>>>> &

Re: Breaking lineage and reducing stages in Spark Streaming

the Streaming tab for Spark 1.4 WebUI. >>> >>> On 9 July 2015 at 15:35, Michel Hubert wrote: >>> >>>> Hi, >>>> >>>> >>>> >>>> I was just wondering how you generated to second image with the charts. >>>>

Re: Breaking lineage and reducing stages in Spark Streaming

ow you generated to second image with the charts. >>> >>> What product? >>> >>> >>> >>> *From:* Anand Nalya [mailto:anand.na...@gmail.com] >>> *Sent:* donderdag 9 juli 2015 11:48 >>> *To:* spark users >>> *Subject:* Breaking

Re: Breaking lineage and reducing stages in Spark Streaming

image with the charts. >> >> What product? >> >> >> >> *From:* Anand Nalya [mailto:anand.na...@gmail.com] >> *Sent:* donderdag 9 juli 2015 11:48 >> *To:* spark users >> *Subject:* Breaking lineage and reducing stages in Spark Streaming >> >> >

Re: Breaking lineage and reducing stages in Spark Streaming

gt; *Sent:* donderdag 9 juli 2015 11:48 > *To:* spark users > *Subject:* Breaking lineage and reducing stages in Spark Streaming > > > > Hi, > > > > I've an application in which an rdd is being updated with tuples coming > from RDDs in a DStream with follow

RE: Breaking lineage and reducing stages in Spark Streaming

Hi, I was just wondering how you generated to second image with the charts. What product? From: Anand Nalya [mailto:anand.na...@gmail.com] Sent: donderdag 9 juli 2015 11:48 To: spark users Subject: Breaking lineage and reducing stages in Spark Streaming Hi, I've an application in which a

Breaking lineage and reducing stages in Spark Streaming

Hi, I've an application in which an rdd is being updated with tuples coming from RDDs in a DStream with following pattern. dstream.foreachRDD(rdd => { myRDD = myRDD.union(rdd.filter(myfilter)).reduceByKey(_+_) }) I'm using cache() and checkpointin to cache results. Over the time, the lineage o