the discussion (re-loading batch RDD from file within spark
> steraming context) - lets leave that since we are not getting anywhere
>
> -Original Message-
> From: Sean Owen [mailto:so...@cloudera.com]
> Sent: Wednesday, April 15, 2015 8:30 PM
> To: Evo Eftimov
> Cc: user@
e not getting anywhere
-Original Message-
From: Sean Owen [mailto:so...@cloudera.com]
Sent: Wednesday, April 15, 2015 8:30 PM
To: Evo Eftimov
Cc: user@spark.apache.org
Subject: Re: adding new elements to batch RDD from DStream RDD
What API differences are you talking about? a DStream gives
only way to keep reloading a batch RDD from file is from a
>> separate thread still using the same spark context. The thread will
>> reload the batch RDD with the same reference ie reassign the reference
>> to the newly instantiated/loaded batch RDD - is that what you mean by
&
h RDDs from file for e.g. a second
time moreover after specific period of time
-Original Message-
From: Sean Owen [mailto:so...@cloudera.com]
Sent: Wednesday, April 15, 2015 8:14 PM
To: Evo Eftimov
Cc: user@spark.apache.org
Subject: Re: adding new elements to batch RDD from DStream RDD
loading batch RDD from file
>
> -Original Message-
> From: Sean Owen [mailto:so...@cloudera.com]
> Sent: Wednesday, April 15, 2015 7:43 PM
> To: Evo Eftimov
> Cc: user@spark.apache.org
> Subject: Re: adding new elements to batch RDD from DStream RDD
>
> What do you m
iginal Message-
From: Sean Owen [mailto:so...@cloudera.com]
Sent: Wednesday, April 15, 2015 7:43 PM
To: Evo Eftimov
Cc: user@spark.apache.org
Subject: Re: adding new elements to batch RDD from DStream RDD
What do you mean by "batch RDD"? they're just RDDs, though store their d
What do you mean by "batch RDD"? they're just RDDs, though store their
data in different ways and come from different sources. You can union
an RDD from an HDFS file with one from a DStream.
It sounds like you want streaming data to live longer than its batch
interval, but that's not something you
