There's no need to extend Spark's API, look at mapWithState for examples. On Thu, Dec 8, 2016 at 4:49 AM, Anty Rao <ant....@gmail.com> wrote:
> > > On Wed, Dec 7, 2016 at 7:42 PM, Anty Rao <ant....@gmail.com> wrote: > >> Hi >> I'm new to Spark. I'm doing some research to see if spark streaming can >> solve my problem. I don't want to keep per-key state,b/c my data set is >> very huge and keep a little longer time, it not viable to keep all per key >> state in memory.Instead, i want to have a bloom filter based state. Does it >> possible to achieve this in Spark streaming. >> >> Is it possible to achieve this by extending Spark API? > >> -- >> Anty Rao >> > > > > -- > Anty Rao >
