Re: reduceByKeyAndWindow, but using log timestamps instead of clock seconds

Hi, On Thu, Jan 29, 2015 at 9:52 AM, Tobias Pfeiffer wrote: > Hi, > > On Thu, Jan 29, 2015 at 1:54 AM, YaoPau wrote: >> >> My thinking is to maintain state in an RDD and update it an persist it >> with >> each 2-second pass, but this also seems like it could get messy. Any >> thoughts or examp

Re: reduceByKeyAndWindow, but using log timestamps instead of clock seconds

Has there been any follow up on this topic? Here there were suggestions that someone was going to publish some code, but no news since (TD himself looked pretty interested). Did anybody come up with something in the last months? -- View this mes

Re: reduceByKeyAndWindow, but using log timestamps instead of clock seconds

inal Message- > From: Tathagata Das [mailto:tathagata.das1...@gmail.com] > Sent: Thursday, January 29, 2015 10:33 AM > To: Tobias Pfeiffer > Cc: YaoPau; user > Subject: Re: reduceByKeyAndWindow, but using log timestamps instead of > clock seconds > > Ohhh nice! Would be great if you

RE: reduceByKeyAndWindow, but using log timestamps instead of clock seconds

2015 10:33 AM To: Tobias Pfeiffer Cc: YaoPau; user Subject: Re: reduceByKeyAndWindow, but using log timestamps instead of clock seconds Ohhh nice! Would be great if you can share us some code soon. It is indeed a very complicated problem and there is probably no single solution that fits all us

Re: reduceByKeyAndWindow, but using log timestamps instead of clock seconds

Ohhh nice! Would be great if you can share us some code soon. It is indeed a very complicated problem and there is probably no single solution that fits all usecases. So having one way of doing things would be a great reference. Looking forward to that! On Wed, Jan 28, 2015 at 4:52 PM, Tobias Pfei

Re: reduceByKeyAndWindow, but using log timestamps instead of clock seconds

Hi, On Thu, Jan 29, 2015 at 1:54 AM, YaoPau wrote: > > My thinking is to maintain state in an RDD and update it an persist it with > each 2-second pass, but this also seems like it could get messy. Any > thoughts or examples that might help me? > I have just implemented some timestamp-based win

Re: reduceByKeyAndWindow, but using log timestamps instead of clock seconds

I'm not an expert on streaming, but I think you can't do anything like this right now. It seems like a very sensible use case, though, so I've created a jira for it: https://issues.apache.org/jira/browse/SPARK-5467 On Wed, Jan 28, 2015 at 8:54 AM, YaoPau wrote: > The TwitterPopularTags example

Re: reduceByKeyAndWindow, but using log timestamps instead of clock seconds

I'm not quiet sure if i understood it correctly, but can you not create a key from the timestamps and do the reduceByKeyAndWindow over it? Thanks Best Regards On Wed, Jan 28, 2015 at 10:24 PM, YaoPau wrote: > The TwitterPopularTags example works great: the Twitter firehose keeps > messages pret