Re: Watermarking in Structured Streaming to drop late data

2016-10-27 Thread Tathagata Das
mail.jtp?type=node&node=19600&i=0>] > *Sent:* Thursday, October 27, 2016 10:17 AM > *To:* Mendelson, Assaf > *Subject:* Re: Watermarking in Structured Streaming to drop late data > > > > Hi all > > I would highly recommend to all users-devs interested in the desi

RE: Watermarking in Structured Streaming to drop late data

2016-10-27 Thread assaf.mendelson
papageorgopoylos [via Apache Spark Developers List] [mailto:ml-node+s1001551n19592...@n3.nabble.com] Sent: Thursday, October 27, 2016 10:17 AM To: Mendelson, Assaf Subject: Re: Watermarking in Structured Streaming to drop late data Hi all I would highly recommend to all users-devs interested in

Re: Watermarking in Structured Streaming to drop late data

2016-10-27 Thread Tathagata Das
>> >> Is this the use case we are trying to solve? >> >> If so, wouldn’t just calculating the bigger time window across the entire >> aggregation solve this? >> >> Am I missing something here? >> >> >> >> *From:* Michael Armbrust [via Apache

Re: Watermarking in Structured Streaming to drop late data

2016-10-27 Thread Ofir Manor
n solve this? > > Am I missing something here? > > > > *From:* Michael Armbrust [via Apache Spark Developers List] [mailto: > ml-node+[hidden email] > <http:///user/SendEmail.jtp?type=node&node=19591&i=0>] > *Sent:* Thursday, October 27, 2016 3:04 AM > *To:*

Re: Watermarking in Structured Streaming to drop late data

2016-10-27 Thread kostas papageorgopoylos
g something here? > > > > *From:* Michael Armbrust [via Apache Spark Developers List] [mailto: > ml-node+[hidden email] > <http:///user/SendEmail.jtp?type=node&node=19591&i=0>] > *Sent:* Thursday, October 27, 2016 3:04 AM > *To:* Mendelson, Assaf > *Subject:*

RE: Watermarking in Structured Streaming to drop late data

2016-10-26 Thread assaf.mendelson
Apache Spark Developers List] [mailto:ml-node+s1001551n19590...@n3.nabble.com] Sent: Thursday, October 27, 2016 3:04 AM To: Mendelson, Assaf Subject: Re: Watermarking in Structured Streaming to drop late data And the JIRA: https://issues.apache.org/jira/browse/SPARK-18124 On Wed, Oct 26, 2016 at 4

Re: Watermarking in Structured Streaming to drop late data

2016-10-26 Thread Michael Armbrust
And the JIRA: https://issues.apache.org/jira/browse/SPARK-18124 On Wed, Oct 26, 2016 at 4:56 PM, Tathagata Das wrote: > Hey all, > > We are planning implement watermarking in Structured Streaming that would > allow us handle late, out-of-order data better. Specially, when we are > aggregating ov