Re: Kafka partition alignment for event time

2016-02-25 Thread Erdem Agaoglu
>>> (late-)windows to be fired after restart, because we now generate >>>> watermarks >>>> off partition-3 which says the windows of the unseen data in >>>> partition-6 are >>>> already complete. >>>> >>>> This also has a side effect of windows not tr

Re: Kafka partition alignment for event time

2016-02-25 Thread Robert Metzger
effect of windows not triggering unless some >>> rebalancing is done beforehand. Since only 1 of the 4 threads will source >>> data and generate watermarks, window triggers won't get watermarks from >>> other 3 sources and wait long past the waterma

Re: Kafka partition alignment for event time

2016-02-18 Thread Erdem Agaoglu
; >> I know producers shouldn't work like that, but consumers shouldn't care. I >> think it may also create some edge cases even if things were not as >> extreme >> as ours. If checkpoints could contain offsets of all of the partitions >> regardless of their contents, probably storing start offsets in first >> run, i >> guess that would solve the problems around restarting. >> >> >> >> -- >> View this message in context: >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4998.html >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive at Nabble.com. >> > > -- erdem agaoglu

Re: Kafka partition alignment for event time

2016-02-18 Thread Stephan Ewen
nk it may also create some edge cases even if things were not as extreme > as ours. If checkpoints could contain offsets of all of the partitions > regardless of their contents, probably storing start offsets in first run, > i > guess that would solve the problems around restarti

Re: Kafka partition alignment for event time

2016-02-18 Thread agaoglu
n first run, i guess that would solve the problems around restarting. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4998.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Kafka partition alignment for event time

2016-02-09 Thread shikhar
mins and starting it back up (the offsets it's resuming from are approx the same number of messages behind). -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4819.html Sent from the Apache Flink

Re: Kafka partition alignment for event time

2016-02-09 Thread Stephan Ewen
ent, perhaps allowing for more than 1 Kafka > >> partition > >> on a source should require opt-in, e.g. allowOversubscription() > >> > >> > >> > >> -- > >> View this message in context: > >> > http://apache-flink-user-mailin

Re: Kafka partition alignment for event time

2016-02-09 Thread shikhar
-alignment-for-event-time-tp4782p4817.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Kafka partition alignment for event time

2016-02-09 Thread shikhar
on >> on a source should require opt-in, e.g. allowOversubscription() >> >> >> >> -- >> View this message in context: >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4788.html >&g

Re: Kafka partition alignment for event time

2016-02-09 Thread Stephan Ewen
> > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4788.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >

Re: Kafka partition alignment for event time

2016-02-09 Thread Fabian Hueske
mark. Having m<=n ensures that the lowest watermark is used. > > > > Re: automatic enforcement, perhaps allowing for more than 1 Kafka > partition > > on a source should require opt-in, e.g. allowOversubscription() > > > > > > > > -- > > View

Re: Kafka partition alignment for event time

2016-02-09 Thread Aljoscha Krettek
> > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4788.html > Sent from the Apache Flink User Mailing List archive. mailing list archive at > Nabble.com.

Re: Kafka partition alignment for event time

2016-02-08 Thread shikhar
ark is used. Re: automatic enforcement, perhaps allowing for more than 1 Kafka partition on a source should require opt-in, e.g. allowOversubscription() -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-

Re: Kafka partition alignment for event time

2016-02-08 Thread Aljoscha Krettek
omatically > enforced. > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4786.html > Sent from the Apache Flink User Mailing List archive. mailing list archive at > Nabble.com.

Re: Kafka partition alignment for event time

2016-02-08 Thread shikhar
documented or automatically enforced. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782p4786.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Kafka partition alignment for event time

2016-02-08 Thread shikhar
://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-tp4782.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.