Hi, roman Thanks for reply very much. In our case, we see some data was dropped in window operator. We found the root cause by adding a temporary metric about number of aligned channels, found an active channel resumed from Idle was not took into account for some time (not alway btw). We could walk around by allow lateness on window operator, however I wonder does it make sense to add a configuration to define different watermark aligned behavior?
Regards JING Roman Khachatryan <ro...@apache.org> 于2021年5月21日周五 上午1:05写道： > > Hi, > > AFAIK, this behavior is not configurable. > However, for this to happen the channel must consistently generate > watermarks smaller than watermarks from ALL aligned channels (and its > elements must have a smaller timestamp). I'm not sure how likely it > is. Is it something you see in production? > > Regards, > Roman > > On Thu, May 20, 2021 at 4:11 PM 张静 <beyond1...@gmail.com> wrote: > > > > Hi community, > > Now after a channel become active from idle, the watermark on this > > channel would not be took into account when align watermarks util it > > generates a watermark equals to or bigger than last emitted watermark. > > It makes sense because it could prevent the newly active task resumed > > from idle dragging down the entire job. > > > > However, if the newly active task generate watermarks which are > > smaller than but very closely to the last emitted one for a long time, > > it could not be took into consideration in watermark alignment which > > may lead to its data maybe dropped at the later window operator. > > > > Is there any way to solve this problem? Or could we add a > > configuration to define different watermark aligned behavior？ > > > > Any suggestion is appreciated. Thanks a lot. > > > > Regards > > JING
