Hi, Chen commented this on the doc (I'm mirroring here so everyone can follow): "It would be cool to be able to access last snapshot of window states before it get purged. Pipeline author might consider put it to external storage and deal with late arriving events by restore corresponding window."
My answer: This is partially covered by the section called "What Happens at Window-Cleanup Time, Who Decides When to Purge". What I want to introduce is that the window can have one final emission if there is new data in the buffers at cleanup time. The work on this will also depend on this proposal: https://cwiki.apache.org/confluence/display/FLINK/FLIP-2+Extending+Window+Function+Metadata With this, the WindowFunction can get meta data about the window firing so it could be informed that this is the last firing before a cleanup and that there already was an earlier, on-time firing. Does this cover your concerns, Chen? Cheers, Aljoscha On Sun, 10 Jul 2016 at 21:24 Chen Qin <qinnc...@gmail.com> wrote: > Sure. Currently, it looks like any element assigned to a too late window > will be dropped silently😓 ? > > Having a late window stream imply somehow Flink needs to add one more state > to window and split window state cleanup from window retirement. > I would suggest as simple as adding a function in trigger called > OnLateElement and always fire_purge it would enable developer aware and > handle this case. > > Chen > > > > On Fri, Jul 8, 2016 at 1:00 AM, Aljoscha Krettek <aljos...@apache.org> > wrote: > > > @Chen I added a section at the end of the document regarding access to > the > > elements that are dropped as late. Right now, the section just mentions > > that we have to do this but there is no real proposal yet for how to do > it. > > Only a rough sketch so that we don't forget about it. > > > > On Fri, 8 Jul 2016 at 07:47 Chen Qin <qinnc...@gmail.com> wrote: > > > > > +1 for allowedLateness scenario. > > > > > > The rationale behind is there are backfills or data issues hold > in-window > > > data till watermark pass end time. It cause sink write partial output. > > > > > > Allow high allowedLateness threshold makes life easier to merge those > > > results and overwrite partial output with correct output at sink. But > > yeah, > > > pipeline author is at risk of blow up statebackend with huge states. > > > > > > Alternatively, In some case, if sink allows read-check-merge operation, > > > window can explicit call out events ingested after allowedLateness. It > > asks > > > pipeline author mitigated these events in a way outside of flink > > ecosystem. > > > The catch is that since everywhere in a flink job can replay and > > > checkpoint, notification should somehow includes these info as well. > > > > > > Thanks > > > Chen > > > > > > On Thu, Jul 7, 2016 at 12:14 AM, Kostas Kloudas < > > > k.klou...@data-artisans.com > > > > wrote: > > > > > > > Hi, > > > > > > > > In the effort to move the discussion to the mailing list, rather than > > the > > > > doc, > > > > there was a comment in the doc: > > > > > > > > “It seems this proposal marries the allowed lateness of events and > the > > > > discarding of window state. In most use cases this should be > > sufficient, > > > > but there are instances where having independent control of these may > > be > > > > useful. > > > > > > > > For instance, you may have a job that computes some aggregate, like a > > > sum. > > > > You may want to keep the window state around for a while, but not too > > > long. > > > > Yet you may want to continue processing late events after you > discarded > > > the > > > > window state. It is possible that your stream sinks can make use of > > this > > > > data. For instance, they may be writing to a data store that returns > an > > > > error if a row already exists, which allow the sink to read the > > existing > > > > row and update it with the new data." > > > > > > > > To which I would like to reply: > > > > > > > > If I understand your use-case correctly, I believe that the proposed > > > > binding of the allowed lateness to the state purging does not impose > > any > > > > problem. The lateness specifies the upper time bound, after which the > > > state > > > > will be discarded. Between the start of a window and its (end + > > > > allowedLateness) you can write custom triggers that fire, purge the > > > state, > > > > or do nothing. Given this, I suppose that, at the most extreme case, > > you > > > > can specify an allowed lateness of Long.MaxValue and do the purging > of > > > the > > > > state "manually". By doing this, you remove the safeguard of letting > > the > > > > system purge the state at some point in time, and you can do your own > > > > custom state management that fits your needs. > > > > > > > > Thanks, > > > > Kostas > > > > > > > > > On Jul 6, 2016, at 5:43 PM, Aljoscha Krettek <aljos...@apache.org> > > > > wrote: > > > > > > > > > > @Vishnu Funny you should ask that because I have a design doc lying > > > > around. > > > > > I'll open a new mail thread to not hijack this one. > > > > > > > > > > On Wed, 6 Jul 2016 at 17:17 Vishnu Viswanath < > > > > vishnu.viswanat...@gmail.com> > > > > > wrote: > > > > > > > > > >> Hi, > > > > >> > > > > >> I was going through the suggested improvements in window, and I > have > > > > >> few questions/suggestion on improvement regarding the Evictor. > > > > >> > > > > >> 1) I am having a use case where I have to create a custom Evictor > > that > > > > will > > > > >> evict elements from the window based on the value (e.g., if I have > > > > elements > > > > >> are of case class Item(id: Int, type:String) then evict elements > > that > > > > has > > > > >> type="a"). I believe this is not currently possible. > > > > >> 2) this is somewhat related to 1) where there should be an option > to > > > > evict > > > > >> elements from anywhere in the window. not only from the beginning > of > > > the > > > > >> window. (e.g., apply the delta function to all elements and remove > > all > > > > >> those don't pass. I checked the code and evict method just returns > > the > > > > >> number of elements to be removed and processTriggerResult just > skips > > > > those > > > > >> many elements from the beginning. > > > > >> 3) Add an option to enables the user to decide if the eviction > > should > > > > >> happen before the apply function or after the apply function. > > > Currently > > > > it > > > > >> is before the apply function, but I have a use case where I need > to > > > > first > > > > >> apply the function and evict afterward. > > > > >> > > > > >> I am doing these for a POC so I think I can modify the flink code > > base > > > > to > > > > >> make these changes and build, but I would appreciate any > suggestion > > on > > > > >> whether these are viable changes or will there any performance > issue > > > if > > > > >> these are done. Also any pointer on where to start(e.g, do I > create > > a > > > > new > > > > >> class similar to EvictingWindowOperator that extends > > WindowOperator?) > > > > >> > > > > >> Thanks and Regards, > > > > >> Vishnu Viswanath, > > > > >> > > > > >> On Wed, Jul 6, 2016 at 9:39 AM, Aljoscha Krettek < > > aljos...@apache.org > > > > > > > > >> wrote: > > > > >> > > > > >>> I did: > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > https://mail-archives.apache.org/mod_mbox/flink-dev/201606.mbox/%3ccanmxww0abttjjg9ewdxrugxkjm7jscbenmvrzohpt2qo3pq...@mail.gmail.com%3e > > > > >>> ;-) > > > > >>> > > > > >>> On Wed, 6 Jul 2016 at 15:31 Ufuk Celebi <u...@apache.org> wrote: > > > > >>> > > > > >>>> On Wed, Jul 6, 2016 at 3:19 PM, Aljoscha Krettek < > > > aljos...@apache.org > > > > > > > > > >>>> wrote: > > > > >>>>> In the future, it might be good to to discussions directly on > the > > > ML > > > > >>> and > > > > >>>>> then change the document accordingly. This way everyone can > > follow > > > > >> the > > > > >>>>> discussion on the ML. I also feel that Google Doc comments > often > > > > >> don't > > > > >>>> give > > > > >>>>> enough space for expressing more complex opinions. > > > > >>>> > > > > >>>> I agree! Would you mind raising this point as a separate > > discussion > > > on > > > > >>> dev@ > > > > >>>> ? > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > >
