Thanks Guozhang, Thanks for that catch. to clarify, currently, events are "late" only when they are older than the retention period. Currently, we detect this in the processor and record it as a "skipped-record". We then do not attempt to store the event in the window store. If a user provided a pre-configured window store with a retention period smaller than the one they specify via Windows#until, the segmented store will drop the update with no metric and record a debug-level log.
With KIP-328, with the introduction of "grace period" and moving retention fully into the state store, we need to have metrics for both "late events" (new records older than the grace period) and "expired window events" (new records for windows that are no longer retained in the state store). I already proposed metrics for the late events, and I've just updated the KIP with metrics for the expired window events. I also updated the KIP to make it clear that neither late nor expired events will count as "skipped-records" any more. -John On Mon, Jul 30, 2018 at 4:22 PM Guozhang Wang <wangg...@gmail.com> wrote: > Hi John, > > Thanks for the updated KIP, +1 from me, and one minor suggestion: > > Following your suggestion of the differentiation of `skipped-records` v.s. > `late-event-drop`, we should probably consider moving the scenarios where > records got ignored due the window not being available any more in windowed > aggregation operators from the `skipped-records` metrics recording to the > `late-event-drop` metrics recording. > > > > Guozhang > > > On Mon, Jul 30, 2018 at 1:36 PM, Bill Bejeck <bbej...@gmail.com> wrote: > > > Thanks for the KIP! > > > > +1 > > > > -Bill > > > > On Mon, Jul 30, 2018 at 3:42 PM Ted Yu <yuzhih...@gmail.com> wrote: > > > > > +1 > > > > > > On Mon, Jul 30, 2018 at 11:46 AM John Roesler <j...@confluent.io> > wrote: > > > > > > > Hello devs, > > > > > > > > The discussion of KIP-328 has gone some time with no new comments, > so I > > > am > > > > calling for a vote! > > > > > > > > Here's the KIP: https://cwiki.apache.org/confluence/x/sQU0BQ > > > > > > > > The basic idea is to provide: > > > > * more usable control over update rate (vs the current state store > > > caches) > > > > * the final-result-for-windowed-computations feature which several > > people > > > > have requested > > > > > > > > Thanks, > > > > -John > > > > > > > > > > > > > -- > -- Guozhang >
