Hey Rajesh,
I do like the idea of customized windows. To make that work, we need to define
a rule-based window size generation. In practical sense, fully randomized
windows are rare and those are usually addressed as session windows. For
calendar based window strategy, the goal is to define a cyclic formula for
window creation. For example, we should define an API like:
new WindowStore(List[Duration] months -> (31, 28, 31, 30...))
and notice that the correct cycle is 4 years (1 extra day on February). This
seems to be a very business specific use case that needs a lot of work to make
it happen. Considering the amount of work, day based window should definitely
easier for you and you just need to do secondary aggregation on top of the raw
data. Do you want to talk more about your use case here, especially how much
impact for your case if the rule-based window store API is developed?
Boyang
________________________________
From: Rajesh Kalyanasundaram <rajeshkalyanasunda...@fico.com>
Sent: Wednesday, April 10, 2019 1:05 AM
To: dev@kafka.apache.org
Subject: Re: Dynamic window size for aggregations
Thanks Boyang & Matthias for your replies.
Boyang you are right.. More generically, I want to create a calendar based
window. If my window is in months then the window shall tumble or hop end of
every month. If my window is in days, then the window shall tumble/advanceby
end of every day.
Perhaps, the thread's subject is misleading in the sense that its not dynamic
windows rather calendar based window. Obviously if my unit is days, then my
window size is NOT dynamic.
Thanks
Regards,
Rajesh
On 09/04/19, 10:12 PM, "Boyang Chen" <bche...@outlook.com> wrote:
Hey Rajesh,
my understanding is that you want to create a month-based time window, is
that correct?
Boyang
________________________________
From: Matthias J. Sax <matth...@confluent.io>
Sent: Tuesday, April 9, 2019 11:42 PM
To: dev@kafka.apache.org
Subject: Re: Dynamic window size for aggregations
Feel free to create a feature request JIRA.
For now, you could use a custom processor (via `.transform()`) that uses
an attached window store to implement the logic you need.
-Matthias
On 4/9/19 4:09 AM, Rajesh Kalyanasundaram wrote:
> Hi all,
> We have a requirement to implement aggregations with TimedWindows which
may have varying window sizes. For example, I may want a tumbling window that
tumbles on 31st of Jan and 28th of Feb, 31st of March and so on.
> We did initial analysis of TimedWindows. Found that the windoSize is
being fixed at many places in the streams code.
>
> Any guidance in supporting this would be very much appreciated.
> Thank you.
> Regards,
> Rajesh
> This email and any files transmitted with it are confidential,
proprietary and intended solely for the individual or entity to whom they are
addressed. If you have received this email in error please delete it
immediately.
>
This email and any files transmitted with it are confidential, proprietary and
intended solely for the individual or entity to whom they are addressed. If you
have received this email in error please delete it immediately.
