Thanks Leah and Sophie for the KIP. 1. I'm a bit surprised that we don't have an advance time. Could we elaborate how the storage layer is structured?
2. IIUC, there will be extra cost in terms of fetching aggregation results, since we couldn't pre-aggregate until the user asks for it. Would be good to also discuss it. 3. We haven't discussed the possibility of supporting sliding windows inherently. For a user who actually uses a hopping window, Streams could detect such an inefficiency doing a window_size/advance_time ratio to reach a conclusion on whether the write amplification is too high compared with some configured threshold. The benefit of doing so is that existing Streams users don't need to change their code, learn a new API, but only to upgrade Streams library to get benefits for their inefficient hopping window implementation. There might be some compatibility issues for sure, but worth listing them out for trade-off. Boyang On Fri, Jul 10, 2020 at 12:40 PM Leah Thomas <ltho...@confluent.io> wrote: > Hey Matthias, > > Thanks for pointing that out. I added the following to the Propose Changes > section of the KIP: > > "Records that come out of order will be processed the same way as in-order > records, as long as they fall within the grace period. Any new windows > created by the late record will still be created, and the existing windows > that are changed by the late record will be updated. Any record that falls > outside of the grace period (either user defined or default) will be > discarded. " > > All the best, > Leah > > On Thu, Jul 9, 2020 at 9:47 PM Matthias J. Sax <mj...@apache.org> wrote: > > > Leah, > > > > thanks a lot for the KIP. Very well written. > > > > The KIP does not talk about the handling of out-of-order data though. > > How do you propose to address this? > > > > > > -Matthias > > > > On 7/8/20 5:33 PM, Leah Thomas wrote: > > > Hi all, > > > I'd like to kick-off the discussion for KIP-450, adding sliding window > > > aggregation support to Kafka Streams. > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-450%3A+Sliding+Window+Aggregations+in+the+DSL > > > > > > Let me know what you think, > > > Leah > > > > > > > >
