> First, some remarks here - sources (in your case the Kafka consumer) will > not stop fetching / producing data when the windows haven’t fired yet. >
This is for sure true. However, the plot shows the number of records produced per second, where each record was assigned a created at timestamp while being created and before being pushed back to Kafka. Sorry I did not clarify this before. Anyway, because of this I would expect to have a certain lag. Of course, messages will not only be produced into Kafka exactly at window expiry and then the produced shutdown - however, what concerns me is that messages were produced to Kafka before the first window expired - hence the questions. > If you’re writing the outputs of the window operation to Kafka (by adding a > Kafka sink after the windowing), then yes it should only write to Kafka when > the window has fired. Hence, I this behaviour that you’ve described and we’ve expected did not occur. If it would help, I can share the source code and a detail Flink configuration. Cheers, Dominik > On 30 Mar 2017, at 13:09, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > > Hi, > > Thanks for the clarification. > >> What are the reasons behind consuming/producing messages from/to Kafka while >> the window has not expired yet? > > First, some remarks here - sources (in your case the Kafka consumer) will > not stop fetching / producing data when the windows haven’t fired yet. Does > this explain what you have plotted in the diagram you attached (sorry, I > can’t really reason about the diagram because I’m not so sure what the values > of the x-y axes represent)? > > If you’re writing the outputs of the window operation to Kafka (by adding a > Kafka sink after the windowing), then yes it should only write to Kafka when > the window has fired. The characteristics will also differ for different > types of windows, so you should definitely take a look at the Windowing docs > [1] about them. > > Cheers, > Gordon > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html#window-assigners > > <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html#window-assigners> > On March 30, 2017 at 2:37:41 PM, Dominik Safaric (dominiksafa...@gmail.com > <mailto:dominiksafa...@gmail.com>) wrote: > >> What are the reasons behind consuming/producing messages from/to Kafka while >> the window has not expired yet?
