Re: [DISCUSS] Handling event-time in continuous file processing.

2016-12-09 Thread Fabian Hueske
Hi Kostas, I think it would be good to open two JIRAs to track these issues: 1) to document the shortcomings of the current solution 2) propose a solution based on your idea of group-ids. Would you like to do that? Thanks, Fabian 2016-12-01 10:48 GMT+01:00 Fabian Hueske : > Hi Kostas, > > Tha

Re: [DISCUSS] Handling event-time in continuous file processing.

2016-12-01 Thread Fabian Hueske
Hi Kostas, Thanks for bringing up this issue and the good explanation! I think we need to do two things: 1) Clearly explain the limitations of the current version in the online documentation and JavaDocs. This should point out that the source does only work correctly with event-time and timestam

[DISCUSS] Handling event-time in continuous file processing.

2016-12-01 Thread Kostas Kloudas
Hi all, This is to open a discussion on how to better handle event-time in continuous file processing. For the sake of illustration of the problem we will use the example of processing hourly server logs. In this case, each server writes its logs in hourly files, with names: