Hi,

The paths of the files to read are distributed across all reader / source
tasks and each task reads the files in order of their modification
timestamp.
The watermark generator is not aware of any files and just looks at the
stream of records produced by the source tasks.
You need to chose the WM generator strategy such that you minimize the
number of late records.

I'd recommend to first investigate how many late records you are dealing
with.
You can use a custom ProcessFunction and compare the timestamp of each
record with the current watermark.

AllowedLateness is also not a magical cure. It will just emit updates
downstream, i.e., you need to remove the results that were updated by a
more complete result.

Best, Fabian

Am Mo., 26. Aug. 2019 um 10:21 Uhr schrieb Hanan Yehudai <
hanan.yehu...@radcom.com>:

> The data  source is generated by an application that monitors some sort of
> sessions.
>
> With the EVENT_TIME column being the session end time .
>
>
>
> It is possible that the files will have out of order data , because of the
> async nature of the application writing  files.
>
>  While the EVENT_TIME is monotonically  increasing in general .  some
> lateness is possible. However ,    I used *allowlateness*  on my stream
> and still got the inconsistencies
>
>
>
> Although the real life use case is generically reading files form a
> folder.  The testing  env has an already set of files in advanced -  these
>  should be read and produce the result.
>
>
>
> You mentioned the “right” order of the files.  Is it sorted by update time
> ?  when running in parallel, is it possible that 2 files will be read in
> parallel. And in case that the latter one is smaller.  The latest timestamp
> will  be handled first ?
>
>
>
>
>
> BTW I tried to use a ContinuousEventTimeTrigger  to make sure the window
> is calculated ?  and got the processing to trigger multiple times  so I’m
> not sure exactly how this type of trigger works..
>
>
>
> Thanks
>
>
>
>
>
>
>
>
>
> *From:* Fabian Hueske <fhue...@gmail.com>
> *Sent:* Monday, August 26, 2019 11:06 AM
> *To:* Hanan Yehudai <hanan.yehu...@radcom.com>
> *Cc:* user@flink.apache.org
> *Subject:* Re: tumbling event time window , parallel
>
>
>
> Hi,
>
>
>
> Can you share a few more details about the data source?
>
> Are you continuously ingesting files from a folder?
>
>
>
> You are correct, that the parallelism should not affect the results, but
> there are a few things that can affect that:
>
> 1) non-determnistic keys
>
> 2) out-of-order data with inappropriate watermarks
>
>
>
> Note that watermark configuration for file ingests can be difficult and
> that you need to ensure that files are read in the "right" order.
>
> AFAIK, Flink's continuous file source uses the modification timestamp of
> files to determine the read order.
>
>
>
> Best, Fabian
>
>
>
> Am So., 25. Aug. 2019 um 19:32 Uhr schrieb Hanan Yehudai <
> hanan.yehu...@radcom.com>:
>
> I have an issue with tumbling windows running  in parallel.
>
>
>
> I run a Job on  a set of CSV files.
>
>
>
> When the parallelism is set to 1.  I get the proper results.
>
> While it runs in parallel.   I get no output.
>
> Is it  due to the fact the parallel streams take the MAX(watermark) from
> all the parallel sources.
>
> And only one of the streams advances the watermark ?
>
>
>
> It seems wrong that the result is not deterministic  and depends on the
> parallel level.
>
> What am I doing wrong ?
>
>
>
>

Reply via email to