[
https://issues.apache.org/jira/browse/FLINK-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415532#comment-15415532
]
ASF GitHub Bot commented on FLINK-4329:
---------------------------------------
GitHub user kl0u opened a pull request:
https://github.com/apache/flink/pull/2350
[FLINK-4329] Fix Streaming File Source Timestamps/Watermarks Handling
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kl0u/flink continuous_file_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2350.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2350
----
commit 6207a9f5da086d808331afe0e8caf0f03b3fabc5
Author: kl0u <kklou...@gmail.com>
Date: 2016-08-09T12:11:45Z
[FLINK-4329] Fix Streaming File Source Timestamps/Watermarks Handling
Now the ContinuousFileReaderOperator ignores the watermarks sent by
the source function and emits its own watermarks in case we are
opearating on Ingestion time. In addition, and for Ingestion time
only, the reader also assigns the correct timestamps to the elements
that it reads.
----
> Fix Streaming File Source Timestamps/Watermarks Handling
> --------------------------------------------------------
>
> Key: FLINK-4329
> URL: https://issues.apache.org/jira/browse/FLINK-4329
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
> Affects Versions: 1.1.0
> Reporter: Aljoscha Krettek
> Assignee: Kostas Kloudas
> Fix For: 1.1.1
>
>
> The {{ContinuousFileReaderOperator}} does not correctly deal with watermarks,
> i.e. they are just passed through. This means that when the
> {{ContinuousFileMonitoringFunction}} closes and emits a {{Long.MAX_VALUE}}
> that watermark can "overtake" the records that are to be emitted in the
> {{ContinuousFileReaderOperator}}. Together with the new "allowed lateness"
> setting in window operator this can lead to elements being dropped as late.
> Also, {{ContinuousFileReaderOperator}} does not correctly assign ingestion
> timestamps since it is not technically a source but looks like one to the
> user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
