Both the sources have to provide watermark. - KafkaIO : It can have its watermark stuck if you don't have any records in one of the partitions. Thats is a bug. The management of timestamps and watermarks in KafkaIO are being updated in https://github.com/apache/beam/pull/4680. - TextIO.watchForNewFiles() - I am not sure how the watermark is handled by TextIO. Didn't notice any mentions of in implementation.
On Tue, Feb 20, 2018 at 10:13 AM, Vilhelm von Ehrenheim < [email protected]> wrote: > Hi all! > I have a somewhat complicated stateful DoFn that i would like to add an > event time timer on. My goal with the timer is to not output anything until > sufficient amount of state has been built up in a Global window. > > In doing this I realize that the watermark doesn’t seem to progress at all > (regardless of the timer) and in Google Dataflow the displayed watermark is > just “-“ when clicking on the ParDo(DoFn) node. > > The DoFn is reading flattened input from a TextIO.watchForNewFiles and > KafkaIO. The Flatten element had a watermark set. > > I have written tests for my DoFn that all pass using TestStream but since > I there explicitly set the watermark progression all is fine. > > What can I do to look into why there is no watermark progression for a > specific PTransform? > > Regards, > Vilhelm von Ehrenheim >
