On 10.03.20 10:13, kant kodali wrote:
If ingestion time programs cannot handle late data then why would it generate watermarks? Isn't the whole point of watermarks is to handle the late data?
Watermarks are not only used for handling late data. Watermarks are the mechanism that is used to update time throughout the streaming topology, starting from the sources. Among other things is is used to detect late data.
When setting the characteristic to "ingestion time" you are essentially instating a watermark extractor that extracts the current processing time at the sources as event time.
My last question was more about this library <https://github.com/vasia/gelly-streaming> I run several algorithms using SimpleEdgeStream.aggregrate(<algoirthm>).print() and I am running into the following error whenever I invoke the following constructor <https://github.com/vasia/gelly-streaming/blob/master/src/main/java/org/apache/flink/graph/streaming/SimpleEdgeStream.java#L69> . But it works if I change it to this <https://github.com/vasia/gelly-streaming/blob/master/src/main/java/org/apache/flink/graph/streaming/SimpleEdgeStream.java#L86> so I am not exactly sure what is happening there.
I don't know what is going on here, could it be that the library internally sets the characteristic to event-time, thereby overriding your ingestion-time setting? In that case you would indeed be missing a watermark extractor. I'm cc'ing Vasia, as the author of that library.
-Aljoscha
