Re: Understanding the semantics of SourceContext.collect

2021-08-04 Thread Caizhi Weng
Oh, and in batch jobs even if the whole DAG is a single node this is not guaranteed. For example, for a sort operator the record will be stored in memory or on disk and only after all records have arrived will these records be sorted and sent to the downstream. So the state in your ASO is still nee

Re: Understanding the semantics of SourceContext.collect

2021-08-04 Thread Caizhi Weng
Hi! There is no such guarantee unless the whole DAG is a single node. Flink's runtime runs the same node (task) in the same thread, while different nodes (tasks) are executed in different threads, even in different machines. Yuval Itzchakov 于2021年8月5日周四 上午2:10写道: > Hi, > > I have a question reg

Understanding the semantics of SourceContext.collect

2021-08-04 Thread Yuval Itzchakov
Hi, I have a question regarding the semantics of event processing from a source downstream that I want to clarify. I have a custom source which offloads data from our data warehouse. In my custom source, I have some state which keeps track of the latest timestamps that were read. When unloading,