Hi Weijie, In our case we do have 7 operators. All the 7 operators are getting executed as one chain within a single StreamTask. As checkpoint barrier is passing through all the operators, there are 7 checkpoints being stored. So our checkpoint size is up by 7 times. We are investigating to see if we can checkpoint the start operator (kafka source) or end operator (BQ sink), we are good and check point size comes down. Hence the question, when the operators are executed in the same StreamTask as one chain, is it possible to checkpoint at operator chain or single operator level?
Thanks, Kishore ________________________________ From: weijie guo <guoweijieres...@gmail.com> Sent: Wednesday, February 1, 2023 6:59 PM To: Talat Uyarer <tuya...@paloaltonetworks.com> Cc: user@flink.apache.org <user@flink.apache.org> Subject: Re: Reducing Checkpoint Count for Chain Operator Hi Talat, In Flink, a checkpoint barrier will be injected from source, and then pass through all operators in turn. Each stateful operator will do checkpoint in this process, the state is managed at operator granularity, not operator chain. So what is the significance of checkpoint based on the granularity of operator chain？ Best regards, Weijie Talat Uyarer <tuya...@paloaltonetworks.com<mailto:tuya...@paloaltonetworks.com>> 于2023年2月2日周四 02:20写道： Hi Weijie, Thanks for replying back. Our job is a streaming job. The OperatorChain contains all operators that are executed as one chain within a single StreamTask. But each operator creates their own checkpoint at checkpointing time . Rather than creating a checkpoint per operator in checkpointing time. Can I have one checkpoint per OperatorChain? This is my question. Thanks On Wed, Feb 1, 2023 at 1:02 AM weijie guo <guoweijieres...@gmail.com<mailto:guoweijieres...@gmail.com>> wrote: Hi Talat, Can you elaborate on what it means to create one checkpoint object per chain operator more than all operators? If you mean to do checkpoint independently for each task, this is not supported. Best regards, Weijie Talat Uyarer via user <user@flink.apache.org<mailto:user@flink.apache.org>> 于2023年2月1日周三 15:34写道： Hi, We have a job that is reading from kafka and writing some endpoints. The job does not have any shuffling steps. I implement it with multiple steps. Flink chained those operators in one operator in submission time. However I see all operators are doing checkpointing. Is there any way to create one checkpoint object per chain operator rather than all operators ? Thanks
