Actually, perhaps I misworded it. This particular checkpoint seems to occur in an operator that is flat mapping (it is actually a keyed processing function) a single blob data-structure into several hundred thousands elements (sometimes a million) that immediately flow into a sink. I am speculating that the sink writes to the database were taking too long and causing a checkpoint to fail, but I changed that sink into a print, and the checkpoint still failed, so it must be something else.
I don't know deep details regarding Flinks internals, but I am speculating that the data between this operator and sink has to be checkpointed before the sink actually does something. On Mon, Feb 1, 2021 at 2:37 AM Chesnay Schepler <ches...@apache.org> wrote: > 1) An operator that just blocks for a long time (for example, because it > does a synchronous call to some external service) can indeed cause a > checkpoint timeout. > > 2) What kind of effects are you worried about? > > On 1/28/2021 8:05 PM, Marco Villalobos wrote: > > Is it possible that checkpointing times out due to an operator taking > > too long? > > > > Also, does windowing affect the checkpoint barriers? > > >
