Hi Marco,
Actually, perhaps I misworded it. This particular checkpoint seems to
> occur in an operator that is flat mapping (it is actually a keyed
> processing function) a single blob data-structure into several hundred
> thousands elements (sometimes a million) that immediately flow into a sink
I've seen checkpoints timeout when using the RocksDB state backend with
very large objects. The issue is that updating a ValueState stored in
RocksDB requires deserializing, updating, and then re-serializing that
object -- and if that's some enormous collection type, that will be slow.
In such case
By default, a checkpoint times out after 10 minutes. This means if not all
operators are able to confirm the checkpoint, it will be cancelled.
If you have an operator that is blocking for more than 10 minutes on a
single record (because this record contains millions of elements that are
written to
Actually, perhaps I misworded it. This particular checkpoint seems to
occur in an operator that is flat mapping (it is actually a keyed
processing function) a single blob data-structure into several hundred
thousands elements (sometimes a million) that immediately flow into a sink.
I am speculatin
1) An operator that just blocks for a long time (for example, because it
does a synchronous call to some external service) can indeed cause a
checkpoint timeout.
2) What kind of effects are you worried about?
On 1/28/2021 8:05 PM, Marco Villalobos wrote:
Is it possible that checkpointing times
Is it possible that checkpointing times out due to an operator taking too
long?
Also, does windowing affect the checkpoint barriers?
