Re: streaming mode with both finite and infinite input sources

[Dawid Wysakowicz]

This should be supported in 1.14 if you enable checkpointing with finished tasks[1], which has been added in 1.14. In 1.15 it will be enabled by default.

Best,

Dawid

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/#enabling-and-configuring-checkpointing

On 25/02/2022 08:18, Jin Yi wrote:

so we have a streaming job where the main work to be done is processing infinite kafka sources.  recently, i added a fromCollection (finite) source to simply write some state once upon startup.  this all seems to work fine.  the finite source operators all finish, while all the infinite source operators continue running with watermarks.

is this mixed finity not a supported setup?

however, the weird thing is that savepointing throws exceptions, and there have been no automatic checkpoints at all while the job has been running for 90+minutes (checkpoint config is unaligned, exactly once every 5m w/ a 1h timeout).

OpenPGP_signature
Description: OpenPGP digital signature