Are you asking how to detect from within the job whether the dump is
complete, or how to combine these 2 jobs?
If you had a way to notice whether the dump is complete, then I would
suggest to create a custom source that wraps 2 kafka sources, and switch
between them at will based on your conditions.
On 03/01/2020 03:53, Terry Wang wrote:
Hi,
I’d like to share my opinion here. It seems that you need adjust the Kafka
consumer to have communication each other. When your begin the dump process,
you need to notify another CDC-topic consumer to wait idle.
Best,
Terry Wang
2020年1月2日 16:49，David Morin <morin.david....@gmail.com> 写道：
Hi,
Is there a way to stop temporarily to consume one kafka source in streaming
mode ?
Use case: I have to consume 2 topics but in fact one of them is more
prioritized.
One of this topic is dedicated to ingest data from db (change data capture) and
one of them is dedicated to make a synchronization (a dump i.e. a SELECT ...
from db). At the moment the last one is performed by one Flink job and we start
this one after stop the previous one (CDC) manually
I want to merge these 2 modes and automatically stop consumption of the topic
dedicated to the CDC mode when a dump is done.
How to handle that with Flink in a streaming way ? backpressure ? ...
Thx in advance for your insights
David
