Awesome !
I gonna implement it.
Thanks a lot Arvid.
Le mer. 8 janv. 2020 à 12:00, Arvid Heise a écrit :
> I'd second Chesnay's suggestion to use a custom source. It would be a
> piece of cake with FLIP-27 [1], but we are not there yet unfortunately.
> It's probably in Flink 1.11 (mid year) if yo
I'd second Chesnay's suggestion to use a custom source. It would be a piece
of cake with FLIP-27 [1], but we are not there yet unfortunately. It's
probably in Flink 1.11 (mid year) if you can wait.
The current way would be a source that wraps the two KafkaConsumer and
blocks the normal consumer fr
My naive solution can't work because a dump can be quite long.
So, yes I have to find a way to stop the consumption from the topic used for
streaming mode when a dump is done :(
Terry, I try to implement something based on your reply and based on this
thread
https://stackoverflow.com/questions/5
Hi,
Thanks for your replies.
Yes Terry. You are right. I can try to create a custom source.
But perhaps, according to my use case, I figured out I can use a technical
field in my data. This is a timestamp and I think I just have to ignore late
events with watermarks or later in the pipeline acc
Are you asking how to detect from within the job whether the dump is
complete, or how to combine these 2 jobs?
If you had a way to notice whether the dump is complete, then I would
suggest to create a custom source that wraps 2 kafka sources, and switch
between them at will based on your condi
Hi,
I’d like to share my opinion here. It seems that you need adjust the Kafka
consumer to have communication each other. When your begin the dump process,
you need to notify another CDC-topic consumer to wait idle.
Best,
Terry Wang
> 2020年1月2日 16:49，David Morin 写道：
>
> Hi,
>
> Is there
Hi,
Is there a way to stop temporarily to consume one kafka source in streaming
mode ?
Use case: I have to consume 2 topics but in fact one of them is more
prioritized.
One of this topic is dedicated to ingest data from db (change data capture) and
one of them is dedicated to make a synchroniza
