Hi Martijn,
Its a continuous requirement so always read from one input source over another, but its does not require a super strict guarantee, so it doesn't matter if on occasion a message is read from the wrong topic. It's mainly due to there consistently being significantly more messages on one source than another which causes issues when we there are too many messages on the stream.
Thanks
Shazia
Its a continuous requirement so always read from one input source over another, but its does not require a super strict guarantee, so it doesn't matter if on occasion a message is read from the wrong topic. It's mainly due to there consistently being significantly more messages on one source than another which causes issues when we there are too many messages on the stream.
Thanks
Shazia
----- Original message -----
From: "Martijn Visser" <mart...@ververica.com>
To: "Shazia Kayani" <shazia.kay...@ibm.com>
Cc: "User" <user@flink.apache.org>
Subject: [EXTERNAL] Re: Input Selectable & Checkpointing
Date: Tue, Nov 23, 2021 2:45 PM
Hi,Do you have a requirement to continuously prioritise one input source over another (like always read topic X from Kafka before topic Y from Kafka) or is it a one-time effort, because you might need to bootstrap some state, so first read all data from file source A before switching over to topic B from Kafka?). If it's the latter, you could look into the HybridSource.Best regards,MartijnOn Tue, 23 Nov 2021 at 15:34, Shazia Kayani <shazia.kay...@ibm.com> wrote:Hi All,
Hope you are well!
I am working on something which has a requirement from flink to prioritise one input datastream over another, to do this I'm currently implemented an operator which extends InputSelectable to do this.
However, because of using input selectable checkpointing is disabled as it is currently not supported.
I was just wondering if anyone has done something similar to this previously? and if so were you able to implement changes which resulted in successful checkpointing?
If anyone has any other tips around the topic that too would also be helpful!
Thanks
Shazia
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
