Thank you for the suggestions, guys!
@Andrew Otto
This is the way we will most likely go. However, this will require us to
meddle with the Flink consumer codebase. And looks like there is no other
way around it. We will add some custom code to perform offset resetting for
specific savepoints.
@Ko
Hi there,
to me the simplest and most reliable solution still seems to be to split
the stream based on event time. It requires a bit of preparation and assume
that you can tolerate some downtime when doing the migration.
1) For Cloud1 you chain a filter to your sources that filters out any
record
Have you tried MirrorMaker 2's consumer offset translation feature? I have
not used this myself, but it sounds like what you are looking for!
https://issues.apache.org/jira/browse/KAFKA-9076
https://kafka.apache.org/26/javadoc/org/apache/kafka/connect/mirror/Checkpoint.html
https://strimzi.io/blog
Thank you for the suggestions, guys!
@Austin Cawley-Edwards
Your idea is spot on! This approach would surely work. We could take a
savepoint of each of our apps, load it using state processor apis and
create another savepoint accounting for the delta on the offsets, and start
the app on the new cl
Hello Hemanga,
MirrorMaker can cause havoc in many respects, for one, it does not have strict
exactly-once.semantics…
The way I would tackle this problem (and have done in similar situaltions):
* For the source topics that need to be have exactly-once-semantics and
that are not intrinsica
Hey Hemanga,
That's quite annoying of MirrorMaker to change the offsets on you. One
solution would be to use the State Processor API[1] to read the savepoint
and update the offsets to the new ones — does MirrorMaker give you these
ahead of time? There might also be more specific tricks people coul
Any ideas, guys?
On Mon, May 2, 2022 at 6:11 PM Hemanga Borah
wrote:
> Hello,
> We are attempting to port our Flink applications from one cloud provider
> to another.
>
> These Flink applications consume data from Kafka topics and output to
> various destinations (Kafka or databases). The appl
