Hi Alex, Maybe Seth (cc'ed) may have an opinion on this.
Cheers, Kostas On Thu, Jul 23, 2020 at 12:08 PM Александр Сергеенко <aleksandr.sergee...@gmail.com> wrote: > > Hi, > > We use so-called "control stream" pattern to deliver settings to the Flink > job using Apache Kafka topics. The settings are in fact an unlimited stream > of events originating from the master DBMS, which acts as a single point of > truth concerning the rules list. > > It may seems odd, since Flink guarantees the "exactly once" delivery > semantics, while a service, which provides the rules publishing mechanism to > Kafka is written using Akka Streams and guarantees the "at least once" > semantics while the rule handling inside Flink Job implemented in an > idempotent manner, but: we have to manage some cases when we need to execute > a reconciliation between the current rules stored at the master DBMS and the > existing Flink State. > > We've looked at the Flink's tooling and found out that the State Processor > API can possibly solve our problem, so we basically have to implement a > periodical process, which unloads the State to some external file (.csv) and > then execute a comparison between the set and the information given at the > master system. > Basically it looks like the lambda architecture approach while Flink is > supposed to implement the kappa architecture and in that case our > reconciliation problem looks a bit far-fetched. > > Are there any best practices or some patterns addressing such scenarios in > Flink? > > Great thanks for any possible assistance and ideas. > > ----- > Alex Sergeenko >
