> Could the `max.in.flight.requests.per.connection=1` parameter help to > prevent the "slightly out-of-order records"?
Yes that helps, but dupes are still possible when MM2 restarts or rebalances, since it will restart at the latest commit. If you are replicating something like CDC or changelogs, then dupes might be fine (downstream state will be eventually consistent). That's a common pattern with MM1 as well. Ryanne On Mon, Apr 27, 2020 at 4:47 AM Péter Sinóros-Szabó <peter.sinoros-sz...@transferwise.com.invalid> wrote: > Hey Ryanne, > > Is there any documentation where I can read more about this "slightly > out-of-order records"? > It would help very much to see how we can use MM2 in our systems. > > Thanks, > Peter > > On Thu, 23 Apr 2020 at 08:56, Péter Sinóros-Szabó < > peter.sinoros-sz...@transferwise.com> wrote: > > > Hey Ryanna, > > > > Could the `max.in.flight.requests.per.connection=1` parameter help to > > prevent the "slightly out-of-order records"? > > Or is there any workaround for that? Duplicates are fine for me, but I'd > > like to have the same order of messages too. > > Can you please add some more detail about why those "slightly > out-of-order > > records" may happen? > > > > Thanks, > > Peter > > > > On Wed, 22 Apr 2020 at 20:16, Ryanne Dolan <ryannedo...@gmail.com> > wrote: > > > >> Hey Peter, Connect will need to support transactions before we can > >> guarantee the order of records in remote topics. We can guarantee that > no > >> records are dropped or skipped, even during consumer failover/migration > >> etc, but we can still have duplicates and slightly out-of-order records > in > >> the downstream remote topics, for now. > >> > >> Ryanne > >> > >> On Wed, Apr 22, 2020 at 3:39 AM Péter Sinóros-Szabó > >> <peter.sinoros-sz...@transferwise.com.invalid> wrote: > >> > >> > Hey, > >> > > >> > so KIP-382 mentions that: > >> > "Partitioning and order of records is preserved between source and > >> remote > >> > topics." > >> > is the ordering of messages (I guess only in a partition) something > >> that is > >> > actually implemented in 2.4 (or in 2.5)? > >> > > >> > Or do I need to set `max.in.flight.requests.per.connection=1` ? > >> > > >> > Thanks, > >> > Peter > >> > > >> > > >
