Hey, so I was just about to set `max.in.flight.requests.per.connection=1` when I found in the logs that it is already set for the "destination" producers :) Cool.
Duplicates are fine for me, we have dedup checks or immutable operations on the consumers. Have a nice day! Peter On Mon, 27 Apr 2020 at 19:45, Ryanne Dolan <ryannedo...@gmail.com> wrote: > > Could the `max.in.flight.requests.per.connection=1` parameter help to > > prevent the "slightly out-of-order records"? > > Yes that helps, but dupes are still possible when MM2 restarts or > rebalances, since it will restart at the latest commit. If you are > replicating something like CDC or changelogs, then dupes might be fine > (downstream state will be eventually consistent). That's a common pattern > with MM1 as well. > > Ryanne > > On Mon, Apr 27, 2020 at 4:47 AM Péter Sinóros-Szabó > <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > > Hey Ryanne, > > > > Is there any documentation where I can read more about this "slightly > > out-of-order records"? > > It would help very much to see how we can use MM2 in our systems. > > > > Thanks, > > Peter > > > > On Thu, 23 Apr 2020 at 08:56, Péter Sinóros-Szabó < > > peter.sinoros-sz...@transferwise.com> wrote: > > > > > Hey Ryanna, > > > > > > Could the `max.in.flight.requests.per.connection=1` parameter help to > > > prevent the "slightly out-of-order records"? > > > Or is there any workaround for that? Duplicates are fine for me, but > I'd > > > like to have the same order of messages too. > > > Can you please add some more detail about why those "slightly > > out-of-order > > > records" may happen? > > > > > > Thanks, > > > Peter > > > > > > On Wed, 22 Apr 2020 at 20:16, Ryanne Dolan <ryannedo...@gmail.com> > > wrote: > > > > > >> Hey Peter, Connect will need to support transactions before we can > > >> guarantee the order of records in remote topics. We can guarantee that > > no > > >> records are dropped or skipped, even during consumer > failover/migration > > >> etc, but we can still have duplicates and slightly out-of-order > records > > in > > >> the downstream remote topics, for now. > > >> > > >> Ryanne > > >> > > >> On Wed, Apr 22, 2020 at 3:39 AM Péter Sinóros-Szabó > > >> <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > >> > > >> > Hey, > > >> > > > >> > so KIP-382 mentions that: > > >> > "Partitioning and order of records is preserved between source and > > >> remote > > >> > topics." > > >> > is the ordering of messages (I guess only in a partition) something > > >> that is > > >> > actually implemented in 2.4 (or in 2.5)? > > >> > > > >> > Or do I need to set `max.in.flight.requests.per.connection=1` ? > > >> > > > >> > Thanks, > > >> > Peter > > >> > > > >> > > > > > >
