That's exactly what my question was: since there is an external producer in the application without a sink node in topology, how will the streams know that task is completed before committing the offset or it will not know at all ?
Second question is: can there be multiple sink nodes if the record is to be produced on different topics based on some conditions (e.g. just giving a simplified example: if record contains field A then produce on topic A, field B then produce on Topic B etc.) On Thu, Jul 15, 2021 at 4:29 AM Matthias J. Sax <mj...@apache.org> wrote: > Yes, if you use async writes, it could lead to data loss in case if > failure as offsets could have been committed before the write succeeded. > Only sync writes guard you from data loss. > > Note though that in Kafka Streams there is not manual commit anyway. > Commits happen based on `commit.interval.ms` config. Calling > `context.commit()` only schedules an earlier commit, but after the call > returned, no commit happened yet (just a request to commit asap was > registered). > > > -Matthias > > On 7/14/21 12:00 AM, Pushkar Deole wrote: > > If async-writes are used either with manual or auto commit, where the > > record is sent for async write and consumer thread commits the offset > > immediately, however async write fails, > > > > this may cause the loss of event ? > > > > On Wed, Jul 14, 2021 at 12:20 AM Matthias J. Sax <mj...@apache.org> > wrote: > > > >> If you want to use EOS, you cannot use your own producer, but you need > >> to use a "sink node" (via `addSink()` or `to()`). > >> > >> For at-least-once, if you use sync-writes, you should still get > >> at-least-once guarantees. > >> > >> -Matthias > >> > >> On 7/9/21 4:12 AM, Pushkar Deole wrote: > >>> Matthias, > >>> > >>> Do you have any inputs on above queries? > >>> > >>> On Wed, Jun 30, 2021 at 7:15 PM Pushkar Deole <pdeole2...@gmail.com> > >> wrote: > >>> > >>>> Hi, > >>>> > >>>> Our application uses kafka streams that reads from a source topic, > does > >>>> processing on records and produces processed record on destination > topic > >>>> through the use of external producer i.e. the producer created via > kafka > >>>> producer API. > >>>> > >>>> Does this model still guarantee exactly once semantic or it won't? > >>>> > >>>> Currently we are using at_least_once, however the question is how > >> streams > >>>> handles offset commits here? > >>>> Though the event is produced using synchronous API, could there be > >>>> possibility of event loss in case streams commit offset before > external > >>>> producer hasn't produced event on destination topic yet? > >>>> > >>> > >> > > >
