Thanks a lot for this Matthias. I have a follow up question. There is a COMMIT_INTERVAL_MS_CONFIG config for streams. This confuses things a little bit. If the value of this config is set to, say 100 ms, does it mean that the offset will be committed after 100 ms? If yes, then how does at least once guarantee work?
On Mon, Jan 30, 2017 at 1:43 PM, Matthias J. Sax <matth...@confluent.io> wrote: > Hi, > > yes, all examples have at-least-once semantics because this is the only > "mode" Kafka Streams supports -- you cannot "disable" it. (btw: we are > currently working on exactly-once for Streams that you will be able to > turn off/on). > > There is not much documentation about how it work internally, because it > uses the same mechanism Kafka consumer user to provide at-least-once > semantics and there is good documentation for this: > http://docs.confluent.io/current/clients/consumer.html > > The basic idea is, that input topic offsets get only committed after the > processing is completely finished and all intermediate state/data is > flushed to reliable storage -- to ensure nothing can get lost. If a > failure happens, just retry from last committed offset (and because some > output date might already be written which cannot be undone you might > get duplicates). > > -Matthias > > On 1/29/17 8:13 PM, Mahendra Kariya wrote: > > Hey All, > > > > I am new to Kafka streams. From the documentation > > <http://docs.confluent.io/3.1.0/streams/architecture.html# > processing-guarantees>, > > it is pretty much clear that streams support at least once semantics. > But I > > couldn't find details about how this is supported. I am interested in > > knowing the finer details / design of this. > > > > Is there some documentation around this? > > Is there some documentation around what semantics are followed by the > > various Kafka streams examples > > <https://github.com/confluentinc/examples/tree/3.1.x/kafka-streams> > > available on Github? Do all of them follow at least once? > > > > > > Thanks, > > Mahendra > > > >
