It is not a kafka streams error, it is an application level error e.g. say, some data required for processing an input event is not available in the GlobalKTable since it is not yet synced with the global topic
On Mon, Sep 21, 2020 at 4:54 PM Bruno Cadonna <br...@confluent.io> wrote: > Hi Pushkar, > > Is the error you are talking about, one that is thrown by Kafka Streams > or by your application? If it is thrown by Kafka Streams, could you > please post the error? > > I do not completely understand what you are trying to achieve, but maybe > max.task.idle.ms [1] is the configuration you are looking for. > > I can assure you that enable.auto.commit is false in Kafka Streams. What > you probably mean is that Kafka Streams periodically commits the > offsets. The commit interval can be controlled with commit.interval.ms > [2]. > > > Best, > Bruno > > > [1] https://kafka.apache.org/documentation/#max.task.idle.ms > [2] https://kafka.apache.org/documentation/#commit.interval.ms > > On 21.09.20 12:38, Pushkar Deole wrote: > > Hi, > > > > I would like to know how to handle following scenarios while processing > > events in a kafka streams application: > > > > 1. the streams application needs data from a globalKtable which loads it > > from a topic that is populated by some other service/application. So, if > > the streams application starts getting events from input source topic > > however it doesn't find required data in GlobalKTable since that other > > application/service hasn't yet loaded that data then the Kafka streams > > application gets error while processing the event and application handles > > the exception by logging an error and it goes onto processing other > > events. Since auto.commit is true, the polling will go on fetching next > > batch and probably it will set the offset of previous batch, causing loss > > of events that had an exception while processing. > > > > I want to halt the processing here if an error occurs while processing > the > > event, so instead of going on to the next event, the processing should > keep > > trying previous event until application level error is resolved. How can > I > > achieve this? > > >
