Re: Kafka streams DSL going into infinite loop

2020-01-30 Thread Matthias J. Sax
>> I really don't know what TOPOLOGY_OPTIMIZATION is for. If you enable optimization, Kafka Streams tries to generate a more efficient Topology when translating the DSL program. We are working on some more documentation of this feature atm: https://cwiki.apache.org/confluence/display/KAFKA/DSL+Opt

Re: Kafka streams DSL going into infinite loop

2020-01-30 Thread Sachin Mittal
Only streams specific props I am using are: props.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0); props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000); props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION, StreamsConfig.OPTIMIZE); Yes there are three sub topologies and key does change betwee

Re: Kafka streams DSL going into infinite loop

2020-01-30 Thread Matthias J. Sax
Your program does not seem to contain a loop. Hence, it's unclear to me atm what could be the issue. Does the application commit offset on the input topic, what would be an indicator that it actually does make progress. Do you change the key between joins, ie, is there multiple sub-topologies tha

Kafka streams DSL going into infinite loop

2020-01-30 Thread Sachin Mittal
Hi, In a case I have found that when I define my topology using streams DSL it tends to go into infinite loop. This usually happens if I start my stream and shut it down and restart it again after a while (by that time source topic has moved ahead). Stream processing seems to be stuck in a loop an