It’s hard to tell, the logs do not contain much, I agree. It could be a number of things.
If it’s happening as you say on restart as well (so it’s reproducible), any chance you could start streaming with DEBUG logs on and collect those logs? I’m hoping something shows up there. Thanks, Eno > On Jun 28, 2017, at 5:30 PM, Dmitriy Vsekhvalnov <dvsekhval...@gmail.com> > wrote: > > Nothing for stat-change.log for giving time window. Last line logged 4 > hours before app stopped. > > Any ideas so far? Personally i don't see anything relevant in logs. > > On Wed, Jun 28, 2017 at 6:33 PM, Bill Bejeck <b...@confluent.io> wrote: > >> Sure, couldn't hurt. >> >> Thanks, >> Bill >> >> On Wed, Jun 28, 2017 at 9:51 AM, Dmitriy Vsekhvalnov < >> dvsekhval...@gmail.com >>> wrote: >> >>> Here are logs: >>> >>> app: >>> https://gist.github.com/dvsekhvalnov/f98afc3463f0c63b1722417e3710a8 >>> e7#file-kafka-streams-log >>> brokers: >>> https://gist.github.com/dvsekhvalnov/8e870f7347394e8d004c282880ef38 >>> 5a#file-kafka-broker-1-2-3-log >>> >>> All broker logs are same, so single gist. >>> >>> There are also state-change.log files, do you want to take a look at >> those >>> as well? >>> >>> On Wed, Jun 28, 2017 at 4:31 PM, Bill Bejeck <b...@confluent.io> wrote: >>> >>>> Hi Dmitry, >>>> >>>> At the moment I don't have anything specific to look for, just trying >> to >>>> get more context around the issue. >>>> >>>> As for the logs maybe broker and streams logs for the last 30 minutes >> up >>> to >>>> the time the application stopped processing records. >>>> >>>> Thanks, >>>> Bill >>>> >>>> On Wed, Jun 28, 2017 at 9:04 AM, Dmitriy Vsekhvalnov < >>>> dvsekhval...@gmail.com >>>>> wrote: >>>> >>>>> Hi Bill, >>>>> >>>>> 1. sure, can extract some logs, what exactly to look for? There are >> 11 >>>>> hours of logs and most of them looks like: >>>>> >>>>> [2017-06-27 03:30:50,553] [] [INFO ] [StreamThread-1] >>>>> [org.apache.kafka.streams.processor.internals.StreamThread] >>>> [stream-thread >>>>> [StreamThread-1] Committing all tasks because the commit interval >>> 5000ms >>>>> has elapsed] >>>>> >>>>> [2017-06-27 03:30:50,553] [] [INFO ] [StreamThread-1] >>>>> [org.apache.kafka.streams.processor.internals.StreamThread] >>>> [stream-thread >>>>> [StreamThread-1] Committing task StreamTask 0_0] >>>>> >>>>> [2017-06-27 03:30:50,554] [] [INFO ] [StreamThread-1] >>>>> [org.apache.kafka.streams.processor.internals.StreamThread] >>>> [stream-thread >>>>> [StreamThread-1] Committing task StreamTask 2_0] >>>>> >>>>> Something specific to search for? >>>>> >>>>> 2. Yes, there are more messages coming to topic. >>>>> >>>>> On Wed, Jun 28, 2017 at 3:43 PM, Bill Bejeck <b...@confluent.io> >>> wrote: >>>>> >>>>>> Hi Dimitry, >>>>>> >>>>>> I'm happy to help, but I could use more information. Can you share >>> the >>>>>> streams logs and broker logs? >>>>>> >>>>>> Have you confirmed messages are still being delivered to topics >> (via >>>>>> console consumer)? >>>>>> >>>>>> Thanks, >>>>>> Bill >>>>>> >>>>>> On Wed, Jun 28, 2017 at 8:24 AM, Dmitriy Vsekhvalnov < >>>>>> dvsekhval...@gmail.com >>>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> looking for some assistance in debugging kafka-streams >> application. >>>>>>> >>>>>>> Kafka broker 0.10.2.1 - x3 Node cluster >>>>>>> kafka-streams 0.10.2.1 - x2 application nodes x 1 stream thread >>>> each. >>>>>>> >>>>>>> In streams configuration only: >>>>>>> - SSL transport >>>>>>> - kafka.streams.commitIntervalMs set to 5000 (instead of default >>>> 30s). >>>>>>> >>>>>>> We running simple aggregation app with several grouping streams. >>>>> Running >>>>>> 2 >>>>>>> instances of an app for redundancy. Both instances were working >>>> pretty >>>>>> fine >>>>>>> for 11 hours 15 minutes then stopped consuming new events from >>> topic. >>>>>>> >>>>>>> Hosting JVM processes were working fine, just streams stopped >>>> reacting >>>>> to >>>>>>> new data. No exceptions, errors, e.t.c. in logs. After restart >>>> streams >>>>>>> still not consuming new messages. >>>>>>> >>>>>>> Below is 2 last entries from kafka-streams logs from both hosts: >>>>>>> >>>>>>> [2017-06-27 14:45:09,663] [] [INFO ] [StreamThread-1] >>>>>>> [org.apache.kafka.streams.processor.internals.StreamThread] >>>>>> [stream-thread >>>>>>> [StreamThread-1] Committing task StreamTask 4_2] >>>>>>> >>>>>>> [2017-06-27 14:45:09,723] [] [INFO ] [StreamThread-1] >>>>>>> [org.apache.kafka.streams.processor.internals.StreamThread] >>>>>> [stream-thread >>>>>>> [StreamThread-1] Committing task StreamTask 2_1] >>>>>>> >>>>>>> Pretty puzzling why they stopped exactly same moment (with >> respect >>> to >>>>>>> millis). >>>>>>> >>>>>>> Really appreciate any ideas where to dig to. >>>>>>> >>>>>>> Thank you. >>>>>>> >>>>>> >>>>> >>>> >>> >>
