Nothing for stat-change.log for giving time window. Last line logged 4 hours before app stopped.
Any ideas so far? Personally i don't see anything relevant in logs. On Wed, Jun 28, 2017 at 6:33 PM, Bill Bejeck <b...@confluent.io> wrote: > Sure, couldn't hurt. > > Thanks, > Bill > > On Wed, Jun 28, 2017 at 9:51 AM, Dmitriy Vsekhvalnov < > dvsekhval...@gmail.com > > wrote: > > > Here are logs: > > > > app: > > https://gist.github.com/dvsekhvalnov/f98afc3463f0c63b1722417e3710a8 > > e7#file-kafka-streams-log > > brokers: > > https://gist.github.com/dvsekhvalnov/8e870f7347394e8d004c282880ef38 > > 5a#file-kafka-broker-1-2-3-log > > > > All broker logs are same, so single gist. > > > > There are also state-change.log files, do you want to take a look at > those > > as well? > > > > On Wed, Jun 28, 2017 at 4:31 PM, Bill Bejeck <b...@confluent.io> wrote: > > > > > Hi Dmitry, > > > > > > At the moment I don't have anything specific to look for, just trying > to > > > get more context around the issue. > > > > > > As for the logs maybe broker and streams logs for the last 30 minutes > up > > to > > > the time the application stopped processing records. > > > > > > Thanks, > > > Bill > > > > > > On Wed, Jun 28, 2017 at 9:04 AM, Dmitriy Vsekhvalnov < > > > dvsekhval...@gmail.com > > > > wrote: > > > > > > > Hi Bill, > > > > > > > > 1. sure, can extract some logs, what exactly to look for? There are > 11 > > > > hours of logs and most of them looks like: > > > > > > > > [2017-06-27 03:30:50,553] [] [INFO ] [StreamThread-1] > > > > [org.apache.kafka.streams.processor.internals.StreamThread] > > > [stream-thread > > > > [StreamThread-1] Committing all tasks because the commit interval > > 5000ms > > > > has elapsed] > > > > > > > > [2017-06-27 03:30:50,553] [] [INFO ] [StreamThread-1] > > > > [org.apache.kafka.streams.processor.internals.StreamThread] > > > [stream-thread > > > > [StreamThread-1] Committing task StreamTask 0_0] > > > > > > > > [2017-06-27 03:30:50,554] [] [INFO ] [StreamThread-1] > > > > [org.apache.kafka.streams.processor.internals.StreamThread] > > > [stream-thread > > > > [StreamThread-1] Committing task StreamTask 2_0] > > > > > > > > Something specific to search for? > > > > > > > > 2. Yes, there are more messages coming to topic. > > > > > > > > On Wed, Jun 28, 2017 at 3:43 PM, Bill Bejeck <b...@confluent.io> > > wrote: > > > > > > > > > Hi Dimitry, > > > > > > > > > > I'm happy to help, but I could use more information. Can you share > > the > > > > > streams logs and broker logs? > > > > > > > > > > Have you confirmed messages are still being delivered to topics > (via > > > > > console consumer)? > > > > > > > > > > Thanks, > > > > > Bill > > > > > > > > > > On Wed, Jun 28, 2017 at 8:24 AM, Dmitriy Vsekhvalnov < > > > > > dvsekhval...@gmail.com > > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > looking for some assistance in debugging kafka-streams > application. > > > > > > > > > > > > Kafka broker 0.10.2.1 - x3 Node cluster > > > > > > kafka-streams 0.10.2.1 - x2 application nodes x 1 stream thread > > > each. > > > > > > > > > > > > In streams configuration only: > > > > > > - SSL transport > > > > > > - kafka.streams.commitIntervalMs set to 5000 (instead of default > > > 30s). > > > > > > > > > > > > We running simple aggregation app with several grouping streams. > > > > Running > > > > > 2 > > > > > > instances of an app for redundancy. Both instances were working > > > pretty > > > > > fine > > > > > > for 11 hours 15 minutes then stopped consuming new events from > > topic. > > > > > > > > > > > > Hosting JVM processes were working fine, just streams stopped > > > reacting > > > > to > > > > > > new data. No exceptions, errors, e.t.c. in logs. After restart > > > streams > > > > > > still not consuming new messages. > > > > > > > > > > > > Below is 2 last entries from kafka-streams logs from both hosts: > > > > > > > > > > > > [2017-06-27 14:45:09,663] [] [INFO ] [StreamThread-1] > > > > > > [org.apache.kafka.streams.processor.internals.StreamThread] > > > > > [stream-thread > > > > > > [StreamThread-1] Committing task StreamTask 4_2] > > > > > > > > > > > > [2017-06-27 14:45:09,723] [] [INFO ] [StreamThread-1] > > > > > > [org.apache.kafka.streams.processor.internals.StreamThread] > > > > > [stream-thread > > > > > > [StreamThread-1] Committing task StreamTask 2_1] > > > > > > > > > > > > Pretty puzzling why they stopped exactly same moment (with > respect > > to > > > > > > millis). > > > > > > > > > > > > Really appreciate any ideas where to dig to. > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > >
