Here are logs: app: https://gist.github.com/dvsekhvalnov/f98afc3463f0c63b1722417e3710a8e7#file-kafka-streams-log brokers: https://gist.github.com/dvsekhvalnov/8e870f7347394e8d004c282880ef385a#file-kafka-broker-1-2-3-log
All broker logs are same, so single gist. There are also state-change.log files, do you want to take a look at those as well? On Wed, Jun 28, 2017 at 4:31 PM, Bill Bejeck <b...@confluent.io> wrote: > Hi Dmitry, > > At the moment I don't have anything specific to look for, just trying to > get more context around the issue. > > As for the logs maybe broker and streams logs for the last 30 minutes up to > the time the application stopped processing records. > > Thanks, > Bill > > On Wed, Jun 28, 2017 at 9:04 AM, Dmitriy Vsekhvalnov < > dvsekhval...@gmail.com > > wrote: > > > Hi Bill, > > > > 1. sure, can extract some logs, what exactly to look for? There are 11 > > hours of logs and most of them looks like: > > > > [2017-06-27 03:30:50,553] [] [INFO ] [StreamThread-1] > > [org.apache.kafka.streams.processor.internals.StreamThread] > [stream-thread > > [StreamThread-1] Committing all tasks because the commit interval 5000ms > > has elapsed] > > > > [2017-06-27 03:30:50,553] [] [INFO ] [StreamThread-1] > > [org.apache.kafka.streams.processor.internals.StreamThread] > [stream-thread > > [StreamThread-1] Committing task StreamTask 0_0] > > > > [2017-06-27 03:30:50,554] [] [INFO ] [StreamThread-1] > > [org.apache.kafka.streams.processor.internals.StreamThread] > [stream-thread > > [StreamThread-1] Committing task StreamTask 2_0] > > > > Something specific to search for? > > > > 2. Yes, there are more messages coming to topic. > > > > On Wed, Jun 28, 2017 at 3:43 PM, Bill Bejeck <b...@confluent.io> wrote: > > > > > Hi Dimitry, > > > > > > I'm happy to help, but I could use more information. Can you share the > > > streams logs and broker logs? > > > > > > Have you confirmed messages are still being delivered to topics (via > > > console consumer)? > > > > > > Thanks, > > > Bill > > > > > > On Wed, Jun 28, 2017 at 8:24 AM, Dmitriy Vsekhvalnov < > > > dvsekhval...@gmail.com > > > > wrote: > > > > > > > Hi all, > > > > > > > > looking for some assistance in debugging kafka-streams application. > > > > > > > > Kafka broker 0.10.2.1 - x3 Node cluster > > > > kafka-streams 0.10.2.1 - x2 application nodes x 1 stream thread > each. > > > > > > > > In streams configuration only: > > > > - SSL transport > > > > - kafka.streams.commitIntervalMs set to 5000 (instead of default > 30s). > > > > > > > > We running simple aggregation app with several grouping streams. > > Running > > > 2 > > > > instances of an app for redundancy. Both instances were working > pretty > > > fine > > > > for 11 hours 15 minutes then stopped consuming new events from topic. > > > > > > > > Hosting JVM processes were working fine, just streams stopped > reacting > > to > > > > new data. No exceptions, errors, e.t.c. in logs. After restart > streams > > > > still not consuming new messages. > > > > > > > > Below is 2 last entries from kafka-streams logs from both hosts: > > > > > > > > [2017-06-27 14:45:09,663] [] [INFO ] [StreamThread-1] > > > > [org.apache.kafka.streams.processor.internals.StreamThread] > > > [stream-thread > > > > [StreamThread-1] Committing task StreamTask 4_2] > > > > > > > > [2017-06-27 14:45:09,723] [] [INFO ] [StreamThread-1] > > > > [org.apache.kafka.streams.processor.internals.StreamThread] > > > [stream-thread > > > > [StreamThread-1] Committing task StreamTask 2_1] > > > > > > > > Pretty puzzling why they stopped exactly same moment (with respect to > > > > millis). > > > > > > > > Really appreciate any ideas where to dig to. > > > > > > > > Thank you. > > > > > > > > > >
