Yeah. Quite possible. Completely missed this possibility. What I simply did was to download and add the kafka-streams jar as a dependency. I didn't update the downstream dependencies. My bad!
On Tue, Apr 18, 2017 at 7:42 PM, Eno Thereska <eno.there...@gmail.com> wrote: > Hi Mahendra, > > I see the java.lang.NoSuchMethodError: org.apache.kafka.clients... error. > Looks like some jars aren't in the classpath? > > Eno > > > On 18 Apr 2017, at 12:46, Mahendra Kariya <mahendra.kar...@go-jek.com> > wrote: > > > > Hey Eno, > > > > I just pulled the latest jar from the link you shared and tried to run my > > code. I am getting the following exception on new KafkaStreams(). The > same > > code is working fine with 0.10.2.0 jar. > > > > > > Exception in thread "main" org.apache.kafka.common.KafkaException: > Failed > > to construct kafka consumer > > at org.apache.kafka.clients.consumer.KafkaConsumer.<init>( > > KafkaConsumer.java:717) > > at org.apache.kafka.clients.consumer.KafkaConsumer.<init>( > > KafkaConsumer.java:566) > > at org.apache.kafka.streams.processor.internals. > > DefaultKafkaClientSupplier.getConsumer(DefaultKafkaClientSupplier. > java:38) > > at org.apache.kafka.streams.processor.internals. > StreamThread.<init>( > > StreamThread.java:316) > > at org.apache.kafka.streams.KafkaStreams.<init>( > > KafkaStreams.java:358) > > at org.apache.kafka.streams.KafkaStreams.<init>( > > KafkaStreams.java:279) > > Caused by: java.lang.NoSuchMethodError: org.apache.kafka.clients. > > Metadata.update(Lorg/apache/kafka/common/Cluster;Ljava/util/Set;J)V > > at org.apache.kafka.streams.processor.internals. > > StreamsKafkaClient.<init>(StreamsKafkaClient.java:98) > > at org.apache.kafka.streams.processor.internals. > > StreamsKafkaClient.<init>(StreamsKafkaClient.java:82) > > at org.apache.kafka.streams.processor.internals. > > StreamPartitionAssignor.configure(StreamPartitionAssignor.java:219) > > at org.apache.kafka.common.config.AbstractConfig. > > getConfiguredInstances(AbstractConfig.java:254) > > at org.apache.kafka.common.config.AbstractConfig. > > getConfiguredInstances(AbstractConfig.java:220) > > at org.apache.kafka.clients.consumer.KafkaConsumer.<init>( > > KafkaConsumer.java:673) > > ... 6 more > > > > > > > > On Tue, Apr 18, 2017 at 5:47 AM, Mahendra Kariya < > mahendra.kar...@go-jek.com > >> wrote: > > > >> Thanks! > >> > >> On Tue, Apr 18, 2017, 12:26 AM Eno Thereska <eno.there...@gmail.com> > >> wrote: > >> > >>> The RC candidate build is here: http://home.apache.org/~ > >>> gwenshap/kafka-0.10.2.1-rc1/ <http://home.apache.org/~ > >>> gwenshap/kafka-0.10.2.1-rc1/> > >>> > >>> Eno > >>>> On 17 Apr 2017, at 17:20, Mahendra Kariya <mahendra.kar...@go-jek.com > > > >>> wrote: > >>>> > >>>> Thanks! > >>>> > >>>> In the meantime, is the jar published somewhere on github or as a part > >>> of > >>>> build pipeline? > >>>> > >>>> On Mon, Apr 17, 2017 at 9:18 PM, Eno Thereska <eno.there...@gmail.com > > > >>>> wrote: > >>>> > >>>>> Not yet, but as soon as 0.10.2 is voted it should be. Hopefully this > >>> week. > >>>>> > >>>>> Eno > >>>>>> On 17 Apr 2017, at 13:25, Mahendra Kariya < > mahendra.kar...@go-jek.com > >>>> > >>>>> wrote: > >>>>>> > >>>>>> Are the bug fix releases published to Maven central repo? > >>>>>> > >>>>>> On Sat, Apr 1, 2017 at 12:26 PM, Eno Thereska < > eno.there...@gmail.com > >>>> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi Sachin, > >>>>>>> > >>>>>>> In the bug fix release for 0.10.2 (and in trunk) we have now set > >>>>>>> max.poll.interval to infinite since from our experience with > streams > >>>>> this > >>>>>>> should not be something that users set: https://github.com/apache/ > >>>>>>> kafka/pull/2770/files <https://github.com/apache/ > >>> kafka/pull/2770/files > >>>>>> . > >>>>>>> > >>>>>>> We're in the process of documenting that change. For now you can > >>>>> increase > >>>>>>> the request timeout without worrying about max.poll.interval > >>> anymore. In > >>>>>>> fact I'd suggest you also increase max.poll.interval as we've done > it > >>>>> above. > >>>>>>> > >>>>>>> Thanks > >>>>>>> Eno > >>>>>>> > >>>>>>>> On 1 Apr 2017, at 03:28, Sachin Mittal <sjmit...@gmail.com> > wrote: > >>>>>>>> > >>>>>>>> Should this timeout be less than max poll interval value? if yes > >>> than > >>>>>>>> generally speaking what should be the ratio between two or range > for > >>>>> this > >>>>>>>> timeout value . > >>>>>>>> > >>>>>>>> Thanks > >>>>>>>> Sachin > >>>>>>>> > >>>>>>>> On 1 Apr 2017 04:57, "Matthias J. Sax" <matth...@confluent.io> > >>> wrote: > >>>>>>>> > >>>>>>>> Yes, you can increase ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG > >>>>>>>> > >>>>>>>> > >>>>>>>> -Matthias > >>>>>>>> > >>>>>>>> > >>>>>>>> On 3/31/17 11:32 AM, Sachin Mittal wrote: > >>>>>>>>> Hi, > >>>>>>>>> So I have added the config ProducerConfig.RETRIES_CONFIG, > >>>>>>>> Integer.MAX_VALUE > >>>>>>>>> and the NotLeaderForPartitionException is gone. > >>>>>>>>> > >>>>>>>>> However we see a new exception especially under heavy load: > >>>>>>>>> org.apache.kafka.streams.errors.StreamsException: task [0_1] > >>>>> exception > >>>>>>>>> caught when producing > >>>>>>>>> at > >>>>>>>>> org.apache.kafka.streams.processor.internals. > RecordCollectorImpl. > >>>>>>>> checkForException(RecordCollectorImpl.java:119) > >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>> at > >>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>> RecordCollectorImpl.flush( > >>>>>>>> RecordCollectorImpl.java:127) > >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] at > >>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>> StreamTask$1.run(StreamTask. > >>>>>>>> java:76) > >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>> at > >>>>>>>>> org.apache.kafka.streams.processor.internals.StreamsMetricsImpl. > >>>>>>>> measureLatencyNs(StreamsMetricsImpl.java:188) > >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>> at > >>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>> StreamTask.commit(StreamTask. > >>>>>>>> java:280) > >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] at > >>>>>>>>> org.apache.kafka.streams.processor.internals. > >>> StreamThread.commitOne( > >>>>>>>> StreamThread.java:787) > >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>> at > >>>>>>>>> org.apache.kafka.streams.processor.internals. > >>> StreamThread.commitAll( > >>>>>>>> StreamThread.java:774) > >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] at > >>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>> StreamThread.maybeCommit( > >>>>>>>> StreamThread.java:749) > >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>> at > >>>>>>>>> org.apache.kafka.streams.processor.internals. > StreamThread.runLoop( > >>>>>>>> StreamThread.java:671) > >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] at > >>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>> StreamThread.run(StreamThread.java:378) > >>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>> org.apache.kafka.common.errors.TimeoutException: Expiring 1 > >>> record(s) > >>>>>>> for > >>>>>>>>> new-part-advice-key-table-changelog-1: 30001 ms has passed since > >>> last > >>>>>>>> append > >>>>>>>>> > >>>>>>>>> So any idea as why TimeoutException is happening. > >>>>>>>>> Is this controlled by > >>>>>>>>> ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG > >>>>>>>>> > >>>>>>>>> If yes > >>>>>>>>> What should the value be set in this given that out consumer > >>>>>>>>> max.poll.interval.ms is defaul 5 minutes. > >>>>>>>>> > >>>>>>>>> Is there any other setting that we should try to avoid such > errors > >>>>> which > >>>>>>>>> causes stream thread to die. > >>>>>>>>> > >>>>>>>>> Thanks > >>>>>>>>> Sachin > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Sun, Mar 26, 2017 at 1:39 AM, Eno Thereska < > >>> eno.there...@gmail.com > >>>>>> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi Sachin, > >>>>>>>>>> > >>>>>>>>>> Not in this case. > >>>>>>>>>> > >>>>>>>>>> Thanks > >>>>>>>>>> Eno > >>>>>>>>>> > >>>>>>>>>>> On Mar 25, 2017, at 6:19 PM, Sachin Mittal <sjmit...@gmail.com > > > >>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> OK. > >>>>>>>>>>> I will try this out. > >>>>>>>>>>> > >>>>>>>>>>> Do I need to change anything for > >>>>>>>>>>> max.in.flight.requests.per.connection > >>>>>>>>>>> > >>>>>>>>>>> Thanks > >>>>>>>>>>> Sachin > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Sat, Mar 25, 2017 at 10:59 PM, Eno Thereska < > >>>>>>> eno.there...@gmail.com> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi Sachin, > >>>>>>>>>>>> > >>>>>>>>>>>> For this particular error, “org.apache.kafka.common.errors. > >>>>>>>>>>>> NotLeaderForPartitionException: This server is not the leader > >>> for > >>>>>>> that > >>>>>>>>>>>> topic-partition.”, could you try setting the number of retries > >>> to > >>>>>>>>>> something > >>>>>>>>>>>> large like this: > >>>>>>>>>>>> > >>>>>>>>>>>> Properties props = new Properties(); > >>>>>>>>>>>> props.put(StreamsConfig.APPLICATION_ID_CONFIG, > applicationId); > >>>>>>>>>>>> ... > >>>>>>>>>>>> props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE); > >>>>>>>>>>>> > >>>>>>>>>>>> This will retry the produce requests and should hopefully > solve > >>>>> your > >>>>>>>>>>>> immediate problem. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks > >>>>>>>>>>>> Eno > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On 25/03/2017, 08:35, "Sachin Mittal" <sjmit...@gmail.com> > >>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Hi, > >>>>>>>>>>>> We have encountered another case of series of errors which I > >>> would > >>>>>>>>>> need > >>>>>>>>>>>> more help in understanding. > >>>>>>>>>>>> > >>>>>>>>>>>> In logs we see message like this: > >>>>>>>>>>>> ERROR 2017-03-25 03:41:40,001 [kafka-producer-network-thread | > >>>>>>>>>>>> 85-StreamThread-3-producer]: > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>> RecordCollectorImpl > >>>>> - > >>>>>>>>>>>> task > >>>>>>>>>>>> [0_1] Error sending record to topic new-part-advice-key-table- > >>>>>>>>>> changelog. > >>>>>>>>>>>> No > >>>>>>>>>>>> more offsets will be recorded for this task and the exception > >>> will > >>>>>>>>>>>> eventually be thrown > >>>>>>>>>>>> > >>>>>>>>>>>> then some millisecond later > >>>>>>>>>>>> ERROR 2017-03-25 03:41:40,149 [StreamThread-3]: > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals.StreamThread - > >>>>>>>>>>>> stream-thread > >>>>>>>>>>>> [StreamThread-3] Failed while executing StreamTask 0_1 due to > >>>>> flush > >>>>>>>>>>>> state: > >>>>>>>>>>>> org.apache.kafka.streams.errors.StreamsException: task [0_1] > >>>>>>>>>> exception > >>>>>>>>>>>> caught when producing > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>> RecordCollectorImpl. > >>>>>>>>>>>> checkForException(RecordCollectorImpl.java:119) > >>>>>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>> RecordCollectorImpl.flush(RecordCollectorImpl.java:127) > >>>>>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>> StreamTask.flushState( > >>>>>>>>>>>> StreamTask.java:422) > >>>>>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>> StreamThread$4.apply( > >>>>>>>>>>>> StreamThread.java:555) > >>>>>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals.StreamThread. > >>>>>>>>>>>> performOnAllTasks(StreamThread.java:513) > >>>>>>>>>>>> [kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>> StreamThread.flushAllState(StreamThread.java:551) > >>>>>>>>>>>> [kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals.StreamThread. > >>>>>>>>>>>> shutdownTasksAndState(StreamThread.java:463) > >>>>>>>>>>>> [kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>> StreamThread.shutdown( > >>>>>>>>>>>> StreamThread.java:408) > >>>>>>>>>>>> [kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>> StreamThread.run(StreamThread.java:389) > >>>>>>>>>>>> [kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> org.apache.kafka.common.errors.NotLeaderForPartitionException > : > >>>>>>> This > >>>>>>>>>>>> server > >>>>>>>>>>>> is not the leader for that topic-partition. > >>>>>>>>>>>> > >>>>>>>>>>>> finally we get this > >>>>>>>>>>>> ERROR 2017-03-25 03:41:45,724 [StreamThread-3]: > >>>>>>>>>>>> com.advice.TestKafkaAdvice > >>>>>>>>>>>> - Uncaught exception: > >>>>>>>>>>>> org.apache.kafka.streams.errors.StreamsException: Exception > >>>>> caught > >>>>>>>>>> in > >>>>>>>>>>>> process. taskId=0_1, processor=KSTREAM-SOURCE-0000000000, > >>>>>>>>>>>> topic=advice-stream, partition=1, offset=48062286 > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>> StreamTask.process(StreamTask.java:216) > >>>>>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>> StreamThread.runLoop( > >>>>>>>>>>>> StreamThread.java:651) > >>>>>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>>>> StreamThread.run(StreamThread.java:378) > >>>>>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> Caused by: org.apache.kafka.streams.errors.StreamsException: > >>> task > >>>>>>>>>>>> [0_1] > >>>>>>>>>>>> exception caught when producing > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>> RecordCollectorImpl. > >>>>>>>>>>>> checkForException(RecordCollectorImpl.java:119) > >>>>>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>> RecordCollectorImpl.send( > >>>>>>>>>>>> RecordCollectorImpl.java:76) > >>>>>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> at > >>>>>>>>>>>> org.apache.kafka.streams.processor.internals. > >>>>>>>>>> RecordCollectorImpl.send( > >>>>>>>>>>>> RecordCollectorImpl.java:64) > >>>>>>>>>>>> ~[kafka-streams-0.10.2.1-SNAPSHOT.jar:na] > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Again it is not clear why in this case we need to shut down > the > >>>>>>>>>> steams > >>>>>>>>>>>> thread and eventually the application. Shouldn't we capture > this > >>>>>>>>>> error > >>>>>>>>>>>> too? > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks > >>>>>>>>>>>> Sachin > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>>> > >>>>> > >>>>> > >>> > >>> > >
