Re: Kafka Scala debugging and diagnostics

2018-05-16 Thread M. Manna
Must have missed that in Kafka-run-class.bat which sets suspend=n by default. Thanks for pointing that out, giving it a try. And didn On Wed, 16 May 2018 at 22:57, Ted Yu wrote: > Have you set the following two environment variables: > > export KAFKA_DEBUG=y; export DEBUG_SUSPEND_FLAG=y; > >

Re: Kafka Scala debugging and diagnostics

2018-05-16 Thread Ted Yu
Have you set the following two environment variables: export KAFKA_DEBUG=y; export DEBUG_SUSPEND_FLAG=y; The above would freeze the process until debugger is attached. Cheers On Wed, May 16, 2018 at 2:46 PM, M. Manna wrote: > Hello, > > Is there anyone who can provide a few steps for setting

Re: Kafka Scala debugging and diagnostics

2018-05-16 Thread Martin Gainty
supposedly scala ide for eclipse should work: http://scala-ide.org/ YMMV Martin From: M. Manna Sent: Wednesday, May 16, 2018 5:46 PM To: Kafka Users Subject: Kafka Scala debugging and diagnostics Hello, Is there anyone who can provide a few steps for settin

Kafka Scala debugging and diagnostics

2018-05-16 Thread M. Manna
Hello, Is there anyone who can provide a few steps for setting up Eclipse IDE to debug Kafka cluster ? Has anyone got eclipse/scala IDE experience in Kafka debugging ? I have been trying for days, but couldn’t start the debugger to connect to port 5005 when the Cluster is running. Everything els

Re: Producer#commitTransaction() Not Being Called if New Records Aren't Processed by StreamTask

2018-05-16 Thread Guozhang Wang
Thanks Matthias, I left a comment on KAFKA-6906 proposing a solution which I think should be sufficient for both. Guozhang On Tue, May 15, 2018 at 3:10 PM, Matthias J. Sax wrote: > Thanks for reporting this @David! > > @Guozhang: I actually think this is two different issues. This is also >

Issue with ISR Shrink and ISR Expand

2018-05-16 Thread Arunkumar
Hi All I am facing problem with ISR metrics. We have production cluster of 3 zookeeper and 3 broker. We have implemented custom metrics code(Also seeing the same on jvisualVM as well). When we initially start the brokers we dis not have any issue and it worked fine. But when we restarted a broke

Re: Issue with GlobalKTable on sink topic

2018-05-16 Thread Matthias J. Sax
Should be ok to read the topic. I cannot spot any error in your configs/program either. However, I am not entirely sure, if I understand the problem correctly. >> The problem is in the first run, where GlobalKTable reads null records (I >> have a json serializer and it reads a record with null v

Re: AW: Exception stopps data processing (Kafka Streams)

2018-05-16 Thread Matthias J. Sax
Depending on you version, that is correct :) And yes, you can use `ProducerConfigs.RETRIES_CONFIG`. At the end, the config only understand String names anyway and parameters like `ProducerConfigs.RETRIES_CONFIG` are just syntactic sugar to avoid types in putting "retries" manually. You might be i

lost committed offsets during reassignment

2018-05-16 Thread Girish Aher
Hello All, We have a 3 node kafka cluster (v0.10.1.1) which hosts about 8500 partitions with incoming byte rate of 25 MBps. We are currently in the process of expanding it to a 6 node cluster and doing the required data reassignment. In order the distribute the load, we reassigned the 50 partitio

Issue with GlobalKTable on sink topic

2018-05-16 Thread Daniele Tria
Dear community, I'm struggling with an issue regarding `GlobalKTable`: that is demanded to read from a topic A, where I add key-mapped records from another stream. This topic A is populated within the same application by ` stream.to` costruct with some events that I need to manipulate before they c

Kafka Consumer - WARN INVALID_FETCH_SESSION_EPOCH

2018-05-16 Thread SenthilKumar K
Hell All , Recently we have upgraded our brokers and clients to 1.1.0 and Consumer is frequently getting below error message. Node 1 was unable to process the fetch request with (sessionId=1388566672, epoch=5): INVALID_FETCH_SESSION_EPOCH. Broker is disconnecting clients. DEBUG Connection with

AW: Exception stopps data processing (Kafka Streams)

2018-05-16 Thread Claudia Wegmann
Thx for the pointers! I didn't set any of the conifg parameters you named. So retries defaults to 0. Another question though: There is no 'retries' config for streams, just 'retry.backoff.ms'. Do I set ProducerConfig.retries in my streams app? Also I do have to set 'max.in.flight.requests.per.con