Re: max.task.idle.ms behavior

Hi Matthias, Thank you for the clarifications. May I ask about some more detail? I'll try to describe the behavior as I understand it. Please correct me if I'm wrong... So in effect each task alternates between NORMAL and ENFORCED processing modes. It starts in NORMAL mode where it is perf

Quickstart: Error with MinGW and two minor suggestions

Hi,   I'm just working through the Quickstart page. https://kafka.apache.org/quickstart   As I'm working on Windows, I use MinGW to emulate Linux. 1) On MinGW, I get this error: me@computer MINGW64 /c/java/kafka_2.12 $ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test log4j:ER

Kafka topics lost after system reboot

Hello, I have a 3 node,3 zookeeper Kafka cluster of a version 0.11.x.x I could see everytime after a reboot,all of my topics are lost. I found in Google that the data is lost if it's stored in /TMP/Kafka-logs directory But in our case the Kafka and the zookeeper data is stored in one of our persis

Using Kafka to replace local .net message queues

Hello, We are planning to use Apache Kafka in our system to replace local MSMQs (.net message queues). *Single Environment* Currently, we have about 100 message queues per environment. In Kafka, we plan to use approximately 20 topics to accommodate the 100 message queues. We will have 100

Re: max.task.idle.ms behavior

That sounds correct. Just to avoid confusion: we don't use the term "pause" for tasks, but "idle", because consumers can "pause/resume" partitions what is something different. In fact, if a task is idle the underlying empty partitions are not paused, because we hope to get data from them before t

Re: Kafka Stream Best Node Module in NodeJs

Hi Marimuthu, I do not know the answer to your first question.. As for the second one, my understanding is that you want to have multi-threading to processing a single Kafka topic-partition (since Streams by default have each Kafka topic-partition processed by a single stream thread), that is doab

Re: Using Kafka to replace local .net message queues

Hi Akshay, In regards to your 3rd question (and indirectly to your 2nd question), instead of having different consumer groups, why not just multiple consumers in the same group? That would ensure that each consumer only reads from one partition in the topic. You can even assign the partition if yo

Best way to monitor and alert Consumer lag

Hello Users, Can any one of you let me know good exporter which exposes kafka consumer lag metrics. Regards, Ramm.

Re: Best way to monitor and alert Consumer lag

Prometheus.. Kafka consumer exporter is good ... https://github.com/braedon/prometheus-kafka-consumer-group-exporter/blob/master/README.md you can also find Grafana dashboard .. alert rules using alert manager .. > On Jan 18, 2019, at 8:59 PM, Rammohan Vanteru wrote: > > Hello Users, > > C