Re: Minimizing global store restoration time

Hi Guozhang I have created https://issues.apache.org/jira/browse/KAFKA-8023 and by accident found https://issues.apache.org/jira/browse/KAFKA-6721 which was what I was looking for at the beginning. Does this need a KIP? I can maybe help with the writeup but I am not sure I should help with the cod

Alternative of (kafka-consumer-groups.sh --describe) in java clients library to get consumer lag

Hi , I want to know if there is any alternative of (kafka-consumer-groups.sh --describe) in java clients library so that I can get consumer lag per client id. Regards, Arunim Chopra

Re: Alternative of (kafka-consumer-groups.sh --describe) in java clients library to get consumer lag

There is a tool called burrow that can do this. It requires a bit more setup and has a GUI web interface. https://github.com/linkedin/Burrow Get Outlook for Android From: Arunim Chopra Sent: Friday, March 1, 2019 3:46:59 AM To: users@kafk

Re: Alternative of (kafka-consumer-groups.sh --describe) in java clients library to get consumer lag

Hi Randall, I think Burrow tracks the consumer lags of group ids and not client ids within the group ids. I need the consumer lag of each client id as well like *kafka-consumer-groups.sh --describe *gives but I have not found any alternative of this command in java clients library. Regards, Arunim

Re: Alternative of (kafka-consumer-groups.sh --describe) in java clients library to get consumer lag

I see what you mean now. I have not seen anything like this as well. That would be very useful. On 3/1/19, 5:12 AM, "Arunim Chopra" wrote: Hi Randall, I think Burrow tracks the consumer lags of group ids and not client ids within the group ids. I need the consumer lag of each c

Re: Alternative of (kafka-consumer-groups.sh --describe) in java clients library to get consumer lag

Hi Randall, There are many monitoring tools which gives the exact information as *kafka-consumer-groups.sh --describe. *I do not know how they are doing it. I have explored Kafka Admin Client as well. There we have a *describeConsumerGroups() *api but it does not give us the consumer lag. I am able

Re: Errors when broker rejoins the cluster 5+ mins after clean shutdown

Hello, Marcos! Try to look at max.block.ms on producer config. I guess, you have to tune this parameter (in your case it should be something close to max.block.ms=36) Tolya > 27 февр. 2019 г., в 22:09, Marcos Juarez написал(а): > > We're doing some testing on Kafka 1.1 brokers in AWS EC2.

Kafka Streams Disk Usage on upgrade to 2.1.0

Hi, We are in the process of attempting to upgrade from Kafka Streams 1.1.0 to 2.1.0(-cp1) however we get some wildly different behaviour with regards to disk usage between these two versions. An update that uses existing state data exhibits the same behaviour as starting with a completely cle

Re: Minimizing global store restoration time

KAFKA-6721 should not need a KIP since it is for refactoring the internal classes only for cleaner code. However 8023 which is aimed for having multi-thread support would not be covered by 6721. Guozhang On Fri, Mar 1, 2019 at 12:15 AM Patrik Kl

Re: KTable.suppress(Suppressed.untilWindowCloses) does not suppress some non-final results when the kafka streams process is restarted

Hello John, hope you are well. I have tested the version 2.2 release candidate (although I know it has been postponed). I have been following this email thread because I think am experiencing the same issue. I have reported in an email to this list and also all the details are in OS ( https://stack

and known issues with --reset-offsets in 1.0.0

I am trying to reset some offsets for a consumer group in a QA environment using to-latest kafka-consumer-groups --bootstrap-server <>:9092 --topic Fraud --reset-offsets --to-latest --group fraudconsumer_fraud_mgroup I run it, it tells me the new offsets for the partitions, but then when the c

Re: Alternative of (kafka-consumer-groups.sh --describe) in java clients library to get consumer lag

Maybe you can take a look to https://github.com/yahoo/kafka-manager It allows you to get the info in a Webpage though. Cheers! -- Jonathan On Fri, Mar 1, 2019 at 1:19 PM Arunim Chopra wrote: > Hi Randall, > There are many monitoring tools which gives the exact information as > *kafka-consume

Re: KTable.suppress(Suppressed.untilWindowCloses) does not suppress some non-final results when the kafka streams process is restarted

BTW, after stopping the app gracefully (Stream#close()), this error shows up repeatedly: 2019-03-01 17:18:07,819 WARN [XXX-116ba7c8-678e-47f7-9074-7d03627b1e1a-StreamThread-1] internals.ProcessorStateManager (ProcessorStateManager.java:327) - task [0_0] Failed to write offset checkpoint file to [/

Re: Kafka Streams Disk Usage on upgrade to 2.1.0

Hello Adrian, What you described did sounds wired to me. I'm not aware of any regressions on rocksDB disk usage from 1.1 to 2.1. Could you file a JIRA ticket with more details like state dir snapshots, your code snippet and configs etc so we can find a way to reproduce it? Guozhang On Fri, Mar

[VOTE] 2.2.0 RC1

Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 2.2.0. - Added SSL support for custom principle name - Allow SASL connections to periodically re-authenticate - Improved consumer group management - default group.id is `null` inste

RE: Lag checking from producer

I worked on a project that needed backpressure for a variety of reasons & causes. We ended up needing to compute lag on all partitions, which was messy because of the boilerplate code implied in Filipp's 3rd option. I looked for a simple Kafka client API to get lags but did not find it. Filipp,

RE: Lag checking from producer

Typo: I meant to say " MetricName and Metric are not super clear parts of the API". -Original Message- From: Bill Gibson Sent: Friday, March 01, 2019 12:12 PM To: users@kafka.apache.org Subject: RE: Lag checking from producer I worked on a project that needed backpressure for a variety

Re: Kafka Streams Disk Usage on upgrade to 2.1.0

Hi Guozhang, thanks for your response. I have done some further investigations. The difference I see between the two versions is the following, in 1.1 this is the stat of the rocksdb MANIFEST files of one of the partitions: root@fooapp-6c4649dd68-wzrxk:/data# stat fooapp/2_5/rocksdb/foo-store/