Re: Few partitions stuck in under replication

. How to resolve this split brain issue ? Thanks, Dhirendra On Wed, Mar 9, 2022 at 10:11 AM Dhirendra Singh wrote: > Some more information. > > It only happens when brokers lose connectivity to zookeeper and it results > in change in active controller. Issue does not occur always

Re: Few partitions stuck in under replication

> further? > > These ones: > > - kafka.zk.KafkaZkClient > - kafka.utils.ReplicationUtils > - kafka.server.ZkIsrManager > > > If you can do that and then share any log output that would be great :) > > Cheers, > > Liam > > On Tue, 8 Mar 202

Re: Few partitions stuck in under replication

issue is not resolved. Thanks, Dhirendra. On Mon, Mar 7, 2022 at 5:45 AM Liam Clarke-Hutchinson wrote: > Ah, I may have seen this error before. Dhirendra Singh, If you grep your > logs, you may find an IllegalStateException or two. > > https://issues.apache.org/jira/browse/KAFKA-12948

Re: Few partitions stuck in under replication

tions you are going to hit > issues. > > If you are running on Kubernetes I would suggest using an Operator like > Strimzi <https://strimzi.io/> which will do a lot of the Kafka admin > tasks like this for you automatically. > > Tom > > On 03/03/2022 16:28, Dhir

Re: Few partitions stuck in under replication

ff the rolling broker temporarily. > > Cheers, > > Tom Cooper > > [@tomncooper](https://twitter.com/tomncooper) | https://tomcooper.dev > > On 03/03/2022 07:38, Dhirendra Singh wrote: > > > Hi All, > > > > We have kafka cluster running in kubernetes. ka

Few partitions stuck in under replication

Hi All, We have kafka cluster running in kubernetes. kafka version we are using is 2.7.1. Every night zookeeper servers and kafka brokers are restarted. After the nightly restart of the zookeeper servers some partitions remain stuck in under replication. This happens randomly but not at every nigh

Re: count of number of producers and consumers in a kafka cluster

entation/#monitoring > > Thank you. > Luke > > > On Thu, Feb 3, 2022 at 1:54 PM Dhirendra Singh > wrote: > > > Hi All, > > does kafka have any metrics for count of number of producers and > consumers > > connected to a kafka cluster and any given time ? > > > > Thanks, > > Dhirendra. > > >

count of number of producers and consumers in a kafka cluster

Hi All, does kafka have any metrics for count of number of producers and consumers connected to a kafka cluster and any given time ? Thanks, Dhirendra.

Re: Log cleaner is marking the partition as uncleanable

Bytes (including CRC, Key, Value,...). I don't think it's good to fix > the corrupted file manually. > > Thank you. > Luke > > On Mon, Nov 15, 2021 at 1:31 PM Dhirendra Singh > wrote: > > > Hi All, > > I have a kafka cluster with 3 brokers

Log cleaner is marking the partition as uncleanable

Hi All, I have a kafka cluster with 3 brokers running in kubernetes. kafka version i am using is 2.7.1 Frequently we are observing that log cleaner is not able to comapct and clean log of some partitions. it mark the partition as uncleanable. Following is the error message we see in the log... [20

Log cleaner is not cleaning log of a partition

Hi All, I have a kafka cluster with 3 brokers. My kafka version is 2.5.0 I have a topic named "order" with 3 partitions and replication factor of 3. cleanup policy of the topic is "compact,delete" topic is evenly distributed across all brokers. broker 0 => partition 0,1,2 broker 1 => partition 0,

difference in disk utilization between replica and leader of a partition

Hi All, I have a kafka cluster running in kubernetes with 3 brokers. I have a topic with 3 partitions and replication factor of 3. Topic is evenly distributed across brokers. I have noticed that there is a huge difference in disk utilization of one partition between replica and leader. Ideally repl

Kafka metrics to calculate number of messages in a topic

Hi All, I have a requirement to display the total number of messages in a topic in grafana dashboard. I am looking at the metrics exposed by kafka broker and came across the following metrics. kafka_log_log_logendoffset kafka_log_log_logstartoffset My understanding is that if I take the difference

warning messages from kafka admin client

Hi All, whenever i run any kafka CLI tool i am getting following warning messages kafka-topics.sh --bootstrap-server kafka-dev:9092 --command-config client.properties --list [2021-07-14 07:02:31,595] WARN The configuration 'ssl.truststore.location' was supplied but isn't a known config. (org.apach

Re: Kafka ACL: topic with wildcard (*)

m>: > > > Hello 👋 in order to grant READ access to all topics like you did instead > > of just * , which in this case will take the files in current directory > and > > as the name of the topic, use "*" with the " before and after.. good luck >

Kafka ACL: topic with wildcard (*)

Hi All, I am trying to add an acl for a user to have read permission on all topics. I executed the following command. kafka-acls.sh --bootstrap-server kafka-dev:9092 --command-config client.properties --add --allow-principal User:kafka-consumer --operation read --topic * The output i got from the

command to get the value of a config from the broker

Hi All, I want to get the value of a config from broker. I do not have access to the server.properties file. is there any command available for it ? Thanks, Dhirendra.

Re: How to avoid storing password in clear text in server.properties file

; Kind regards, > > Tom > > On Tue, Jun 22, 2021 at 2:43 AM Dhirendra Singh > wrote: > > > I am using apache kafka. any such feature available in it ? > > > > On Mon, Jun 21, 2021 at 7:17 PM Ran Lupovich > > wrote: > > > > > Using Conf

Re: How to avoid storing password in clear text in server.properties file

ent/security/secrets.html > > בתאריך יום ב׳, 21 ביוני 2021, 16:26, מאת Dhirendra Singh ‏< > dhirendr...@gmail.com>: > > > Hi All, > > I am currently storing various passwords like "ssl.keystore.password", > > "ssl.truststore.password", SASL plain us

How to avoid storing password in clear text in server.properties file

Hi All, I am currently storing various passwords like "ssl.keystore.password", "ssl.truststore.password", SASL plain user password in cleartext in server.properties file. is there any way to store the password in encrypted text ? i am using kafka version 2.5.0

ssl principal mapping rule

Anyone please help me in understanding the kafka ssl principal mapping rule. for example in the following mapping rule RULE:^CN=(.*?),OU=(.*?),O=(.*?),L=(.*?),ST=(.*?),C=(.*?)$/$1@$2/, what is the significance of ".*?" inside parenthesis ? my distinguished name is "CN=kafka(S12345),OU=Services,DC=

Re: kafka 2 way ssl authentication

ot;keytool > -importkeystore -srckeystore server.p12 -destkeystore > kafka.server.keystore.jks -srcstoretype pkcs12" > > > בתאריך יום ו׳, 4 ביוני 2021, 07:40, מאת Dhirendra Singh ‏< > dhirendr...@gmail.com>: > > > I am trying to setup 2 way ssl authentication. My requireme

kafka 2 way ssl authentication

I am trying to setup 2 way ssl authentication. My requirement is broker should authenticate only specific clients. My organization has a CA which issue all certificates in pkcs12 format. steps i followed are as follows. 1. get a certificate for the broker and configured it in the broker keystore

Re: Producer batch size

ou effectively disable > batching, but may see improvements in latency. The differences can be very > large and noticeable to the application. > > > On Wed, Dec 23, 2020 at 11:48 PM Dhirendra Singh > wrote: > > > Thanks steve ! > > So if I understand correct

Re: Producer batch size

ed to allocate memory > within the configured max blocking time 6 ms > > The 60 seconds it is given to dutifully clear some room in buffer.memory by > successfully sending messages is controlled by max.block.ms. > > Thanks, > > Steve > > On Wed, Dec 23, 2020 at 12:11

Producer batch size

Hi, I have a question related to batch.size producer configuration. What happens when batch.size has reached and the producer app thread sends more data ? Does the thread block till space becomes available in the buffer containing the batch ? Thanks, Dhirendra.

Producer closed while allocating memory error

I am getting following error in kafka producer. org.apache.kafka.common.KafkaException: Producer closed while allocating memory at org.apache.kafka.clients.producer.internals.BufferPool.allocate(BufferPool.java:151) ~[kafka-clients-2.5.0.jar:?] at org.apache.kafka.clients.producer.internals.Reco

increase in latency with increase in number of partitions per broker

Hi, I am trying to measure throughput of my 3 broker kafka cluster. I have created a topic with 3 partitions spread equally over the 3 brokers. Run a producer application to produce messages. producer application has 3 thread (one for each partition) to produce. Then i increased the number of parti

Re: Kafka BrokerState Metric Value 3

t the underreplicated partition count metric. > > -- Peter (from phone) > > > On Aug 19, 2020, at 12:52 AM, Dhirendra Singh > wrote: > > > > So is this metric just gives information that broker process up and > running > > ? or does it indicate something more of br

Re: Kafka BrokerState Metric Value 3

> On Mon, Aug 17, 2020 at 3:29 PM Dhirendra Singh > wrote: > > > I have a question regarding Kafka BrokerState Metric value 3. According > to > > the documentation value 3 means running state. > > What does this running state mean for the broker? Does it mean data of > all

Kafka BrokerState Metric Value 3

I have a question regarding Kafka BrokerState Metric value 3. According to the documentation value 3 means running state. What does this running state mean for the broker? Does it mean data of all partitions on this broker is in sync ? Is it safe to assume that when broker transition to state 3 aft

RequestHandlerAvgIdlePercent Metric

I have all kafka broker metrics exported to prometheus and trying to build grafana dashboard. Kafka version i am using is 2.5.0 I am confused about kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent metric. according to kafka documentation it is "The average fraction of tim