Re: Imbalance in Commit Messages Across __consumer_offsets Topic Partitions

niformStickyPartitioner that is > being worked on right now, see > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner > > I hope this helps > > Kind regards, > > > Richard Bosch > > Developer Advocate > > Axual BV >

Imbalance in Commit Messages Across __consumer_offsets Topic Partitions

Hello Kafka users, I’m facing an issue with a Kafka cluster, specifically with the __consumer_offsets topic. There seems to be an imbalance in the number of commit messages across its partitions. Most of the commit messages are concentrated in a single partition, which is causing high CPU usage on

Delay issue when replicating topics creation with Kafka MM2 cluster

Hello, I am currently using Kafka mirror maker v2 v3.2.0 to replicate a Kafka cluster version 3.2.0 to another Kafka cluster version 3.2.0. However, I am experiencing a delay issue where it takes around 10 minutes for a topic to be created in the destination cluster after it has been created in th

Re: Kafka certificate monitoring

Hello Sandeep, When using Strimzi and Kafka I created a small app that parses the cert file and pushes the validity period as a prom metric. You can run the app periodically in a cron job, you would mount the secret within the job pod and parse the file. I was using prom push gateway for that bu

Re: Move all partitions from one log.dir to another

set of topics > at > a time and separating large topics. > > Yes, this is also the option I can think of. > Not sure if there are other suggestions from the community. > > Thank you. > Luke > > On Wed, Apr 13, 2022 at 4:09 PM Fares Oueslati > wrote: > >

Re: Move all partitions from one log.dir to another

(0.01 MB/sec), 82.3 ms avg latency, 739.0 ms max latency. 51 records sent, 10.0 records/sec (0.01 MB/sec), 4.7 ms avg latency, 52.0 ms max latency. Le mer. 13 avr. 2022 à 08:54, Fares Oueslati a écrit : > Hi Luke, > > For now I’m validating the operation on a 3 brokers test cluster wit

Re: Move all partitions from one log.dir to another

écrit : > Hi Fares, > > > Are you aware of any way to throttle the movement of data between disks? > > Interesting question! We've never considered the throttle in disk IO. > Does it impact the normal throughput a lot? > > Thank you. > Luke > > On Wed, Apr 13

Re: Move all partitions from one log.dir to another

know if there is other simpler solution, but I think the > `kafka-reassign-partitions` command is the safest way. > > > Thank you. > Luke > > On Wed, Apr 6, 2022 at 11:32 PM Fares Oueslati > wrote: > > > Hey 👋 > > I am using a jbod setup in a 2.8 Kafka clu

Move all partitions from one log.dir to another

Hey 👋 I am using a jbod setup in a 2.8 Kafka cluster. I started with only one disk in my JBOD, all partitions are on one volume (one log.dir) I have added a disk with the right log.dir and the brokers are well configured, I would like to move all replicas of all partitions without exception from

Re: Few partitions stuck in under replication

I don't know about the root cause, but if you're trying to solve the issue, restarting the controller broker pod should do the trick. Fares On Thu, Mar 3, 2022 at 8:38 AM Dhirendra Singh wrote: > Hi All, > > We have kafka cluster running in kubernetes. kafka version we are using is > 2.7.1. > E

Re: consumer hpa autoscaling

Hello You can look at keda.sh which allows to auto scale workloads on kubernetes according to many custom metrics including Kafka lag for instance. Fares Le mer. 2 mars 2022 à 01:07, David Ballano Fernandez < dfernan...@demonware.net> a écrit : > Thanks Liam, > > I am trying hpa but using cpu u

Topics naming convention

Hello, I'm here to get some advice regarding Kafka topics naming. If I name my topic "users" and I publish Avro events with a schema registry using a given compatibility mode. One day, due to business reasons, I have to break my schema, what would you do? add a new topic (let's call it "users-v1

Re: Question about log flusher real frequency

ctly *when* your data is on the disk? > > [1] https://www.kernel.org/doc/Documentation/sysctl/vm.txt > > Le lun. 9 mars 2020 à 15:58, Fares Oueslati a > écrit : > > > > Hello, > > > > By default, both log.flush.interval.ms and log.flush.interval.messages >

Question about log flusher real frequency

Hello, By default, both log.flush.interval.ms and log.flush.interval.messages are set to Long.MAX_VALUE. As I understand, it makes Kafka flush log to disk (fsync) only depends on file system. Is there any simple way to monitor that frequency ? Is there a rule of thumb to estimate that value dep

Re: Mistake in official documentation ?

t; > On Thu, Feb 6, 2020 at 8:25 AM Fares Oueslati > wrote: > > > Thanks for your quick feedback. > > > > Here is a link to the hosted screenshot https://we.tl/t-TDiRc7jOmV > > > > Regards, > > Fares > > > > > > > > On Thu, Feb

Re: Mistake in official documentation ?

ailing list will allow it to be attached. > > It could very well be a mistake. highlight it and send it to > d...@kafka.apache.org and if it is, we'll make a ticket and address it. > > On Thu, Feb 6, 2020 at 10:53 AM Fares Oueslati > wrote: > > > Hello, > > &g

Mistake in official documentation ?

Hello, While going through the official docs https://kafka.apache.org/documentation/#messageformat If I'm not wrong, I believe there is a mismatch between description of a segment and the diagram illustrating the concept. I pointed out the issue in the attached screenshot. Didn't really know wh