Re: Kafka 2.0.0 - How to verify if Kafka compression is working

You can read the data from the disk and see compression type. https://thehoard.blog/how-kafkas-storage-internals-work-3a29b02e026 Thanks, Nitin On Wed, May 12, 2021 at 11:10 AM Shantanu Deshmukh wrote: > I am trying snappy compression on my producer. Here's my setup > > Kafka - 2.0.0 > Spring-K

Re: Two MirrorMakers 2 for two DCs

Did you keep the cluster name the same ? If yes, then it will cause conflict in metadata stored in MM2 internal topics. Thanks, Nitin On Mon, Sep 21, 2020 at 10:36 PM Oleg Osipov wrote: > Hello! > > I have two datacenters DC1 and DC2. When I deploy M2M in DC1 or DC2 all > things look correct. I

Re: partition.assignment.strategy for MM2?

If I understand correctly there is only one consumer object created per task, how would the setting of partition.assignment.strategy help here ? Thanks, Nitin On Sun, Sep 20, 2020 at 10:37 PM Ning Zhang wrote: > Here my initial thoughts: > > (1) MM2 assigned partitions over tasks based on round

Questions around Time based retention policy

Hi All, I have a few questions around time based retention policy. It will be really helpful if you can answer. 1. Does Kafka use CreateTime for building its time based indices? Does CreateTime have any role in deciding time based log rotation ? 2. If CreateTime is provided by the clients then wou

Re: MM2 max.request.size setting

e Engineer > Verizon - Protectwise > > 1601 Wewatta Street > #700 > Denver, Co > > > On Aug 31, 2020, at 1:08 PM, nitin agarwal > wrote: > > > > If you have clusters like DC1 and DC2 then you can add the following > > configuration: > > DC1->DC2.

Re: MM2 max.request.size setting

y > > Senior Infrastructure Engineer > Verizon - Protectwise > > 1601 Wewatta Street > #700 > Denver, Co > > > On Aug 31, 2020, at 1:04 PM, nitin agarwal > wrote: > > > > How are you running MM2 ? > > > > Thanks, > > Nitin > > > > On

Re: MM2 max.request.size setting

How are you running MM2 ? Thanks, Nitin On Mon, Aug 31, 2020 at 11:23 PM James Lavoy wrote: > Good day, > > I'm trying to use MM2 to setup replication between two Kafka clusters, I > have hit an issue that I believe is because of hitting the ProducerConfig > max.request.size setting, however I

How to update configurations in MM 2.0

Hi All, If we run MM 2.0 in a dedicated MirrorMaker cluster, how do we update a few properties like increasing the number of tasks, adding a topic in whitelist or blacklist etc. What I have observed is that if we update the properties in connect-mirror-maker.properties and restart the MM 2.0 node

Enabling Unclean leader election

Hi, Is there a way to enable Unclean leader election in Kafka without restarting the broker? We have a use case where we want to enable the Unclean leader election conditionally. Thanks, Nitin

Re: compression on topic vs producer?

to implement the producer level > compression, so I am now thinking about trying to implement topic level > compression as that should make the compression unrelated to mm2 itself. > > Thanks, > Iftach > > > On Thu, Jul 9, 2020 at 8:12 AM nitin agarwal > wrote: > > > Wh

Re: compression on topic vs producer?

What is the issue with compression at producer level in mm2 ? Can you please explain? Thanks, Nitin On Thu, Jul 9, 2020 at 10:37 AM Iftach Ben-Yosef wrote: > Hello, > > Up until now we have configured compression on producer level. Since moving > to mm2, we are having some issues with producer

Re: a few URP causes high latencies to producers and consumers

**BUMP** Thanks, Nitin On Thu, May 28, 2020 at 10:17 PM nitin agarwal wrote: > Hi, > > We have noticed that where a URPs in the cluster causes increase in > producer and consumer latencies. > The cause of the URP was either one of the broker went down or Kafka > rebalancer w

a few URP causes high latencies to producers and consumers

Hi, We have noticed that where a URPs in the cluster causes increase in producer and consumer latencies. The cause of the URP was either one of the broker went down or Kafka rebalancer was running. Is it the expected scenario ? Thanks, Nitin

Should there be separate Kafka for producers and consumers

Hi All, We have Camus consumer which reads data from Kafka and writes to HDFS. Whenever this consumer runs, there is impact on producer latencies. We don't want producer latencies to be increased in anyway. I want to know which design pattern is more suited for supporting this use case. 1) Single

Re: Cant Able to start Kafka MirrorMaker

There is typo in your consumer configuration, it should be auto.offset.reset. Thanks, Nitin On Fri, May 1, 2020 at 12:28 PM vishnu murali wrote: > Hey Guys, > > I am trying to move data between one cluster to another cluster > > > > *Source* > > *Destination* > > *Zookeeper* > > 2181 > > 2182 >

Re: Retention period for __consumer_offsets topic

**BUMP** Thanks, Nitin On Thu, Apr 16, 2020 at 6:51 PM nitin agarwal wrote: > Hi, > > What should be the retention period for __consumer_offsets topic? Should > it be the same as other Kafka topics? > > Thanks, > Nitin >

Retention period for __consumer_offsets topic

Hi, What should be the retention period for __consumer_offsets topic? Should it be the same as other Kafka topics? Thanks, Nitin

Re: Adding new connector Jars dynamically in Kafka Connect

**BUMP** On Tue, Apr 7, 2020 at 10:54 PM nitin agarwal wrote: > Hi, > > I have a use case where new connectors will keep on adding to existing > running Kafka Connect cluster. Is there any way in Kafka Connect to submit > the new connector jar dynamically without restarting the

Adding new connector Jars dynamically in Kafka Connect

Hi, I have a use case where new connectors will keep on adding to existing running Kafka Connect cluster. Is there any way in Kafka Connect to submit the new connector jar dynamically without restarting the Connect process? Thanks, Nitin