Re: MirrorMaker 2 and Negative Lag

Hello Frank, Thanks for helping on analyzing the issue. Regarding when the CG offsets at destination cluster will be updated. From the current codebase, there seems 2 criteria: (1) if the CG offsets do not contain a pair of , simply sync the offsets from source (2) for a pair of , if the conve

Re: Confluence Control Center On Windows 10

Hi, By the way why do you want it to setup on windows? You can setup and run it on linux and access the user interface from windows if you have firewall opened for running port. Default is 9021. On Fri, 26 Mar 2021 at 10:39 AM, Satendra Negi wrote: > Hello Guys, > > Is there any way to run the

Confluence Control Center On Windows 10

Hello Guys, Is there any way to run the confluent kafka control center on windows ? the binaries provided by confluent seems not to support the windows 10 by default. Thanks in advance. -- *Thanks & Regards*

Re: Kafka Streams Processor API state stores not restored via changelog topics

We have not tried running 2.6 brokers and 2.7 client, I will try and get back to you. We are not enabling EOS on the streams, we have it set to AT_LEAST_ONCE. The shutdowns and restarts of the stream app are clean each time. I see in the broker logs certain lines indicating that the log segment

Re: Kafka Streams Processor API state stores not restored via changelog topics

BTW, yes that indicates the record in the changelog was already truncated (logically). But since we only physically truncate logs by segments, which is 1GB by default, it should still be physically on the log. Are you enabling EOS on Streams, and when you shutdown the streams app, is that a clean s

Re: Kafka Streams Processor API state stores not restored via changelog topics

That's indeed weird. Have you tried to run Kafka brokers with 2.6 while Kafka Streams client with 2.7? On Thu, Mar 25, 2021 at 2:34 PM Upesh Desai wrote: > Hello Guozhang, > > > > I have tried your suggestions with an inMemoryStore FYI and seen the > following: > > > >1. I have the record a

Re: Kafka Streams Processor API state stores not restored via changelog topics

Hello Guozhang, I have tried your suggestions with an inMemoryStore FYI and seen the following: 1. I have the record added to the state store, stopped the application, and check the earliest and latest offsets via the command line tools. This shows that the earliest offset is 1, and the lat

Re: MirrorMaker 2 and Negative Lag

Thanks Alan for the investigation and bug report! On Thu, Mar 25, 2021, 3:25 PM Alan Ning wrote: > Another update on this. I am pretty sure I have found a bug in > MirrorSourceTask. The details are written in > https://issues.apache.org/jira/browse/KAFKA-12558. I hope this helps > others > who h

Re: Kafka Streams Processor API state stores not restored via changelog topics

Hello Upesh, Could you confirm a few more things for me: 1. After you stopped the application, and wiped out the state dir; check if the corresponding changelog topic has one record indeed at offset 0 --- this can be done via the admin#listOffsets (get the earliest and latest offset, which should

Re: MirrorMaker 2 and Negative Lag

Another update on this. I am pretty sure I have found a bug in MirrorSourceTask. The details are written in https://issues.apache.org/jira/browse/KAFKA-12558. I hope this helps others who have encountered this issue. ... Alan On Thu, Mar 25, 2021 at 9:40 AM Alan Ning wrote: > Just an update o

Re: Is it safe to delete old log segments manually?

In this case, yes, for any given topic-partition on the broker, you should be able to delete the oldest log segment, its associated index and timeindex files, and the snapshot file (which will be recreated on startup) in order to gain some free space. — Peter Bukowinski > On Mar 25, 2021, at 1

Re: Is it safe to delete old log segments manually?

Thank you for the response Peter. However, for us all the brokers are currently offline. So if I delete the entire topic-partition directory in one of the brokers, the first broker would start with no means to replicate the data which we just deleted. What are your thoughts on this? Do you think th

Re: Is it safe to delete old log segments manually?

Hi Sankalp, As long as you have replication, I’ve found it is safest to delete entire topic-partition directories than it is to delete individual log segments from them. For one, you get back more space. Second, you don’t have to worry about metadata corruption. When I’ve run out of disk space

Is it safe to delete old log segments manually?

Hi All, Brokers in one of our Apache Kafka clusters are continuously crashing as they have run out of disk space. As per my understanding, reducing the value of retention.ms and retention.bytes properties will not work because the broker is crashing before the log-retention thread can be scheduled

Re: MirrorMaker 2 and Negative Lag

Just an update on this. It has been an adventure. Sam, I think you are right that `consumer.auto.offset.reset:latest` does not work. However, I was also seeing issues with the default behavior (which is consumer.auto.offset.reset:earliest). After a lot of trial and error, I have found that my MM2