Kafka Producer NetworkException and Timeout Exceptions

We are getting random NetworkExceptions and TimeoutExceptions in our production environment: Brokers: 3 Zookeepers: 3 Servers: 3 Kafka: 0.10.0.1 Zookeeeper: 3.4.3 We are occasionally getting this exception in my producer logs: Expiring 10 record(s) for TOPIC:XX: 5608 ms has passed since bat

Re: Client addressable listener

That did it, thanks so much! For what it's worth, the fix ended up being setting the same listener name and port but with an empty host for listeners and then reusing that same exact information but injecting the public DNS name in the advertised.listeners property. Thanks again! On 11/08/

Re: Kafka Monitoring..

Have you seen this thread ? http://search-hadoop.com/m/Kafka/uyzND1Q6wyNBj42g?subj=Kafka+Monitoring On Wed, Nov 8, 2017 at 5:10 PM, chidigam . wrote: > Hi All, > What is the simplest way of monitoring the metrics in kaka brokers? > Is there any opensource available? > Any help in this regards i

Kafka Monitoring..

Hi All, What is the simplest way of monitoring the metrics in kaka brokers? Is there any opensource available? Any help in this regards is appreciated. Regards Bhanu

Re: Kafka JVM heap limit

Yes, It is included in my spec. Open to use anything,it should be optimal for throughput. Regards bhanu On Wed, Nov 8, 2017 at 8:00 PM, Ted Yu wrote: > Did you use G1GC ? > Thanks > Original message From: John Yost > Date: 11/8/17 5:48 AM (GMT-08:00) To: users@kafka.apache.or

Re: GlobalKTable never finishes restoring

Issue created https://issues.apache.org/jira/browse/KAFKA-6190 I've not contributed anything to Kafka before, I've followed the guidelines in the contributing wiki except that I put the PR into 0.11.0 rather than trunk as that seemed to be what you were suggesting. Let me know if that's incorrect.

Re: Client addressable listener

You have the port ":9090" twice there and the "http://"; protocol should not be there. You have: listeners=CLIENT://:9090 http://my.public.dns.name:9090 ,PLAINTEXT://:9092 But it should be listeners=CLIENT://my.public.dns.name:9090,PLAINTEXT://:9092 Also as I mentioned before the DNS should

Re: Client addressable listener

Thank you for the reply! I think I'm doing something wrong. I tried using exactly verbatim what you had for listeners: listeners=CLIENT://:9090 ,PLAINTEXT://:9092 Only substituting 'my.public.dns.name` for my actual dns name. It seems as though it didn't like t

Re: Kafka Streams : CommitFailedException

Hello Guozhang, Thanks for the analysis. Figured out the reason for the OOM and it was actually caused by an in memory queue in the app itself. I have fixed it and right now it all looks good. Sorry for the inconvenience and thanks for helping out. Thanks, Tony On Wed, Nov 8, 2017 at 1:20 AM, Gu

Kafka 0.10.1.1 clients crash after node restart

Hi, We ran into a problem where clients crash after we restarted a node using kill -15 and then starting it using (broker 1001). 2 of the brokers including 1001 also can't sync with each other. Is this a known issue and if so, is it fixed in later versions? Details: We see logs similar to the

Re: Reg. Kafka transactional producer and consumer

Hi, Your log segment dump and the producer log don't correlate. The producer log shows the producerId == 4001. But your log segment dumps don't have this producerId. Please share data from the same run where you reproduce this issue. For the producerId's 0-4 (shown in the dump), there seem to be

RE: [Possibly spoofed] Re: 0.11.0.1: ReplicaFetcherThread exceptions after clearing corrupted broker data and restarting

Thanks Ismael, I hope this will solve the issue in the future. For now, unless we find a workaround soon, we'll probably backup the data we have and rebuild the cluster from scratch... Luc -Original Message- From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of Ismael Juma Sen

Re: 0.11.0.1: ReplicaFetcherThread exceptions after clearing corrupted broker data and restarting

Hi Luc, The first RC for 0.11.0.2 will be released this week. Ismael On Wed, Nov 8, 2017 at 3:33 PM, Vanlerberghe, Luc < luc.vanlerber...@bvdinfo.com> wrote: > Hi, > > We have a kafka setup with 6 brokers and topics having replication factor > 3 (single partition). > > After an improper shutdow

Re: GlobalKTable never finishes restoring

No problem, it was an interesting little bughunt. I'll create a JIRA and PR this afternoon/evening. Thanks Alex On Wed, Nov 8, 2017 at 3:05 PM Matthias J. Sax wrote: > Cool. Thanks for cycling back. What you describe verifies my suspicion. > Your patch make sense. > > Do you want to create a JI

0.11.0.1: ReplicaFetcherThread exceptions after clearing corrupted broker data and restarting

Hi, We have a kafka setup with 6 brokers and topics having replication factor 3 (single partition). After an improper shutdown, we had corrupted index files on two of our production servers, causing "WARN Found a corrupted index file due to requirement failed: Corrupt index found," messages an

Kafka 1.0.0 FATAL Shutdown broker - failed logs

Kafka 1.0.0  runs fine for a while, but then fails because of all log dirs failing. Deleting logs is the only way to start it again, however, that means all data are gone as well. Before 1.0.0 I used 0.10.2.1 and it never had any issues. I have reported this already, but would appreciate any ins

Re: GlobalKTable never finishes restoring

Cool. Thanks for cycling back. What you describe verifies my suspicion. Your patch make sense. Do you want to create a JIRA and provide a PR to fix this? We could include this into 0.11.0.2 that we plan to release soon (so if you want to contribute the patch, please do it right away -- we want to

Re: Kafka JVM heap limit

I did and it did not help. The heap size was the issue. --John On Wed, Nov 8, 2017 at 9:30 AM, Ted Yu wrote: > Did you use G1GC ? > Thanks > Original message From: John Yost > Date: 11/8/17 5:48 AM (GMT-08:00) To: users@kafka.apache.org Cc: > ja...@scholz.cz Subject: Re: Kaf

Re: Kafka JVM heap limit

Did you use G1GC ? Thanks Original message From: John Yost Date: 11/8/17 5:48 AM (GMT-08:00) To: users@kafka.apache.org Cc: ja...@scholz.cz Subject: Re: Kafka JVM heap limit In addition, in my experience, a memory heap > 8 GB leads to long GC pauses which causes the ISR statu

Re: Kafka connect python api

At the moment, no, there are no python connect implementations as of yet. I have been looking for a something similar. On Wednesday, November 8, 2017, 12:32:46 AM EST, swaapnika guntaka wrote: Hi Is there a python API for the source and sink connect implementation? My source is - Fil

Re: Kafka JVM heap limit

Hi there, There are some cases you may want to consider bigger heaps. Heroku runs some clusters with 20GB heaps now, because said clusters: 1. Use SSL exclusively for connectivity. SSL means message bytes traverse the JVM heap during encryption/etc, which uses more memory 2. Have a high number of

Re: Kafka JVM heap limit

In addition, in my experience, a memory heap > 8 GB leads to long GC pauses which causes the ISR statuses to constantly change, leading to an unstable cluster. --John On Wed, Nov 8, 2017 at 4:30 AM, chidigam . wrote: > Meaning, already read the doc, but couldn't relate, having large Heap for >

Re: [ANNOUNCE] New committer: Onur Karaman

Congratulations Onur!! On Wed, Nov 8, 2017 at 9:19 AM, UMESH CHAUDHARY wrote: > Congratulations Onur! > > On Tue, 7 Nov 2017 at 21:44 Jun Rao wrote: > > > Affan, > > > > All known problems in the controller are described in the doc linked from > > https://issues.apache.org/jira/browse/KAFKA-502

Re: Kafka JVM heap limit

Meaning, already read the doc, but couldn't relate, having large Heap for JVM will not help Now it all make sense. Many thanks. Bhanu On Wed, Nov 8, 2017 at 2:34 PM, chidigam . wrote: > Hi Jakub, > Thank you very much. I have read this concept I/O cache in design section. > But couldn't connect

Re: Kafka JVM heap limit

Hi Jakub, Thank you very much. I have read this concept I/O cache in design section. But couldn't connect the dots. Regards Bhanu On Wed, Nov 8, 2017 at 2:10 PM, Jakub Scholz wrote: > This is probably because Kafka uses quite heavily the disk cache maintained > by the operating system instead o

Re: Kafka JVM heap limit

This is probably because Kafka uses quite heavily the disk cache maintained by the operating system instead of storing messages in the JVM memory. So the requirements for the heap memory can be fairly small. The design section of the documentation describes the details: http://kafka.apache.org/docu

Kafka JVM heap limit

Hi All, I have basic question on Kafka JVM configuration, in most of forums I have seen max heap as 8GB. Why it is not recommended beyond that. Is there any design limitation ? Any help in this regards is highly appreciated. Regards Bhanu

Re: Client addressable listener

Try something like this: listeners=CLIENT://:9090 ,PLAINTEXT://:9092 advertised.listeners=CLIENT://my.public.dns.name:9090,PLAINTEXT://:9092 This will tell the listener to listen on your local ip addresses but to advertise the DNS name. Jakub On Tue, Nov 7, 2017