Re: Monitoring Kafka client

2016-09-02 Thread Otis Gospodnetić
Hi, By "kafka client" I assume you mean you Kafka producer and/or consumers? If so, any decent Kafka monitoring solution should let you monitor that. See https://sematext.com/spm/integrations/kafka-monitoring/ for an example. Otis -- Monitoring - Log Management - Alerting - Anomaly Detection Solr

Partitioning at the edges

2016-09-02 Thread Andy Chambers
Hey Folks, We are having quite a bit trouble modelling the flow of data through a very kafka centric system As I understand it, every stream you might want to join with another must be partitioned the same way. But often streams at the edges of a system *cannot* be partitioned the same way becaus

Re: Producer request latency increase after client 0.10 upgrade

2016-09-02 Thread Yifan Ying
The load before and after the upgrade are pretty similar. And all these configs stayed same too. Also please correct me if I am wrong. I think the request latency = response received time - request created time. And request created time is created when the records are ready to be sent after 'linge

Re: kafka streams: join 2 Windowed tables

2016-09-02 Thread Matthias J. Sax
Hi Ara, The problem is, that type Window is a composite key of "String-Long" that encoded your original key of type String the window start time of type Long. Thus, even if two record have the same "string-key" they might be different, because the belong to different windows. Try to insert a map(

Re: Producer: metadata refresh when leader down

2016-09-02 Thread Javier Holguera
Hi Yuto, Thanks for the link. We aren’t setting `retry.backoff.ms` so it would be running with its default of 1 sec. Correct me if I’m wrong, but in that case we may experience a 1 sec delay pulling the metadata. But we are observing long waits, sometimes for minutes. That’s why I thought the `m

Re: Producer: metadata refresh when leader down

2016-09-02 Thread Yuto KAWAMURA
HI Javier, Not sure but just wondering if this could be related to your case: https://issues.apache.org/jira/browse/KAFKA-4024?focusedCommentId=15458639&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15458639 Are you sure that metadata expiration is exactly the trigg

Re: Broker disappear from zookeeper

2016-09-02 Thread Alexey Ozeritsky
Hello, I've seen this. I've been using kafka since 0.8.0-snapshot and all versions have this problem. I think it happens in case of network problems. I just make a monitoring on this case and I restart broker if it disapper from zookeeper. 02.09.2016, 12:21, "李斯宁" : > No one have seen this pro

Re: Broker disappear from zookeeper

2016-09-02 Thread 李斯宁
No one have seen this problem before? On Fri, Aug 26, 2016 at 1:11 PM, 李斯宁 wrote: > hi, guys > I have a three nodes kafka cluster. > Some times, one of the Kafka brokers may disappear from zookeeper > "/brokers/ids", but the process is still live and prints logs seems normal. > > Have anyone of