Performance of producer sending to many vs to few topics

2016-07-19 Thread Krzysztof Nawara
Hi, I have been testing Kafka in order to determine how does number of partitions affects performance. In order to do that, I have set up 2 Kafka nodes and 1 Zookeeper nodes, and used 8 producers running on different machines to send messages. First, my producers were sending messages to broker

Re: Performance of producer sending to many vs to few topics

2016-07-20 Thread Krzysztof Nawara
as above. You may want to try > again by increasing these metadata exchange timeouts and memory as you > increase topics/partitions. > > > > > > > > On Tue, Jul 19, 2016 at 2:33 PM, Krzysztof Nawara > wrote: > > > Hi, > > > > I have be

Problems with replication and performance

2016-07-27 Thread Krzysztof Nawara
1 topic it oscilated around 100 records/request (5KB records), in 4k topics scenario it was just 1 record/request. Can you think of any explanation for that? Code I use for testing: https://github.com/BlueEyedHush/kafka_perf/tree/itrac Thank you, Krzysztof Nawara

RE: Problems with replication and performance

2016-07-27 Thread Krzysztof Nawara
the same draw backs described in this article regardless of what you do. -David On 7/27/16, 4:51 PM, "Krzysztof Nawara" wrote: Hi! I've been testing Kafka. I've hit some problems, but I can't really understand what's going on, so I'd like to ask f

Connection to broker which lost Zk connection

2016-07-30 Thread Krzysztof Nawara
Hello! I've got the following question: Let's assume broker A is leader for partition P1, to which my producer is sending data. At one point A loses its connection to Zk, but my producer can communicate with A without problems. Will my messages be lost? What'll trigger switching to the new lead

RAID 10

2016-07-30 Thread Krzysztof Nawara
Hello, Could you share your experience with using RAID 10 for brokers? I'm especially interested in whether you can keep broker fully operational during RAID rebuild or it falls behind so much that it must go through lengthy resynchronization process anyway? Thanks, Krzysiek

RE: Too Many Open Files

2016-07-31 Thread Krzysztof Nawara
Maybe you are exhausting your sockets, not file handles for some reason? From: Kessiler Rodrigues [kessi...@callinize.com] Sent: 31 July 2016 22:14 To: users@kafka.apache.org Subject: Re: Too Many Open Files I’m still experiencing this issue… Here are th

Connections to Zookeeper

2016-08-05 Thread Krzysztof Nawara
Hello, How many connections to Zookeeper should correctly working Kafka broker have opened at the same time? Krzysiek