Re: kafka 0.8 producer throughput

2013-01-25 Thread Jay Kreps
Some folks came up with a cool hack in 0.8 that makes acks=0 send no response. This changes the performance for small message sends to be equivalent to 0.7. This is proposed for inclusion in 0.8. It would obviously be less useful for the java/scala client in 0.9 if we are able to properly pipeline

Re: kafka 0.8 producer throughput

2013-01-09 Thread Jay Kreps
We haven't done a ton of performance work on 0.8 yet. Regardless, requiring the ack will certainly reduce per-producer throughput, but it is too early to say by how much. Obviously this won't impact broker throughput (so if you have many producers you may not notice). The plan to fix this is just

Re: kafka 0.8 producer throughput

2013-01-09 Thread Jun Rao
In 0.8, ack is always required. The ack returns an errorcode that indicates the reason if a produce request fails (e.g., the request is sent to a broker that's not a leader). It also returns the offset of the produced messages. However, the producer can choose when to receive the acks (e.g., when

Re: kafka 0.8 producer throughput

2013-01-09 Thread S Ahmed
What's the ack for? If it fails, it will try another broker? Can this be disabled or it's a major design change? On Wed, Jan 9, 2013 at 12:40 AM, Jun Rao wrote: > The 50MB/s number is for 0.7. We haven't carefully measured the performance > in 0.8 yet. We do expect the throughput that a singl

Re: kafka 0.8 producer throughput

2013-01-08 Thread Jun Rao
The 50MB/s number is for 0.7. We haven't carefully measured the performance in 0.8 yet. We do expect the throughput that a single producer can drive in 0.8 to be less. This is because the 0.8 producer needs to wait for an RPC response from the broker while in 0.7, there is no ack for the producer.