This is weird. The writing to different partitions should be independent.
Could you enable request log at the broker and see if the requests are what
you expect?
Thanks,
Jun
On Thu, Jan 2, 2014 at 4:12 AM, yosi botzer wrote:
> Thanks Jun,
>
> I thought about your answer and came up with the f
Thanks Jun,
I thought about your answer and came up with the following solution:
Create a pool of async producers (the size of the pool is the same as the
number of partitions) and define my own partitioner.
Whenever I get a message I decide which producer to use with the same exact
logic as my
When the producer send thread sends a batch of messages, it first
determines which partition each message should go to. It then groups
messages by broker (based on the leader of the partition of each
message) and sends a produce request per broker (each request may include
multiple partitions). Tho
Yes I am specifying a key for each message.
The same producer code works much slower when sending messages to a topic
with multiple partitions comparing to a topic with a single partition. This
doesn't make any sense to me at all.
If I understand correctly I need multiple partitions in order to s
In 0.7, we have 1 producer send thread per broker. This is changed in 0.8,
where there is only 1 producer send thread per producer. If a producer
needs to send messages to multiple brokers, the send thread will do that
serially, which will reduce the throughput. We plan to improve that in 0.9
throu
I see. If I'm following correctly you're talking about trying more
application threads passing messages to the producer, which then serializes
them down to a single threaded send.
That sounds right, multiple producer instances in the application are the
only real way to use the stock producer to g
The network is 10gbit so it seems unlikely. The 5 brokers were running
without much load or probs. The bottle neck is that no matter how many
threads I use for sending, the sync block in the send method will never go
faster and its always limited to a single thread.
I also use snappy outside and n
Have you found what the actual bottleneck is? Is it the network send? Of
course this would be highly influenced by the brokers' performance. After
removing all compression work from the brokers we were able to get enough
throughput from them that it's not really a concern.
Another rough side-effec
I've seen this bottle neck regardless of using compression or not, bpth
situations give me poor performance on sending to kafka via the scala
producer api.
On 1 Jan 2014 16:42, "Chris Hogue" wrote:
> Hi.
>
> When writing that blog we were using Kafka 0.7 as well. Understanding that
> it probably
Hi.
When writing that blog we were using Kafka 0.7 as well. Understanding that
it probably wasn't the primary design goal, the separate send threads per
broker that offered a separation of compression were a convenient
side-effect of that design.
We've since built new systems on 0.8 that have con
This is very interesting, this is what I see as well. I wish someone could
explain why it is not as explained here:
http://engineering.gnip.com/kafka-async-producer/
On Wed, Jan 1, 2014 at 2:39 PM, Gerrit Jansen van Vuuren <
gerrit...@gmail.com> wrote:
> I don't know the code enough to comment o
I don't know the code enough to comment on that (maybe someone else on the
user list can do that), but from what I've seen doing some heavy profiling
I only see one thread per producer instance, it doesn't matter how many
brokers or topics you have the number of threads is always 1 per producer.
If
But shouldn't I see a separate thread per broker (I am using the async
mode)? Why do I get a better performance sending a message that has fewer
partitions?
On Wed, Jan 1, 2014 at 2:22 PM, Gerrit Jansen van Vuuren <
gerrit...@gmail.com> wrote:
> The producer is heavily synchronized (i.e. all th
The producer is heavily synchronized (i.e. all the code in the send method
is encapsulated in one huge synchronized block).
Try creating multiple producers and round robin send over them.
e.g.
p = producers[ n++ % producers.length ]
p.send msg
This will give you one thread per producer instance.
Hi,
I am using kafka 0.8. I have 3 machines each running kafka broker.
I am using async mode of my Producer. I expected to see 3 different threads
with names starting with ProducerSendThread- (according to this article:
http://engineering.gnip.com/kafka-async-producer/)
However I can see only on
15 matches
Mail list logo
