As a data point on one system, while Snappy compression is significantly
better than gzip, for our system isn't wasn't enough to offset the
decompress/compress on the broker. No matter how fast the compression is,
doing that on the broker will always be slower than not.
We went the route the origi
servers and with my payload.
>
> My end conclusion was that its impossible to scale a single producer
> instance, and more threads make no difference on the sending side.
> On 1 Jan 2014 17:31, "Chris Hogue" wrote:
>
> > Have you found what the actual bottleneck is? Is it
it Jansen van Vuuren <
gerrit...@gmail.com> wrote:
> I've seen this bottle neck regardless of using compression or not, bpth
> situations give me poor performance on sending to kafka via the scala
> producer api.
> On 1 Jan 2014 16:42, "Chris Hogue" wrote:
>
> >
Hi.
When writing that blog we were using Kafka 0.7 as well. Understanding that
it probably wasn't the primary design goal, the separate send threads per
broker that offered a separation of compression were a convenient
side-effect of that design.
We've since built new systems on 0.8 that have con
I would generally agree with the key goals you've suggested.
I'm just coming to this discussion after some recent testing with 0.8 so I
may be missing some background. The reference I found to this discussion is
the JIRA issue below. Please let me know if there are others things I
should look at.
Another note I'd suggest for the zookeeper name-spacing doc is that the
path must exist in zookeeper before starting the brokers. There is an error
message, but it's not always clear what the problem is (I think there is
even a Jira issue to make the error more clear).
-Chris
On Sat, Aug 10, 201
hat is the
> root cause rather than just our inefficiency before we do anything too
> drastic design wise. If this is really killing you guys, and if that turns
> out to be the cause, we would definitely take a patch to optimize that path
> now.
>
> -Jay
>
>
>
>
> On Fr
Thanks for the responses. Additional follow-up inline.
On Fri, Aug 2, 2013 at 2:21 PM, Jay Kreps wrote:
> Great comments, answers inline!
>
> On Fri, Aug 2, 2013 at 12:28 PM, Chris Hogue wrote:
>
> > These sounds like great steps. A couple of votes and questions:
These sounds like great steps. A couple of votes and questions:
1. Moving serialization out and basing it all off of byte[] for key and
payload makes sense. Echoing a response below, we've ended up doing that in
some cases anyway, and the others do a trivial transform to bytes with an
Encoder.
2
are of too many use cases of custom event handler.
> So, we only made the serializer and partitioner customizable. We can
> revisit this if there are good use cases of custom event handler. For your
> use case, could you just do the batching/customized compression outside of
> produ
A couple of notes on name-spacing since there are a couple of gotchas when
trying to configure it:
It's specified via the zookeeper.connect property described here:
http://kafka.apache.org/08/configuration.html
If you have multiple servers in the zk cluster rather than a standalone
node the path
We're a heavy 0.7 user and are now digging into 0.8 for some new projects.
One of the features we used in 0.7 appears to be different and not clearly
supported in 0.8.
We use the EventHandler plug-point in 0.7, specifically to do custom
batching before the messages are actually sent to the broker.
12 matches
Mail list logo
