We're a heavy 0.7 user and are now digging into 0.8 for some new projects. One of the features we used in 0.7 appears to be different and not clearly supported in 0.8.
We use the EventHandler plug-point in 0.7, specifically to do custom batching before the messages are actually sent to the broker. We were hoping to use this feature in 0.8, albeit for a slightly different reason. In testing 0.8 we found the broker had a pretty significant choke point with compression. I'm sure this is a known performance consideration, where using the built-in compression forces the broker to un-compress and re-compress the messages to assign offsets. We changed our test to batch and compress the messages before it went to the producer, leaving the compression.codec set to none. With this change we saw on the order of a 3x increase in throughput on the broker while still retaining compressed messages across the wire and on disk. This was good for the tests but in the app we are planning to use semantic partitioning, so pre-batching and compressing before going to the producer is trickier. The hope was to use the producer's batching and partition selection, just plugging into the event handler for the compression piece. While I see the DefaultEventHandler is there and potentially replaceable via the constructor, it appears to have taken on more functionality and would thus be more involved to replace. Combined with all produce requests across brokers now being on a single thread, this is appearing to be difficult to achieve. I can imagine why some of these changes are in place in light of the replication feature, but wanted to throw it out there (a) in case we missed something or (b) at least as a potential consideration in the client re-design project that is under way. I'd be happy to have any input or other ideas on how to accomplish this that we may have missed. We'll look at other ways to tackle this and follow the other threads about the new clients in the meantime. Thanks. -Chris
