-- Matteo Merli <mme...@apache.org> On Tue, Jul 12, 2022 at 12:54 AM Enrico Olivelli <eolive...@gmail.com> wrote: > > Hello, > I think that we could implement a small but effective enhancement to batching. > > It may happen that even if you enable batching you come to create > entries with 1 only message. > > Processing batch messages requires a good amount of resources, both on > the broker and on the client side. > > Especially when you are using PIP-105, Broker side filtering, you have > to unpack (and decompress) the whole entry in order to process the > very single message. > > So my proposal is to change the (Java) producer, to make it produce a > regular message instead of a batch message if the batch contains only > 1 message
I'm ok with the change, though I don't see a big difference in the cost for clients and brokers. Brokers are never going to look into batches, they will just check at the batch header, without decompression, and not dig into the individual messages properties. Also I don't see the significant additional CPU usage in building a batch of 1 compared to a non-batched message. Keep in mind that the reason we have the 2 formats is purely due to the fact that in the very beginning we didn't have support for batches. > Especially when you are using PIP-105, Broker side filtering, you have > to unpack (and decompress) the whole entry in order to process the > very single message. That is more a testament to the fact that the PIP-105 is not a good model. Whether the message is part of a batch or not, it changes very little.
