Once we have automatic break down and reassembly of bigger messages, then the client will be able to use the "discovered" max message size in the same way a TCP client uses the MSS.
-- Matteo Merli <matteo.me...@gmail.com> On Tue, May 14, 2019 at 5:08 PM Joe Francis <j...@verizonmedia.com.invalid> wrote: > > They are not unrelated, because msg size automatically server double duty > as a resource unit now, for all buffer allocations, and decoupling the two > is good. > It's perfectly reasonable to have one resource unit and be able to support > small and large messages, instead of having to run separate instances tuned > for each size > > On Tue, May 14, 2019 at 3:51 PM Matteo Merli <matteo.me...@gmail.com> wrote: > > > On Tue, May 14, 2019 at 3:14 PM Sijie Guo <guosi...@gmail.com> wrote: > > > > > Rajan, > > > > > > Since this is a different topic than configuring max message size, does > > it > > > make sense to move it to a separated topic? > > > > > > > > +1 > > > > having the max message size configurable is a good thing anyways. Admins > > can decide what they want to support. > > > > How to handle “huge” messages is a different discussion. > > > > > > > > > > Also I have a few questions regarding chunking. I will leave it in your > > > doc. > > > > > > - Sijie > > > > > > On Fri, May 10, 2019 at 1:08 PM Rajan Dhabalia <rdhaba...@apache.org> > > > wrote: > > > > > > > Hi, > > > > > > > > >> It can be easier to achieve when Transaction is supported. A large > > > > message > > > > can be split into a txn of smaller messages. > > > > > > > > I would like to propose another approach "Chunking and combine" to > > > publish > > > > large size of messages. Transaction deployment might not be convenient > > > all > > > > the time specially for the large traffic pulsar system due to multiple > > > > reasons which I have shared in the doc. > > > > > > > > > > > > > https://docs.google.com/document/d/13F2QLcOBVUJhVKQJVsV7yajYpfdp5pDL0qmMbTb5A3Q/edit?usp=sharing > > > > So, we would like to propose above approach to address publishing of > > > large > > > > message size. > > > > > > > > Thanks, > > > > Rajan > > > > > > > > On Thu, May 9, 2019 at 7:38 PM Yong Zhang <zhangyong1025...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > If we just let the broker/proxy decide the max size, then we can > > > avoid > > > > > > the configuration > > > > > > in the client. That removes a lot of complexity in how to handle > > > > > > situation when client max-size > > > > > > is smaller than broker max-size and broker needs to deliver a > > message > > > > > > to that consumer. > > > > > > > > > > > > > > > > Thank you. Actually implement as this way. I will fix the description > > > in > > > > > pip. > > > > > On Fri, 10 May 2019 at 01:43, Matteo Merli <matteo.me...@gmail.com> > > > > wrote: > > > > > > > > > > > Thanks Yong for starting the work on this. > > > > > > > > > > > > I think there should be more conditions specified in the PIP and we > > > > > should > > > > > > have > > > > > > a clear documentation on what the behavior will be and what kind of > > > > > errors > > > > > > an > > > > > > application will see. Especially we should cover how this will > > > > > > interact with proxy. > > > > > > > > > > > > > On client side, client should set max message size in > > configuration > > > > and > > > > > > it should be smaller than server support. > > > > > > > > > > > > If we just let the broker/proxy decide the max size, then we can > > > avoid > > > > > > the configuration > > > > > > in the client. That removes a lot of complexity in how to handle > > > > > > situation when client max-size > > > > > > is smaller than broker max-size and broker needs to deliver a > > message > > > > > > to that consumer. > > > > > > > > > > > > Also, it becomes much easier to make the change from ops > > perspective, > > > > > > 1 single system instead of having to > > > > > > configure all the client applications. > > > > > > > > > > > > -- > > > > > > Matteo Merli > > > > > > <matteo.me...@gmail.com> > > > > > > > > > > > > > > > > > > On Wed, May 8, 2019 at 8:34 PM Sijie Guo <guosi...@gmail.com> > > wrote: > > > > > > > > > > > > > > Thanks Yong for the PIP. > > > > > > > > > > > > > > I moved your gist to Pulsar wiki page: > > > > > > > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size > > > > > > > > > > > > > > The proposal looks good to me. +1 > > > > > > > > > > > > > > - Sijie > > > > > > > > > > > > > > On Mon, May 6, 2019 at 6:04 PM Yong Zhang < > > > > zhangyong1025...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > hi all, > > > > > > > > > > > > > > > > > > > > > > > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t > > be > > > > > > modified > > > > > > > > in server configuration. So there is no way when user want to > > > > modify > > > > > > the > > > > > > > > limit to transfer larger size message. > > > > > > > > > > > > > > > > Hence i propose adding a `MaxMessageSize` config in > > `broker.conf` > > > > to > > > > > > solve > > > > > > > > this problem. Because broker server will decide how much > > message > > > > size > > > > > > will > > > > > > > > be received so client need know how much message client can be > > > > sent. > > > > > > > > > > > > > > > > for details track following pip: > > > > > > > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > -- > > Matteo Merli > > <matteo.me...@gmail.com> > >
