They are not unrelated, because msg size automatically server double duty as a resource unit now, for all buffer allocations, and decoupling the two is good. It's perfectly reasonable to have one resource unit and be able to support small and large messages, instead of having to run separate instances tuned for each size
On Tue, May 14, 2019 at 3:51 PM Matteo Merli <matteo.me...@gmail.com> wrote: > On Tue, May 14, 2019 at 3:14 PM Sijie Guo <guosi...@gmail.com> wrote: > > > Rajan, > > > > Since this is a different topic than configuring max message size, does > it > > make sense to move it to a separated topic? > > > > +1 > > having the max message size configurable is a good thing anyways. Admins > can decide what they want to support. > > How to handle “huge” messages is a different discussion. > > > > > > Also I have a few questions regarding chunking. I will leave it in your > > doc. > > > > - Sijie > > > > On Fri, May 10, 2019 at 1:08 PM Rajan Dhabalia <rdhaba...@apache.org> > > wrote: > > > > > Hi, > > > > > > >> It can be easier to achieve when Transaction is supported. A large > > > message > > > can be split into a txn of smaller messages. > > > > > > I would like to propose another approach "Chunking and combine" to > > publish > > > large size of messages. Transaction deployment might not be convenient > > all > > > the time specially for the large traffic pulsar system due to multiple > > > reasons which I have shared in the doc. > > > > > > > > > https://docs.google.com/document/d/13F2QLcOBVUJhVKQJVsV7yajYpfdp5pDL0qmMbTb5A3Q/edit?usp=sharing > > > So, we would like to propose above approach to address publishing of > > large > > > message size. > > > > > > Thanks, > > > Rajan > > > > > > On Thu, May 9, 2019 at 7:38 PM Yong Zhang <zhangyong1025...@gmail.com> > > > wrote: > > > > > > > > > > > > > If we just let the broker/proxy decide the max size, then we can > > avoid > > > > > the configuration > > > > > in the client. That removes a lot of complexity in how to handle > > > > > situation when client max-size > > > > > is smaller than broker max-size and broker needs to deliver a > message > > > > > to that consumer. > > > > > > > > > > > > > Thank you. Actually implement as this way. I will fix the description > > in > > > > pip. > > > > On Fri, 10 May 2019 at 01:43, Matteo Merli <matteo.me...@gmail.com> > > > wrote: > > > > > > > > > Thanks Yong for starting the work on this. > > > > > > > > > > I think there should be more conditions specified in the PIP and we > > > > should > > > > > have > > > > > a clear documentation on what the behavior will be and what kind of > > > > errors > > > > > an > > > > > application will see. Especially we should cover how this will > > > > > interact with proxy. > > > > > > > > > > > On client side, client should set max message size in > configuration > > > and > > > > > it should be smaller than server support. > > > > > > > > > > If we just let the broker/proxy decide the max size, then we can > > avoid > > > > > the configuration > > > > > in the client. That removes a lot of complexity in how to handle > > > > > situation when client max-size > > > > > is smaller than broker max-size and broker needs to deliver a > message > > > > > to that consumer. > > > > > > > > > > Also, it becomes much easier to make the change from ops > perspective, > > > > > 1 single system instead of having to > > > > > configure all the client applications. > > > > > > > > > > -- > > > > > Matteo Merli > > > > > <matteo.me...@gmail.com> > > > > > > > > > > > > > > > On Wed, May 8, 2019 at 8:34 PM Sijie Guo <guosi...@gmail.com> > wrote: > > > > > > > > > > > > Thanks Yong for the PIP. > > > > > > > > > > > > I moved your gist to Pulsar wiki page: > > > > > > https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size > > > > > > > > > > > > The proposal looks good to me. +1 > > > > > > > > > > > > - Sijie > > > > > > > > > > > > On Mon, May 6, 2019 at 6:04 PM Yong Zhang < > > > zhangyong1025...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > hi all, > > > > > > > > > > > > > > > > > > > > > Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t > be > > > > > modified > > > > > > > in server configuration. So there is no way when user want to > > > modify > > > > > the > > > > > > > limit to transfer larger size message. > > > > > > > > > > > > > > Hence i propose adding a `MaxMessageSize` config in > `broker.conf` > > > to > > > > > solve > > > > > > > this problem. Because broker server will decide how much > message > > > size > > > > > will > > > > > > > be received so client need know how much message client can be > > > sent. > > > > > > > > > > > > > > for details track following pip: > > > > > > > https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a > > > > > > > > > > > > > > > > > > > > > > -- > -- > Matteo Merli > <matteo.me...@gmail.com> >
