Re: PIP 36: Max message size

>> Also I have a few questions regarding chunking. I will leave it in your doc. can you please use below link as previous doc-link was not allowing to comment. https://docs.google.com/document/d/13F2QLcOBVUJhVKQJVsV7yajYpfdp5pDL0qmMbTb5A3Q/edit?usp=sharing >> Since this is a different topic than

Re: PIP 36: Max message size

Once we have automatic break down and reassembly of bigger messages, then the client will be able to use the "discovered" max message size in the same way a TCP client uses the MSS. -- Matteo Merli On Tue, May 14, 2019 at 5:08 PM Joe Francis wrote: > > They are not unrelated, because msg size a

Re: PIP 36: Max message size

They are not unrelated, because msg size automatically server double duty as a resource unit now, for all buffer allocations, and decoupling the two is good. It's perfectly reasonable to have one resource unit and be able to support small and large messages, instead of having to run separate instan

Re: PIP 36: Max message size

On Tue, May 14, 2019 at 3:14 PM Sijie Guo wrote: > Rajan, > > Since this is a different topic than configuring max message size, does it > make sense to move it to a separated topic? +1 having the max message size configurable is a good thing anyways. Admins can decide what they want to suppo

Re: PIP 36: Max message size

Rajan, Since this is a different topic than configuring max message size, does it make sense to move it to a separated topic? Also I have a few questions regarding chunking. I will leave it in your doc. - Sijie On Fri, May 10, 2019 at 1:08 PM Rajan Dhabalia wrote: > Hi, > > >> It can be easi

Re: PIP 36: Max message size

Hi, >> It can be easier to achieve when Transaction is supported. A large message can be split into a txn of smaller messages. I would like to propose another approach "Chunking and combine" to publish large size of messages. Transaction deployment might not be convenient all the time specially f

Re: PIP 36: Max message size

> > If we just let the broker/proxy decide the max size, then we can avoid > the configuration > in the client. That removes a lot of complexity in how to handle > situation when client max-size > is smaller than broker max-size and broker needs to deliver a message > to that consumer. > Thank you

Re: PIP 36: Max message size

On Fri, May 10, 2019 at 3:52 AM Joe F wrote: > I think 5MB is too large and it should be reduced. :-) . I am with Kafka > on this one.. 1MB is good enough. It seems common to see a message larger than 5MB in MySQL cdc :( > > Have we completely ruled out a split/join abstraction on the Pulsa

Re: PIP 36: Max message size

> I think there should be more conditions specified in the PIP and we should > have a clear documentation on what the behavior will be and what kind of errors an application will see. Especially we should cover how this will interact with proxy. Just to clarify my thoughts here, I was thinking of

Re: PIP 36: Max message size

> I think 5MB is too large and it should be reduced. :-) . I am with Kafka on this one.. 1MB is good enough. It would be easy to configure in such way if broker advertises 1MB and client complies with it. -- Matteo Merli On Thu, May 9, 2019 at 12:52 PM Joe F wrote: > > I think 5MB is too lar

Re: PIP 36: Max message size

I think 5MB is too large and it should be reduced. :-) . I am with Kafka on this one.. 1MB is good enough. Have we completely ruled out a split/join abstraction on the Pulsar produce/consumer APIs? No backward compatibility, no client-server capability mismatches, now that we have dedup, split

Re: PIP 36: Max message size

Thanks Yong for starting the work on this. I think there should be more conditions specified in the PIP and we should have a clear documentation on what the behavior will be and what kind of errors an application will see. Especially we should cover how this will interact with proxy. > On client

Re: PIP 36: Max message size

Thanks Yong for the PIP. I moved your gist to Pulsar wiki page: https://github.com/apache/pulsar/wiki/PIP-36%3A-Max-Message-Size The proposal looks good to me. +1 - Sijie On Mon, May 6, 2019 at 6:04 PM Yong Zhang wrote: > hi all, > > > Currently `MaxMessageSize` is hardcoded in Pulsar and it

PIP 36: Max message size

hi all, Currently `MaxMessageSize` is hardcoded in Pulsar and it can’t be modified in server configuration. So there is no way when user want to modify the limit to transfer larger size message. Hence i propose adding a `MaxMessageSize` config in `broker.conf` to solve this problem. Because brok