+ 1 for skipping the topic level message size check for the chunk message
in the broker.
On Tue, Dec 28, 2021 at 9:14 AM guo jiwei <techno...@apache.org> wrote:
> +1
>
>
> Regards
> Jiwei Guo (Tboy)
>
> On Tue, Dec 28, 2021 at 5:44 AM 陳智弘 <thomasec...@gmail.com> wrote:
> >
> > +1
> >
> > Skip the topic level setting is better
> >
> > Hang Chen <chenh...@apache.org> 於 2021年12月27日 週一 22:09 寫道：
> >
> > > +1
> > >
> > > We'd better skip the topic level maxMessageSize check for chunk
> messages.
> > >
> > > Best,
> > > Hang
> > >
> > > PengHui Li <peng...@apache.org> 于2021年12月27日周一 22:07写道：
> > > >
> > > > +1,
> > > >
> > > > We can only skip the topic level messages size check for the chunk
> > > message.
> > > >
> > > > Regards,
> > > > Penghui
> > > >
> > > > On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang <
> jianghait...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Pulsar Community,
> > > > >
> > > > > I discovered a bug that chunk messages producing fails if topic
> level
> > > > > maxMessageSize is set [1]. The root cause of this issue is because
> > > chunk
> > > > > message is using broker level maxMessageSize as chunk size. And
> topic
> > > level
> > > > > maxMessageSize is always <= broker level maxMessageSize. So once
> it is
> > > set,
> > > > > the on-going chunk message producing fails.
> > > > >
> > > > > ## Proposed changes
> > > > > I would like to fix this by just skipping topic level
> maxMessageSize
> > > check
> > > > > in
> > > > >
> > >
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > > > > level maxMessageSize is introduced in [2], for the purpose of
> "easier
> > > to
> > > > > plan resource quotas for client allocation". And IMO this change
> will
> > > not
> > > > > bring further complex into this.
> > > > >
> > > > > ## Alternative
> > > > > Add a client side topic level maxMessageSize and keep it synced
> with
> > > > > broker.
> > > > >
> > > > > Required changes:
> > > > > - [client] Add a new field
> > > > > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store
> this
> > > > > client side topic level maxMessageSize.
> > > > > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > > > > CommandProducerSuccess for the initial value of
> > > ProducerBase#maxMessageSize
> > > > > - [PulsarApi.proto] Add a new Command like
> > > > > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > > > > ProducerBase#maxMessageSize when topic level maxMessageSize is
> updated.
> > > > > Further more, some other data consistency issues need be handled
> very
> > > > > carefully when maxMessageSize is updated.
> > > > > This alternative is complex but can also solve other topic level
> > > > > maxMessageSize issue [3] when batching is enabled (non-batching
> case is
> > > > > solved with PR [4]).
> > > > >
> > > > > Any suggestions or other use cases of topic level maxMessageSize
> will
> > > be
> > > > > appreciated.
> > > > >
> > > > > Thanks,
> > > > > Haiting Jiang
> > > > >
> > > > > [1] https://github.com/apache/pulsar/issues/13360
> > > > > [2] https://github.com/apache/pulsar/pull/8732
> > > > > [3] https://github.com/apache/pulsar/issues/12958
> > > > > [4] https://github.com/apache/pulsar/pull/13147
> > > > >
> > >
>
--
Zike Yang
