+1,
We can only skip the topic level messages size check for the chunk message.
Regards,
Penghui
On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang <jianghait...@apache.org>
wrote:
> Hi Pulsar Community,
>
> I discovered a bug that chunk messages producing fails if topic level
> maxMessageSize is set [1]. The root cause of this issue is because chunk
> message is using broker level maxMessageSize as chunk size. And topic level
> maxMessageSize is always <= broker level maxMessageSize. So once it is set,
> the on-going chunk message producing fails.
>
> ## Proposed changes
> I would like to fix this by just skipping topic level maxMessageSize check
> in
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> level maxMessageSize is introduced in [2], for the purpose of "easier to
> plan resource quotas for client allocation". And IMO this change will not
> bring further complex into this.
>
> ## Alternative
> Add a client side topic level maxMessageSize and keep it synced with
> broker.
>
> Required changes:
> - [client] Add a new field
> org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> client side topic level maxMessageSize.
> - [PulsarApi.proto] Add a new field maxMessageSize in the
> CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> - [PulsarApi.proto] Add a new Command like
> CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> Further more, some other data consistency issues need be handled very
> carefully when maxMessageSize is updated.
> This alternative is complex but can also solve other topic level
> maxMessageSize issue [3] when batching is enabled (non-batching case is
> solved with PR [4]).
>
> Any suggestions or other use cases of topic level maxMessageSize will be
> appreciated.
>
> Thanks,
> Haiting Jiang
>
> [1] https://github.com/apache/pulsar/issues/13360
> [2] https://github.com/apache/pulsar/pull/8732
> [3] https://github.com/apache/pulsar/issues/12958
> [4] https://github.com/apache/pulsar/pull/13147
>
