Re: [DISSCUSS] Don't retain null-key messages during topic compaction

2023-11-08 Thread Cong Zhao
Hi Enrico, I don't think they conflict. We can do them separately. We can add the policy to the per topic and per namespace again when we need it in the future, and the current PIP can just add a configuration to broker.conf first so that we can not retain the null-key messages and cherry-pick

Re: [DISSCUSS] Don't retain null-key messages during topic compaction

2023-11-07 Thread Enrico Olivelli
This is a good point. Is it worth to make it configurable per topic and per namespace? Even if this behavior is kind if a side effect or bug, maybe some applications rely on it. Flags that change the behavior per cluster are hard to adopt in nig cluster with many tenants. We should always take m

Re: [DISSCUSS] Don't retain null-key messages during topic compaction

2023-11-07 Thread Cong Zhao
Hello everyone, I open a PIP-318: https://github.com/apache/pulsar/pull/21541 for this discussion. Any feedback and suggestions are welcome. Thanks, Cong Zhao On 2023/11/07 02:55:52 Cong Zhao wrote: > Hi, Pulsar community > > Currently, we retain all null-key messages during topic compaction,

Re: [DISSCUSS] Don't retain null-key messages during topic compaction

2023-11-06 Thread Cong Zhao
Hi mattison, Thanks for your suggestion, I agree with you, we can add a configuration to smooth the migrate this change. Let's see if anyone else has any other ideas, and if everyone agrees with this approach, I'll implement it. Thanks, Cong Zhao On 2023/11/07 03:03:58 mattison chao wrote: >

Re: [DISSCUSS] Don't retain null-key messages during topic compaction

2023-11-06 Thread mattison chao
Hi, Cong IMO, Please do not break the previous directly. We can migrate it smoothly. We can add a configuration and give the Timeline of making this configuration default and removing it in the next release version. For example: - Add configuration: `compactionRemainNullKey=true` by default (