Hello everyone, I'd like to initiate a discussion regarding the compaction feature in Pulsar topics. Currently, there are two types of compactors available: `TwoPhaseCompactor` and `StrategicTwoPhaseCompactor`. The latter is specifically utilized for internal load balancing purposes and is not employed for regular compaction of Pulsar topics. On the other hand, the former can be configured via `CompactionServiceFactory` in the `broker.conf`.
I believe it could be advantageous to introduce another type of topic compactor that operates based on event time. Such a compactor would have the capability to maintain desired messages within the topic while preserving the order expected by external applications. Although applications may send messages with the current event time, variations in network conditions or redeliveries could result in messages being stored in the Pulsar topic in a different order than intended. Implementing event time-based checks could mitigate this inconvenience. To address this, I've drafted a pull request with a sample implementation, which can be found at: https://github.com/apache/pulsar/pull/22517 I would greatly appreciate any feedback or thoughts you may have on this proposal. -- *Marek Czajkowski* Backend developer Website: www.ds.pl Email: marek.czajkow...@ds.pl Mob: +48 698167319 Dynamic Solutions sp. z o.o. sp.k. ul. Poleska 45/1, 15-114 Białystok Your personal data controller is Dynamic Solutions Sp. z o.o. sp. k. For GDPR compliance details, visit our website <https://www.ds.pl/information-clause-email-correspondence.html>.
