Tao, I think that the proposal is valuable. We have to clarify which datatypes are supported both for the keys and for the values. Are keys and values only raw arrays of bytes?
It would be better to have a way to detect the types, so that automatic tools, like CLIs and UIs can represent the value in a human readable format or even perform some sort of filtering. Second question: are these properties writable only at ledger creation time? Can you update them ? Which are the rules? How does coordination happen ? Enrico Il Mar 11 Feb 2025, 07:38 Tao Jiuming <dao...@apache.org> ha scritto: > Hi all, > > I open PIP-404 to introduce per ledger properties, which will add a > `properties` field to `LedgerInfo` to store extra properties for every > `Ledger`. > > # Background knowledge > As we don't have a secondary index on the Bookkeeper, so we can't query > entries by message metadata efficiently. > The `ManagedCursor` provides a method `asyncFindNewestMatching` to find the > newest entry that matches the given > predicate by binary search(See `OpFindNewest.java`). > In https://github.com/apache/pulsar/pull/22792, we optimized `seeking by > timestamp` by calculating > the range of ledgers that may contain the target timestamp by > `LedgerInfo#timestamp` and we don't need to scan all > ledgers. > However, when we enabled `AppendIndexMetadataInterceptor` and we want to > query entries by `BrokerEntryMetadata#index`, > there is no more efficient way, > we have to scan all ledgers by binary search to find the target entry. > > # Motivation > Introduce per-ledger properties and we can store the extra per-ledger > properties in the `LedgerInfo`, > so we can query entries by `incremental index` more efficiently, say, > `BrokerEntryMetadata#index`. > > PR link: https://github.com/apache/pulsar/pull/23837/files > > Please help review the PIP and vote. > > Thanks, > Tao Jiuming(dao-jun) >
