Re: [DISCUSS] PIP-345: Optimize finding message by timestamp

2024-03-16 Thread 太上玄元道君
Hi Yubiao, Thanks for your review! 1. As I mentioned in the *Alternatives* part of the PIP `LedgerInfo#timestamp` is broker's timestamp, finding message by timestamp is client's timestamp(publishTimestamp). We have to consider about client's clock doesn't sync to broker's clock, and we are find

Re: [DISCUSS] PIP-345: Optimize finding message by timestamp

2024-03-16 Thread Yubiao Feng
Hi Jiuming Firstly, I think the idea you provided is great. It seems the field `beginPublishTimestamp` is also not needed, and there is an existing field `ledgerInfo.timestamp` to use, - The current ledger's timestamp can be used as `beginPublishTimestamp` - The next ledger's timestamp can be use

Re: [DISCUSS] PIP-345: Optimize finding message by timestamp

2024-03-14 Thread 太上玄元道君
Hi, Girish, Thanks for your feedback! In general, it's a very good suggestion, we can just use one single `beginPublishTimestamp` to achieve our goal, but the actual problem will be a bit more complex. Actually, the naming of `beginPublishTimestamp` and `endPublishTimestamp` has a little problem

Re: [DISCUSS] PIP-345: Optimize finding message by timestamp

2024-03-14 Thread Girish Sharma
One suggestion, I think you can make do with storing just begin timestamp. Any search utilising these values will work the same way with just one of those timestamps compared to both begin and end. Any particular reason you need both the timestamps? Regards On Fri, Mar 15, 2024, 9:39 AM 太上玄元道君

Re: [DISCUSS] PIP-345: Optimize finding message by timestamp

2024-03-14 Thread 太上玄元道君
bump 太上玄元道君 于2024年3月10日 周日06:41写道: > Hi Pulsar community, > > A new PIP is opened, this thread is to discuss PIP-345: Optimize finding > message by timestamp. > > Motivation: > Finding message by timestamp is widely used in Pulsar: > * It is used by the `pulsar-admin` tool to get the message id b

Re: [DISCUSS] PIP-345: Optimize finding message by timestamp

2024-03-11 Thread 太上玄元道君
Hi Dave, Thanks for your review! Perhaps it's because I wrote more detailed steps, but the key points is: 1. Deserialize MessageMetadata once broker received message 2. Pass MessageMetadata to `PublishContext` 3. After add entries finished, get `publishTimestamp` from `PublishContext#messageMetada

Re: [DISCUSS] PIP-345: Optimize finding message by timestamp

2024-03-11 Thread Dave Fisher
What can you say about the time spent to maintain these new ledger fields? I think you are asking to modify the main message logic which is highly optimized., but I’m not sure. Have you tried your code on your own hardware? Do you have performance comparisons of the normal flow? > On Mar 11, 20

Re: [DISCUSS] PIP-345: Optimize finding message by timestamp

2024-03-11 Thread 太上玄元道君
bump 太上玄元道君 于2024年3月11日 周一17:55写道: > bump > > 太上玄元道君 于2024年3月10日周日 06:41写道: > >> Hi Pulsar community, >> >> A new PIP is opened, this thread is to discuss PIP-345: Optimize finding >> message by timestamp. >> >> Motivation: >> Finding message by timestamp is widely used in Pulsar: >> * It is use

Re: [DISCUSS] PIP-345: Optimize finding message by timestamp

2024-03-11 Thread 太上玄元道君
bump 太上玄元道君 于2024年3月10日周日 06:41写道: > Hi Pulsar community, > > A new PIP is opened, this thread is to discuss PIP-345: Optimize finding > message by timestamp. > > Motivation: > Finding message by timestamp is widely used in Pulsar: > * It is used by the `pulsar-admin` tool to get the message id