Hi Michael,
This would work but it will need something to store every messages read
to reverse them before answer which can be heavy in RAM usages. The key
point of the future is to read message by message from a MessageId to
past with stop read possible conditions.
Best,
Kannar
On 3/7/23 22:10, Michael Marshall wrote:
The goal is to start from a known MessageId and read the N message
before this MessageId.
Have you looked at the seek implementation to see if it would be
feasible to extend the implementation and add a method to "seekBefore"
a message id in the way you described? I haven't considered all of the
implications, but if the main goal is to move the cursor, I think the
solution should be about moving the cursor, not about reading a topic
in reverse.
Thanks,
Michael
On Tue, Mar 7, 2023 at 6:50 AM Alexandre DUVAL <kanna...@gmail.com> wrote:
Hi Yong,
The goal is to start from a known MessageId and read the N message
before this MessageId.
Best,
Kannar
On 3/7/23 01:53, Yong Zhang wrote:
Hi Kannar,
Just interested in what exactly your case.
Why do you need to read messages in a reversed order? What is your case?
Best,
Yong
On Mon, 6 Mar 2023 at 23:37, Alexandre DUVAL<kanna...@gmail.com> wrote:
Hi,
I'm wondering if it is possible to introduce a new feature on Pulsar
which will enable users to read topic from a defined MessageId to
previous messages until the begin of the topic.
I tried to use Pulsar SQL but it requires so much RAM even for little
queries (due to Presto design).
Currently, every read in Pulsar are expected to be going forward. So it
might be a bit tricky to prevent every weird behavior by introducing the
feature.
I'm currently tried to make an MVP/POC by introducting a readReverse
field in the CommandSubscribe that is used by ReaderAPI and currently
looking for to create a getFirstMessageId() on ManagedLedger
(https://github.com/CleverCloud/pulsar/pull/3). I also removed
startPosition < endPosition sanity checks in BookKeeper locally
(https://github.com/CleverCloud/bookkeeper/pull/2).
We definitely prefer a readPrevious(), hasPreviousMessageAvailable() in
the ReaderAPI.
I'm not familiar with these internals such as NonDurableCursor,
RangeEntryCache, ManagedCursor so it's a bit tricky.
So I wondering someone to help/guide me or even directly handle the
subject (or the discuss).
Regards,
Kannar
