Use case is for extended time processing of the messages which guarantee that a message is not lost from the queue if a client taking it fails to complete its processing and is not able to return it to the queue (ex crash, process termination, bug). It is how AWS defines its queue API https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html but it might be simpler for them because I think their queue is not a blocking queue and relies on polling
- There are multiple consumers of a queue - Message posted to the queue - Consumer A take message off the queue with Visibility Timeout 60 sec (message is not removed from queue but instead it becomes invisible for 60 sec so that subsequent messages can be taken) - Other consumers do not see the message and can take subsequent messages - Consumer A deletes the message before 60 sec are over and the workflow is done - Consumer A failed to delete the message within 60 seconds and it reappears on the queue and will be taken by the next waiting consumer - Consumer A may need to monitor processing and extend visibility timeout for the message in processing Without something like that (a two step queue processing - logical removal from queue (visibility) followed by physical removal upon completion of processing) it is necessary to have some sort of recovery on node startup and/or timer should a process take an entry from a queue and fail to complete processing or return it to the queue BTW how ignite transactions work if multiple queues are modified within transaction - start transaction - take from A - copy to B - remove or create an entry to C - commit if c mutation fails and I rollback transaction everything will be rolled back? Is there a more detailed description of transactional behavior particularly with a queue where order of entries is also involved? I only see https://ignite.apache.org/docs/latest/key-value-api/transactions On Wed, Mar 5, 2025 at 10:50 AM Jeremy McMillan <j...@gridgain.com> wrote: > I don't think it's clear what you're trying to accomplish. Can you > describe the desired behavior as a narrative? > > > On Tue, Mar 4, 2025 at 5:42 PM Alex Roytman <roytm...@gmail.com> wrote: > >> Hello I was wondering if there is anything in Ignite queue to provide >> behaviour similar to Queue Visibility Timeout or Processing Completion to >> support long running processing of queue items >> >> When item is taken of a queue it becomes "invisible" to subsequent queue >> reads until its configurable visibility times out. By that time is >> should be processed and removed by the consumer or its timeout extended or >> if consumer failed it should reappear on the queue >> >> Perhaps it can be achieved with transactions but I never fully understood >> how transactions would work for queue consumed concurrently. Perhaps >> transactions with main queue and journaling queue/list where picked items >> copied transactionally upon taking them off the main queue (take and push >> happen in the same transaction) and then pushed to end of main queue upon >> journaling queue expiration >> >> I would appreciate your take on it and would like to encourage >> implementing it when you get to work on distributed data structures in >> Ignite 3 (and when it may be? :-) >> >
