Re: Design of a reliable task processing queue

2025-01-21 Thread Dominique Devienne
On Sun, Jan 19, 2025 at 9:23 AM Alex Burkhart wrote: > I'm looking for help to organize locks and transaction for a reliable task > queue. Have a look at https://github.com/tembo-io/pgmq for inspiration maybe. --DD

Re: Design of a reliable task processing queue

2025-01-19 Thread auxsvr
On Saturday, 18 January 2025 12:44:07 EET Alex Burkhart wrote: > Hey team, > > I'm looking for help to organize locks and transaction for a reliable task > queue. > > REQUIREMENTS > > 1. Pending actions are persisted to a database. There's a trace once they > are done. > 2. Application workers p

Re: Design of a reliable task processing queue

2025-01-19 Thread Shaheed Haque
As with all things distributed, it's useful to start with some notion of what the word "reliable" means to you. For example, when your pseudo code, is replaced by the real code, can that fail? And if it fails, is it known that the work in hand can simply be redone? Those answers will need to be un

Re: Design of a reliable task processing queue

2025-01-19 Thread Peter J. Holzer
On 2025-01-18 19:44:07 +0900, Alex Burkhart wrote: > REQUIREMENTS > > 1. Pending actions are persisted to a database. There's a trace once they are > done. > 2. Application workers pick actions one by one. At any given time, each action > can be assigned to at most one worker (transaction). > 3. I

Design of a reliable task processing queue

2025-01-19 Thread Alex Burkhart
Hey team, I'm looking for help to organize locks and transaction for a reliable task queue. REQUIREMENTS 1. Pending actions are persisted to a database. There's a trace once they are done. 2. Application workers pick actions one by one. At any given time, each action can be assigned to at most o