Re: [GENERAL] Multithreaded queue in PgSQL

2008-07-15 Thread Jeff Peck
> We are doing the same (newsletter) and there is no problem to lock the > whole table for a short time with an advisory lock as the java id > fetching worker is locking the table (that does not lock the table for > reading or writing, it is only locking his java worker brothers that > are using th

Re: [GENERAL] Multithreaded queue in PgSQL

2008-07-15 Thread valgog
> Whole point is to have multiple services accessing same table and > dividing the work, so locking with waiting for lock to be released is > out of question. > We are doing the same (newsletter) and there is no problem to lock the whole table for a short time with an advisory lock as the java id

Re: [GENERAL] Multithreaded queue in PgSQL

2008-06-11 Thread Klint Gore
Stevo Slavić wrote: I'm trying to make implementation more generic, not to use Postgres specific SQL, and through Hibernate and Spring configuration make services acquire lock on batch of rows, when trying to acquire lock on batch of rows an exception should be thrown if rows are already locked

Re: [GENERAL] Multithreaded queue in PgSQL

2008-06-11 Thread Csaba Nagy
We also have such a queue here, and our solution is an algorithm like this: 1. get the next processor_count * 2 queue ids which are not marked as taken; 2. choose randomly one of these ids; 3. lock for update with nowait; 4. if locking succeeds: 4.1. check again the item, as it could have

Re: [GENERAL] Multithreaded queue in PgSQL

2008-06-11 Thread Stevo Slavić
Hello all, I've initially brought the question to Nix, so I'll try to clarify situation. Whole point is to have multiple services accessing same table and dividing the work, so locking with waiting for lock to be released is out of question. I want to deploy same Java Spring Web applicatio

Re: [GENERAL] Multithreaded queue in PgSQL

2008-06-11 Thread valgog
On Jun 10, 1:58 pm, [EMAIL PROTECTED] (Nikola Milutinovic) wrote: > > You may find that the PGQ component of skytools is what you want: > >  http://pgfoundry.org/projects/skytools > >  http://skytools.projects.postgresql.org/doc/ > >  http://skytools.projects.postgresql.org/doc/pgq-sql.html > > Tha

Re: [GENERAL] Multithreaded queue in PgSQL

2008-06-10 Thread Scott Marlowe
On Tue, Jun 10, 2008 at 4:40 AM, Nikola Milutinovic <[EMAIL PROTECTED]> wrote: > Hi all. > > This may be trivial, but I cannot find good references for it. The problem > is this: > > Suppose we have one table in PgSQL which is a job queue, each row represents > one job with several status flags, ID

Re: [GENERAL] Multithreaded queue in PgSQL

2008-06-10 Thread laser
Nikola Milutinovic wrote: Hi all. This may be trivial, but I cannot find good references for it. The problem is this: Suppose we have one table in PgSQL which is a job queue, each row represents one job with several status flags, IDs,... Several processes will attempt to access the queue an

Re: [GENERAL] Multithreaded queue in PgSQL

2008-06-10 Thread Nikola Milutinovic
> You may find that the PGQ component of skytools is what you want: > http://pgfoundry.org/projects/skytools > http://skytools.projects.postgresql.org/doc/ > http://skytools.projects.postgresql.org/doc/pgq-sql.html > Thanks, we will look into it. Still, I am surprised to learn that SQL as such

Re: [GENERAL] Multithreaded queue in PgSQL

2008-06-10 Thread Dimitri Fontaine
Le mardi 10 juin 2008, Nikola Milutinovic a écrit : > Suppose we have one table in PgSQL which is a job queue, each row > represents one job with several status flags, IDs,... Several processes > will attempt to access the queue and "take" their batch of jobs, the batch > will have some parameteriz

[GENERAL] Multithreaded queue in PgSQL

2008-06-10 Thread Nikola Milutinovic
Hi all. This may be trivial, but I cannot find good references for it. The problem is this: Suppose we have one table in PgSQL which is a job queue, each row represents one job with several status flags, IDs,... Several processes will attempt to access the queue and "take" their batch of jobs,