Re: [GENERAL] Processing a work queue

2007-05-01 Thread Andrew - Supernews
On 2007-04-30, "John D. Burger" <[EMAIL PROTECTED]> wrote: > Andrew - Supernews wrote: > >>> Anyone have any ideas on how to handle a work queue? >> >> Advisory locks (userlocks in pre-8.2). > > Can someone explain why these are a better fit than whatever locks > SELECT FOR UPDATE acquires? They

Re: [GENERAL] Processing a work queue

2007-05-01 Thread Merlin Moncure
On 5/1/07, Alban Hertroys <[EMAIL PROTECTED]> wrote: Whether that improves your performance pretty much depends on the queue sizes, the frequency of processing and the load processing causes... Heh, I just realize we have a work queue table here as well, I'd like to know the result. the optimiz

Re: [GENERAL] Processing a work queue

2007-05-01 Thread Alban Hertroys
Merlin Moncure wrote: > take another look at my example. there are two things happening that > have to be logically combined into one operation. one is checking the > last_value column of two sequences and the other is the nextval(). > the advisory lock protects against this: > > session a: worke

Re: [GENERAL] Processing a work queue

2007-05-01 Thread Merlin Moncure
On 5/1/07, Alban Hertroys <[EMAIL PROTECTED]> wrote: Merlin Moncure wrote: > ok, here's an example. I was thinking that my sequence idea might not > be safe because of race conditions revolving around querying the > sequence table. Here is how I might use advisory locks eliminate the I've seen

Re: [GENERAL] Processing a work queue

2007-05-01 Thread Alban Hertroys
Merlin Moncure wrote: > ok, here's an example. I was thinking that my sequence idea might not > be safe because of race conditions revolving around querying the > sequence table. Here is how I might use advisory locks eliminate the I've seen your name pop up regularly on this list (or are you fr

Re: [GENERAL] Processing a work queue

2007-05-01 Thread Merlin Moncure
On 4/30/07, John D. Burger <[EMAIL PROTECTED]> wrote: Can someone explain why [advisory locks] are a better fit than whatever locks SELECT FOR UPDATE acquires? ok, here's an example. I was thinking that my sequence idea might not be safe because of race conditions revolving around querying the

Re: [GENERAL] Processing a work queue

2007-05-01 Thread Merlin Moncure
On 4/30/07, John D. Burger <[EMAIL PROTECTED]> wrote: Andrew - Supernews wrote: >> Anyone have any ideas on how to handle a work queue? > > Advisory locks (userlocks in pre-8.2). Can someone explain why these are a better fit than whatever locks SELECT FOR UPDATE acquires? They are fast, and

Re: [GENERAL] Processing a work queue

2007-04-30 Thread John D. Burger
Andrew - Supernews wrote: Anyone have any ideas on how to handle a work queue? Advisory locks (userlocks in pre-8.2). Can someone explain why these are a better fit than whatever locks SELECT FOR UPDATE acquires? Thanks. - John D. Burger MITRE ---(end of br

Re: [GENERAL] Processing a work queue

2007-04-28 Thread Andrew - Supernews
On 2007-04-26, Steve Crawford <[EMAIL PROTECTED]> wrote: > Anyone have any ideas on how to handle a work queue? Advisory locks (userlocks in pre-8.2). -- Andrew, Supernews http://www.supernews.com - individual and corporate NNTP services ---(end of broadcast)

Re: [GENERAL] Processing a work queue

2007-04-28 Thread Lexington Luthor
Steve Crawford wrote: begin; select item-id, item-info from the-queue where available order by priority limit 1 for update; update the-queue set status = 'assigned' where item-id = previously-selected-item-id; commit; I do something similar in one of my apps: BEGIN; up

Re: [GENERAL] Processing a work queue

2007-04-27 Thread Chris Browne
[EMAIL PROTECTED] ("John D. Burger") writes: > I wrote: > >> I use a variant of The Tom Lane Solution previously pointed to, >> your Plan 1 is very similar. > > Hmm, per that pointed-to post: > > http://archives.postgresql.org/pgsql-general/2003-05/msg00351.php > > I decided to run a periodic vac

Re: [GENERAL] Processing a work queue

2007-04-27 Thread John D. Burger
I decided to run a periodic vacuum on my work queue. Lo and behold, I get this: ERROR: tuple concurrently updated Which PG version is this, and do you have autovacuum enabled? 7.4.8, so no autovac, right? In addition, all of my workers locked up, apparently indefinitely. [ squint..

Re: [GENERAL] Processing a work queue

2007-04-27 Thread Tom Lane
"John D. Burger" <[EMAIL PROTECTED]> writes: > I decided to run a periodic vacuum on my work queue. Lo and behold, > I get this: >ERROR: tuple concurrently updated Which PG version is this, and do you have autovacuum enabled? Awhile back it was possible to get this error if autovac and a

Re: [GENERAL] Processing a work queue

2007-04-27 Thread John D. Burger
I wrote: I use a variant of The Tom Lane Solution previously pointed to, your Plan 1 is very similar. Hmm, per that pointed-to post: http://archives.postgresql.org/pgsql-general/2003-05/msg00351.php I decided to run a periodic vacuum on my work queue. Lo and behold, I get this: ERR

Re: [GENERAL] Processing a work queue

2007-04-26 Thread John D. Burger
Steve Crawford wrote: Anyone have any ideas on how to handle a work queue? I've been thinking about optimizing this process for quite a while. I use a variant of The Tom Lane Solution previously pointed to, your Plan 1 is very similar. This does not produce desirable results. In the case

Re: [GENERAL] Processing a work queue

2007-04-26 Thread Merlin Moncure
On 4/27/07, Merlin Moncure <[EMAIL PROTECTED]> wrote: how about this: create table job(job_id int, [...]) create sequence worker; couple typos: here is an example that works: create table job(job_id serial); create sequence worker; -- get next available job create function next_job() returns

Re: [GENERAL] Processing a work queue

2007-04-26 Thread Merlin Moncure
On 4/27/07, Steve Crawford <[EMAIL PROTECTED]> wrote: Anyone have any ideas on how to handle a work queue? I've been thinking about optimizing this process for quite a while. Basically, my queue table consists of a few-hundred-thousand records describing "things to do". To pare things to the min

Re: [GENERAL] Processing a work queue

2007-04-26 Thread Alexander Staubo
On 4/27/07, Steve Crawford <[EMAIL PROTECTED]> wrote: Anyone have any ideas on how to handle a work queue? I've been thinking about optimizing this process for quite a while. I have been using PostgreSQL for the exact same thing, except I have not yet reached the stage where I need to process q

[GENERAL] Processing a work queue

2007-04-26 Thread Steve Crawford
Anyone have any ideas on how to handle a work queue? I've been thinking about optimizing this process for quite a while. Basically, my queue table consists of a few-hundred-thousand records describing "things to do". To pare things to the minimum, a queue record can be considered to have a status