Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-03-14 Thread Merlin Moncure
On Tue, Feb 1, 2011 at 11:18 AM, Tom Lane wrote: > "Yngve Nysaeter Pettersen" writes: >> To avoid having the processes trample each other's queries (the first >> attempt was to select the first matching entries of the table, which >> caused one to block all other transactions), one of the steps I

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-03-14 Thread Andy Colson
On 3/14/2011 10:13 AM, Yngve N. Pettersen (Developer Opera Software ASA) wrote: Hello all, Just a quick update of how it went. I ended up using code similar to a combination of Andy Colson's and David Johnston's suggestions below, and performance is back at what is was before. Thanks for the su

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-03-14 Thread Yngve N. Pettersen (Developer Opera Software ASA)
Hello all, Just a quick update of how it went. I ended up using code similar to a combination of Andy Colson's and David Johnston's suggestions below, and performance is back at what is was before. Thanks for the suggestions BTW: AFAICT I never got a response from Tom Lane about whether it was

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-02 Thread Yngve N. Pettersen (Developer Opera Software ASA)
Hello David, On Wed, 02 Feb 2011 01:36:15 +0100, David Johnston wrote: If random sampling is desirable would the following construct limit locking only to the sampled rows? SELECT id FROM tasktable WHERE id IN (SELECT random_id_sample()) FOR UPDATE The "random_id_sample" would supply a

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread David Johnston
If random sampling is desirable would the following construct limit locking only to the sampled rows? SELECT id FROM tasktable WHERE id IN (SELECT random_id_sample()) FOR UPDATE The "random_id_sample" would supply a configurable group of IDs off of tasktable which the FOR UPDATE would then lock

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread Yngve Nysaeter Pettersen
On Tue, 01 Feb 2011 20:04:31 +0100, Andy Colson wrote: On 2/1/2011 12:51 PM, Yngve Nysaeter Pettersen wrote: So, AFAICT I am afraid it would not work in the general case for my project :( . However, it might be useful in somebody else's project :) . No, I didn't think it would work for y

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread Radosław Smogura
Hmm... May I ask how this look in details. If e.g. I do select * from myeshop offset 100 limit 20, I have 1000 rows which rows will be locked? a) 0 to 120, or b) all rows will be locked.? Kind regards, Radek Tom Lane Tuesday 01 February 2011 18:18:17 > In 9.0, LIMIT/OFFSET processing is done

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread Andy Colson
On 2/1/2011 12:51 PM, Yngve Nysaeter Pettersen wrote: Thanks Andy, On Tue, 01 Feb 2011 19:29:08 +0100, Andy Colson wrote: On 2/1/2011 12:10 PM, Yngve Nysaeter Pettersen wrote: On Tue, 01 Feb 2011 18:11:23 +0100, Andy Colson I think the random limit thing is a race condition itself. Whenev

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread Yngve Nysaeter Pettersen
Thanks Andy, On Tue, 01 Feb 2011 19:29:08 +0100, Andy Colson wrote: On 2/1/2011 12:10 PM, Yngve Nysaeter Pettersen wrote: On Tue, 01 Feb 2011 18:11:23 +0100, Andy Colson I think the random limit thing is a race condition itself. Whenever you have multiple processes hitting the same rows yo

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread Andy Colson
On 2/1/2011 12:10 PM, Yngve Nysaeter Pettersen wrote: On Tue, 01 Feb 2011 18:11:23 +0100, Andy Colson I think the random limit thing is a race condition itself. Whenever you have multiple processes hitting the same rows you're going to run into problems. Have you thought of using a sequence ins

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread Yngve Nysaeter Pettersen
On Tue, 01 Feb 2011 18:11:23 +0100, Andy Colson wrote: On 2/1/2011 10:59 AM, Yngve Nysaeter Pettersen wrote: Hi, Thanks for the quick answer, Andy. On Tue, 01 Feb 2011 16:19:17 +0100, Andy Colson wrote: So, if I understand correctly, you: q = SELECT record_id FROM queue WHERE project_

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread Yngve Nysaeter Pettersen
On Tue, 01 Feb 2011 18:18:17 +0100, Tom Lane wrote: "Yngve Nysaeter Pettersen" writes: To avoid having the processes trample each other's queries (the first attempt was to select the first matching entries of the table, which caused one to block all other transactions), one of the steps I too

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread Tom Lane
"Yngve Nysaeter Pettersen" writes: > To avoid having the processes trample each other's queries (the first > attempt was to select the first matching entries of the table, which > caused one to block all other transactions), one of the steps I took was > to select a set of idle rows at a ran

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread Andy Colson
On 2/1/2011 10:59 AM, Yngve Nysaeter Pettersen wrote: Hi, Thanks for the quick answer, Andy. On Tue, 01 Feb 2011 16:19:17 +0100, Andy Colson wrote: So, if I understand correctly, you: q = SELECT record_id FROM queue WHERE project_id = my_project AND state = idle LIMIT n OFFSET i FOR UPDATE

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread Yngve Nysaeter Pettersen
Hi, Thanks for the quick answer, Andy. On Tue, 01 Feb 2011 16:19:17 +0100, Andy Colson wrote: So, if I understand correctly, you: q = SELECT record_id FROM queue WHERE project_id = my_project AND state = idle LIMIT n OFFSET i FOR UPDATE while not q.eof update queu

Re: [GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread Andy Colson
On 2/1/2011 6:32 AM, Yngve Nysaeter Pettersen wrote: Hello all, I am in the process of migrating a system from Postgresql 8.3 to 9.0, and have run into a problem with the task queue systems I am using. The task queue controls the allocation of tasks between about 1000 processes working in paral

[GENERAL] Select for update with offset interferes with concurrent transactions

2011-02-01 Thread Yngve Nysaeter Pettersen
Hello all, I am in the process of migrating a system from Postgresql 8.3 to 9.0, and have run into a problem with the task queue systems I am using. The task queue controls the allocation of tasks between about 1000 processes working in parallel, and is essentially a table of record_id