Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE - visibility semantics

2013-09-27 Thread Peter Geoghegan
On Tue, Sep 24, 2013 at 2:14 AM, Andres Freund wrote: > Various messages are discussing semantics around visibility. I by now > have a hard time keeping track. So let's keep the discussion of the > desired semantics to this thread. Yes, it's pretty complicated. I meant to comment on this here, b

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-27 Thread Peter Geoghegan
On Fri, Sep 27, 2013 at 5:36 AM, Robert Haas wrote: >> I don't have another idea either. In fact, I'd go so far as to say >> that doing any third thing that's better than those two to any >> reasonable person is obviously impossible. But I'd add that we simple >> cannot rollback at read committed,

Re: [HACKERS] record identical operator - Review

2013-09-27 Thread Kevin Grittner
Bruce Momjian wrote: > On Thu, Sep 19, 2013 at 06:31:38PM -0400, Steve Singer wrote: >> On 09/12/2013 06:27 PM, Kevin Grittner wrote: >>> Attached is a patch for a bit of infrastructure I believe to be >>> necessary for correct behavior of REFRESH MATERIALIZED VIEW >>> CONCURRENTLY as well as incr

Re: [HACKERS] Extra functionality to createuser

2013-09-27 Thread Christopher Browne
Attached is a patch implementing the "-g / --roles" option for createuser. I'll be attaching it to the open CommitFest shortly. createuser.diff Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgr

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-09-27 Thread Andres Freund
Hi, What confuses me is that pg_read_barrier() is just a compiler barrier on x86[-64] in barrier.h. According to my knowledge it needs to be an lfence or the full barrier? The linked papers from Paul McKenney - which are a great read - seem to agree? On 2013-09-27 23:12:17 +0200, Andres Freund wr

Re: [HACKERS] Wait free LW_SHARED acquisition

2013-09-27 Thread Andres Freund
On 2013-09-27 14:46:50 +0200, Florian Pflug wrote: > On Sep27, 2013, at 00:55 , Andres Freund wrote: > > So the goal is to have LWLockAcquire(LW_SHARED) never block unless > > somebody else holds an exclusive lock. To produce enough appetite for > > reading the rest of the long mail, here's some n

Re: [HACKERS] logical changeset generation v6

2013-09-27 Thread Andres Freund
Hi Steve, On 2013-09-27 17:06:59 -0400, Steve Singer wrote: > >I've determined that when in this test the walsender seems to be hitting > >this when it is decode the transactions that are behind the slonik > >commands to add tables to replication (set add table, set add sequence). > >This is befor

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-09-27 Thread Andres Freund
On 2013-09-27 23:28:37 +0300, Heikki Linnakangas wrote: > On 27.09.2013 22:00, Merlin Moncure wrote: > >Attached is simplified patch that replaces the spinlock with a read > >barrier based on a suggestion made by Andres offlist. The approach > >has different performance characteristics -- a barrie

Re: [HACKERS] logical changeset generation v6

2013-09-27 Thread Steve Singer
On 09/26/2013 02:47 PM, Steve Singer wrote: I've determined that when in this test the walsender seems to be hitting this when it is decode the transactions that are behind the slonik commands to add tables to replication (set add table, set add sequence). This is before the SUBSCRIBE SET i

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-09-27 Thread Peter Geoghegan
On Fri, Sep 27, 2013 at 1:28 PM, Heikki Linnakangas wrote: > Yep. I only added the first caller of the barriers altogether in the > xlog-insertion scaling patch. Robert wrote the infrastructure in 9.3, but it > wasn't used until now, in 9.4. FWIW, it was actually during 9.2 development that Rober

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-09-27 Thread Heikki Linnakangas
On 27.09.2013 22:00, Merlin Moncure wrote: On Fri, Sep 27, 2013 at 7:56 AM, Merlin Moncure wrote: On Thu, Sep 26, 2013 at 10:14 PM, Merlin Moncure wrote: On Thu, Sep 26, 2013 at 6:08 PM, Andres Freund wrote: On 2013-08-27 12:17:55 -0500, Merlin Moncure wrote: On Tue, Aug 27, 2013 at 10:55

Re: [HACKERS] Re: Request for Patch Feedback: Lag & Lead Window Functions Can Ignore Nulls

2013-09-27 Thread Heikki Linnakangas
On 27.09.2013 14:12, Robert Haas wrote: On Thu, Sep 26, 2013 at 4:20 PM, Alvaro Herrera wrote: But how do we ensure that the BMS is allocated in a context? You'd have to switch contexts each time you call bms_add_member. I don't have a good answer to this. The coding of bms_add_member is p

Re: [HACKERS] Minmax indexes

2013-09-27 Thread Jim Nasby
On 9/27/13 1:43 PM, Greg Stark wrote: Honestly, I think we actually need more obfuscation between what happens on the filesystem and the rest of postgres... we're starting to look at areas where that would help. For example, the recent idea of being able to truncate individual relation files a

Re: [HACKERS] [PERFORM] Cpu usage 100% on slave. s_lock problem.

2013-09-27 Thread Merlin Moncure
On Fri, Sep 27, 2013 at 7:56 AM, Merlin Moncure wrote: > On Thu, Sep 26, 2013 at 10:14 PM, Merlin Moncure wrote: >> On Thu, Sep 26, 2013 at 6:08 PM, Andres Freund >> wrote: >>> On 2013-08-27 12:17:55 -0500, Merlin Moncure wrote: On Tue, Aug 27, 2013 at 10:55 AM, Andres Freund wrote:

Re: [HACKERS] Minmax indexes

2013-09-27 Thread Greg Stark
On Fri, Sep 27, 2013 at 7:22 PM, Jim Nasby wrote: > > Yeah, we obviously kept things simpler when adding forks in order to get the > feature out the door. There's improvements that need to be made. But IMHO > that's not reason to automatically avoid forks; we need to consider the cost > of impr

Re: [HACKERS] Extra functionality to createuser

2013-09-27 Thread Jim Nasby
On 9/27/13 6:01 AM, Robert Haas wrote: Thus, the above commands might be replaced by: >PGUSER=postgres createuser -D -S -l -g >app_readonly_role,app2_writer_role my_new_user > >Would this be worth adding to the ToDo list? I'd be inclined to favor a patch implementing this. +1 -- Jim C. Nas

Re: [HACKERS] Minmax indexes

2013-09-27 Thread Jim Nasby
On 9/26/13 2:46 PM, Robert Haas wrote: On Thu, Sep 26, 2013 at 2:58 PM, Jim Nasby wrote: Why would we add additional code complexity when forks do the trick? That seems like a step backwards, not forward. Well, they sorta do the trick, but see e.g. commit ece01aae479227d9836294b287d872c5a6146

Re: [HACKERS] backup.sgml-cmd-v003.patch

2013-09-27 Thread Robert Haas
On Fri, Sep 27, 2013 at 9:49 AM, Karl O. Pinc wrote: > Hi Robert, > On 09/27/2013 05:56:52 AM, Robert Haas wrote: > >> 1. Attempting to encourage people to consider custom format dumps. > >> What's important is what you can do... > > Your critique seems obvious in retrospect. Sorry you had > to s

[HACKERS] hstore extension version screwup

2013-09-27 Thread Andrew Dunstan
When adding json support to hstore, I made a major blunder and added the new functionality to the existing sql script instead of bumping the version, renaming the script and adding an update script. This was lazy and there's no real excuse, although I will note that it was a mistake far too

Re: [HACKERS] logical changeset generation v6.1

2013-09-27 Thread Andres Freund
On 2013-09-27 16:35:53 +0100, Thom Brown wrote: > On 27 September 2013 16:14, Andres Freund wrote: > > Hi, > > > > Attached you can find an updated version of the series taking in some of > > the review comments (the others are queued, not ignored), including: > > * split of things from the big "I

Re: [HACKERS] logical changeset generation v6.1

2013-09-27 Thread Thom Brown
On 27 September 2013 16:14, Andres Freund wrote: > Hi, > > Attached you can find an updated version of the series taking in some of > the review comments (the others are queued, not ignored), including: > * split of things from the big "Introduce wal decoding via ..." patch > * fix the bug Steve n

Re: [HACKERS] backup.sgml-cmd-v003.patch

2013-09-27 Thread Karl O. Pinc
Hi Robert, On 09/27/2013 05:56:52 AM, Robert Haas wrote: > 1. Attempting to encourage people to consider custom format dumps. > What's important is what you can do... Your critique seems obvious in retrospect. Sorry you had to step in here and do my job. The above point is particularly sali

Re: [HACKERS] Wait free LW_SHARED acquisition

2013-09-27 Thread Florian Pflug
On Sep27, 2013, at 00:55 , Andres Freund wrote: > So the goal is to have LWLockAcquire(LW_SHARED) never block unless > somebody else holds an exclusive lock. To produce enough appetite for > reading the rest of the long mail, here's some numbers on a > pgbench -j 90 -c 90 -T 60 -S (-i -s 10) on a

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-27 Thread Robert Haas
On Thu, Sep 26, 2013 at 11:58 PM, Peter Geoghegan wrote: > On Thu, Sep 26, 2013 at 12:15 PM, Robert Haas wrote: >>> Well, I think we can rule out value locks that are held for the >>> duration of a transaction right away. That's just not going to fly. >> >> I think I agree with that. I don't thi

Re: [HACKERS] Re: Request for Patch Feedback: Lag & Lead Window Functions Can Ignore Nulls

2013-09-27 Thread Robert Haas
On Thu, Sep 26, 2013 at 4:20 PM, Alvaro Herrera wrote: > But how do we ensure that the BMS is allocated in a context? You'd have > to switch contexts each time you call bms_add_member. I don't have a > good answer to this. The coding of bms_add_member is pretty funky. Why doesn't it just repal

Re: [HACKERS] Extra functionality to createuser

2013-09-27 Thread Robert Haas
On Thu, Sep 26, 2013 at 1:04 PM, Christopher Browne wrote: > Sitting on my todo list for a while has been to consider the idea of > adding a bit of additional functionality to createuser. > > One of the functions of CREATE ROLE is to associate the role with > other roles, thus... > >create rol

Re: [HACKERS] backup.sgml-cmd-v003.patch

2013-09-27 Thread Robert Haas
On Thu, Sep 26, 2013 at 1:15 PM, Ivan Lezhnjov IV wrote: > Thanks for a detailed response. I attached a patch file that builds on your > corrections and introduces some of the edits discussed above. This patch makes at least five unrelated sets of changes: 1. Attempting to encourage people to c

Re: [HACKERS] insert throw error when year field len > 4 for timestamptz datatype

2013-09-27 Thread Haribabu kommi
On 27 September 2013 15:04 Rushabh Lathia wrote: >>On Tue, Sep 17, 2013 at 6:23 PM, Haribabu kommi >>mailto:haribabu.ko...@huawei.com>> wrote: >>I feel changing the year value to accept the length (>4) is not simple. >>So many places the year length crossing more than length 4 is not considered. >

Re: [HACKERS] Patch for fail-back without fresh backup

2013-09-27 Thread Sawada Masahiko
On Fri, Sep 27, 2013 at 5:18 PM, Pavan Deolasee wrote: > On Fri, Sep 27, 2013 at 1:28 PM, Sawada Masahiko > wrote: >> >> >> > >> >> Thank you for comment. I think it is good simple idea. >> In your opinion, if synchronous_transfer is set 'all' and >> synchronous_commit is set 'on', >> the master

Re: [HACKERS] insert throw error when year field len > 4 for timestamptz datatype

2013-09-27 Thread Rushabh Lathia
Sorry for delay in reply. On Tue, Sep 17, 2013 at 6:23 PM, Haribabu kommi wrote: > On Tue, 17 September 2013 14:33 Rushabh Lathia wrote: > > >>On Mon, Sep 16, 2013 at 7:22 PM, Haribabu kommi < > haribabu.ko...@huawei.com> wrote: > > *>>>**On *14 August 2013 Rushabh Lathia wrote: >

Re: [HACKERS] Patch for fast gin cache performance improvement

2013-09-27 Thread Etsuro Fujita
I wrote: > I had a look over this patch. I think this patch is interesting and very useful. > Here are my review points: > 8. I think there are no issues in this patch. However, I have one question: > how this patch works in the case where gin_fast_limit/fast_cache_size = 0? In > this case, in

Re: [HACKERS] Wait free LW_SHARED acquisition

2013-09-27 Thread Andres Freund
On 2013-09-27 11:11:56 +0300, Heikki Linnakangas wrote: > On 27.09.2013 10:21, Andres Freund wrote: > >Heaps better. In the case causing this investigation lots of the pages > >with hot spinlocks were the simply the same ones over and over again, > >partitioning the lockspace won't help much there.

Re: [HACKERS] Patch for fail-back without fresh backup

2013-09-27 Thread Pavan Deolasee
On Fri, Sep 27, 2013 at 1:28 PM, Sawada Masahiko wrote: > > > > > Thank you for comment. I think it is good simple idea. > In your opinion, if synchronous_transfer is set 'all' and > synchronous_commit is set 'on', > the master wait for data flush eve if user sets synchronous_commit to > 'local' o

Re: [HACKERS] Wait free LW_SHARED acquisition

2013-09-27 Thread Andres Freund
On 2013-09-27 09:57:07 +0200, Andres Freund wrote: > On 2013-09-27 09:21:05 +0200, Andres Freund wrote: > > > >So the goal is to have LWLockAcquire(LW_SHARED) never block unless > > > >somebody else holds an exclusive lock. To produce enough appetite for > > > >reading the rest of the long mail, he

Re: [HACKERS] Wait free LW_SHARED acquisition

2013-09-27 Thread Heikki Linnakangas
On 27.09.2013 10:21, Andres Freund wrote: Hi, On 2013-09-27 10:14:46 +0300, Heikki Linnakangas wrote: On 27.09.2013 01:55, Andres Freund wrote: We have had several customers running postgres on bigger machines report problems on busy systems. Most recently one where a fully cached workload com

Re: [HACKERS] Patch for fail-back without fresh backup

2013-09-27 Thread Sawada Masahiko
On Thu, Sep 26, 2013 at 8:54 PM, Pavan Deolasee wrote: > > On Thu, Sep 19, 2013 at 4:02 PM, Fujii Masao wrote: >> >> >> >> Hmm... when synchronous_transfer is set to data_flush, >> IMO the intuitive behaviors are >> >> (1) synchronous_commit = on >> A data flush should wait for the corresponding

Re: [HACKERS] Wait free LW_SHARED acquisition

2013-09-27 Thread Andres Freund
On 2013-09-27 09:21:05 +0200, Andres Freund wrote: > > >So the goal is to have LWLockAcquire(LW_SHARED) never block unless > > >somebody else holds an exclusive lock. To produce enough appetite for > > >reading the rest of the long mail, here's some numbers on a > > >pgbench -j 90 -c 90 -T 60 -S (-

Re: [HACKERS] Wait free LW_SHARED acquisition

2013-09-27 Thread Andres Freund
Hi, On 2013-09-27 10:14:46 +0300, Heikki Linnakangas wrote: > On 27.09.2013 01:55, Andres Freund wrote: > >We have had several customers running postgres on bigger machines report > >problems on busy systems. Most recently one where a fully cached > >workload completely stalled in s_lock()s due to

Re: [HACKERS] Wait free LW_SHARED acquisition

2013-09-27 Thread Heikki Linnakangas
On 27.09.2013 01:55, Andres Freund wrote: Hello, We have had several customers running postgres on bigger machines report problems on busy systems. Most recently one where a fully cached workload completely stalled in s_lock()s due to the *shared* lwlock acquisition in BufferAlloc() around the b

Re: [HACKERS] Some interesting news about Linux 3.12 OOM

2013-09-27 Thread Daniel Farina
On Wed, Sep 25, 2013 at 8:00 AM, Greg Stark wrote: > > On Wed, Sep 25, 2013 at 12:15 AM, Daniel Farina wrote: >> >> Enable the memcg OOM killer only for user faults, where it's really the >> only option available. > > > Is this really a big deal? I would expect most faults to be user faults. > >

Re: [HACKERS] pgbench filler columns

2013-09-27 Thread Pavan Deolasee
On Thu, Sep 26, 2013 at 7:20 PM, Noah Misch wrote: > On Thu, Sep 26, 2013 at 03:23:30PM +0530, Pavan Deolasee wrote: > > > Should we just fix the comment and say its applicable for all tables > except > > accounts ? > > Please do. > > How about something like this ? Patch attached. Thanks, Pavan