Re: [HACKERS] FW: REVIEW: Allow formatting in log_line_prefix

2013-09-26 Thread David Rowley
On Fri, Sep 27, 2013 at 10:28 AM, Robert Haas wrote: > On Thu, Sep 26, 2013 at 5:18 PM, Robert Haas > wrote: > > On Thu, Sep 26, 2013 at 3:55 PM, David Rowley > wrote: > >> I think I must have forgot to save it before I emailed it... > >> > >> Here's the correct version. > > > > Ah ha. Looks b

Re: [HACKERS] Minmax indexes

2013-09-26 Thread Amit Kapila
On Fri, Sep 27, 2013 at 11:49 AM, Amit Kapila wrote: > On Thu, Sep 26, 2013 at 1:46 AM, Alvaro Herrera > wrote: >> Amit Kapila escribió: >>> On Sun, Sep 15, 2013 at 5:44 AM, Alvaro Herrera >>> wrote: >> > >> >>> On Windows, patch gives below compilation errors: >>> src\backend\access\minmax\

Re: [HACKERS] Minmax indexes

2013-09-26 Thread Amit Kapila
On Thu, Sep 26, 2013 at 1:46 AM, Alvaro Herrera wrote: > Amit Kapila escribió: >> On Sun, Sep 15, 2013 at 5:44 AM, Alvaro Herrera >> wrote: > > >> On Windows, patch gives below compilation errors: >> src\backend\access\minmax\mmtuple.c(96): error C2057: expected >> constant expression > > I

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Peter Geoghegan
On Thu, Sep 26, 2013 at 12:33 PM, Robert Haas wrote: > I think one thing that's pretty clear at this point is that almost any > version of this feature could be optimized for either the insert case > or the update case. For example, my proposal could be modified to > search for a conflicting tupl

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Peter Geoghegan
On Thu, Sep 26, 2013 at 12:15 PM, Robert Haas wrote: >> Well, I think we can rule out value locks that are held for the >> duration of a transaction right away. That's just not going to fly. > > I think I agree with that. I don't think I remember hearing that proposed. I think I might have been

Re: [HACKERS] backup.sgml-cmd-v003.patch

2013-09-26 Thread Karl O. Pinc
On 09/26/2013 12:15:25 PM, Ivan Lezhnjov IV wrote: > > On Sep 3, 2013, at 6:56 AM, Karl O. Pinc wrote: > > > On 07/31/2013 12:08:12 PM, Ivan Lezhnjov IV wrote: > > > >> Patch filename: backup.sgml-cmd-v003.patch > >> > >> The third version of this patch takes into consideration feedback > >> r

Re: [HACKERS] [PATCH] bitmap indexes

2013-09-26 Thread Abhijit Menon-Sen
At 2013-09-26 08:39:05 -0700, jeff.ja...@gmail.com wrote: > > I don't know about grottiness, but it certainly seems like it would > deadlock like crazy. Hi Jeff. I don't understand the deadlock scenario you're thinking of. Could you explain, please? -- Abhijit -- Sent via pgsql-hackers mailin

Re: [HACKERS] record identical operator - Review

2013-09-26 Thread Bruce Momjian
On Thu, Sep 26, 2013 at 05:50:08PM -0400, Bruce Momjian wrote: > I think we need to see a patch from Kevin that shows all the row > comparisons documented and we can see the impact of the user-visibile > part. > > One interesting approach would be to only allow the operator to be > called from SPI

Re: [HACKERS] Wait free LW_SHARED acquisition

2013-09-26 Thread Andres Freund
On 2013-09-26 16:56:30 -0700, Peter Geoghegan wrote: > On Thu, Sep 26, 2013 at 3:55 PM, Andres Freund wrote: > > We have had several customers running postgres on bigger machines report > > problems on busy systems. Most recently one where a fully cached > > workload completely stalled in s_lock()

Re: [HACKERS] Wait free LW_SHARED acquisition

2013-09-26 Thread Peter Geoghegan
On Thu, Sep 26, 2013 at 3:55 PM, Andres Freund wrote: > We have had several customers running postgres on bigger machines report > problems on busy systems. Most recently one where a fully cached > workload completely stalled in s_lock()s due to the *shared* lwlock > acquisition in BufferAlloc() a

[HACKERS] Wait free LW_SHARED acquisition

2013-09-26 Thread Andres Freund
Hello, We have had several customers running postgres on bigger machines report problems on busy systems. Most recently one where a fully cached workload completely stalled in s_lock()s due to the *shared* lwlock acquisition in BufferAlloc() around the buffer partition lock. Increasing the paddin

Re: [HACKERS] FW: REVIEW: Allow formatting in log_line_prefix

2013-09-26 Thread Robert Haas
On Thu, Sep 26, 2013 at 5:18 PM, Robert Haas wrote: > On Thu, Sep 26, 2013 at 3:55 PM, David Rowley wrote: >> I think I must have forgot to save it before I emailed it... >> >> Here's the correct version. > > Ah ha. Looks better. > > I'm working on getting this committed now. Aside from some st

Re: [HACKERS] GIN improvements part 1: additional information

2013-09-26 Thread Alexander Korotkov
On Wed, Sep 25, 2013 at 5:22 PM, Peter Eisentraut wrote: > On 9/23/13 5:36 PM, Alexander Korotkov wrote: > > In the attached version of patch double finding of ItemPointer during > > insert is avoided. Overhead becomes lower as expected. > > Fails cpluspluscheck: > > ./src/include/access/gin_priv

Re: [HACKERS] [PATCH] pg_upgrade: Split off pg_fatal() from pg_log()

2013-09-26 Thread Bruce Momjian
On Thu, Sep 12, 2013 at 10:50:42PM -0400, Peter Eisentraut wrote: > The experiences with elog() and ereport() have shown that having one > function that can return or not depending on some log level parameter > isn't a good idea when you want to communicate well with the compiler. > In pg_upgrade,

Re: [HACKERS] record identical operator - Review

2013-09-26 Thread Bruce Momjian
On Thu, Sep 19, 2013 at 06:31:38PM -0400, Steve Singer wrote: > On 09/12/2013 06:27 PM, Kevin Grittner wrote: > >Attached is a patch for a bit of infrastructure I believe to be > >necessary for correct behavior of REFRESH MATERIALIZED VIEW > >CONCURRENTLY as well as incremental maintenance of matvi

Re: [HACKERS] FW: REVIEW: Allow formatting in log_line_prefix

2013-09-26 Thread Robert Haas
On Thu, Sep 26, 2013 at 3:55 PM, David Rowley wrote: > I think I must have forgot to save it before I emailed it... > > Here's the correct version. Ah ha. Looks better. I'm working on getting this committed now. Aside from some stylistic things, I've found one serious functional bug, which is

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Bruce Momjian
On Thu, Sep 26, 2013 at 03:33:34PM -0400, Robert Haas wrote: > On Thu, Sep 26, 2013 at 3:07 PM, Peter Geoghegan wrote: > > When you consider that the feature will frequently be used with the > > assumption that updating is a much more likely outcome, it becomes > > clear that we need to be careful

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Andres Freund
On 2013-09-27 05:41:26 +0900, Michael Paquier wrote: > In this case, doing a call to WaitForOldSnapshots after the swap phase > is enough. It was included in past versions of the patch but removed > in the last 2 versions. I don't think it is. I really, really suggest following the protocol used b

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Michael Paquier
On Thu, Sep 26, 2013 at 8:56 PM, Andres Freund wrote: > On 2013-09-26 20:47:33 +0900, Michael Paquier wrote: >> On Thu, Sep 26, 2013 at 8:43 PM, Andres Freund >> wrote: >> > On 2013-09-26 20:40:40 +0900, Michael Paquier wrote: >> >> On Thu, Sep 26, 2013 at 7:34 PM, Andres Freund >> >> wrote: >

Re: [HACKERS] gaussian distribution pgbench

2013-09-26 Thread Peter Eisentraut
On 9/20/13 2:42 AM, KONDO Mitsumasa wrote: > I create gaussinan distribution pgbench patch that can access records with > gaussian frequency. And I submit this commit fest. This patch no longer applies. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to yo

Re: [HACKERS] Re: Request for Patch Feedback: Lag & Lead Window Functions Can Ignore Nulls

2013-09-26 Thread Alvaro Herrera
I gave this a quick look. It took me a while to figure out how to apply it -- turns out you used the "ignore whitespace" option to diff, so the only way to apply it was with patch -p1 --ignore-whitespace. Please don't do that. I ran pgindent over the patched code and there were a number of chang

Re: [HACKERS] dynamic shared memory

2013-09-26 Thread Robert Haas
On Thu, Sep 26, 2013 at 2:45 PM, Jim Nasby wrote: > On 9/26/13 8:27 AM, Noah Misch wrote: >> >> On Tue, Sep 24, 2013 at 12:19:51PM -0400, Robert Haas wrote: >>> >>> >On Fri, Sep 20, 2013 at 7:44 AM, Andres Freund >>> > wrote: > > > >>Actually, I was wondering if we ought > > >>to have

Re: [HACKERS] FW: REVIEW: Allow formatting in log_line_prefix

2013-09-26 Thread David Rowley
On Fri, Sep 27, 2013 at 4:44 AM, Robert Haas wrote: > On Wed, Sep 25, 2013 at 4:46 AM, David Rowley > wrote: > > Ok, I think I've managed to narrow the performance gap to just about > nothing > > but noise, though to do this the code is now a bit bigger. I've added a > > series of tests to see i

Re: [HACKERS] Minmax indexes

2013-09-26 Thread Robert Haas
On Thu, Sep 26, 2013 at 2:58 PM, Jim Nasby wrote: > Why would we add additional code complexity when forks do the trick? That > seems like a step backwards, not forward. Well, they sorta do the trick, but see e.g. commit ece01aae479227d9836294b287d872c5a6146a11. I doubt that's the only code that

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Robert Haas
On Thu, Sep 26, 2013 at 3:07 PM, Peter Geoghegan wrote: > When you consider that the feature will frequently be used with the > assumption that updating is a much more likely outcome, it becomes > clear that we need to be careful about this sort of interplay. I think one thing that's pretty clear

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Robert Haas
On Tue, Sep 24, 2013 at 10:15 PM, Peter Geoghegan wrote: > Well, I think we can rule out value locks that are held for the > duration of a transaction right away. That's just not going to fly. I think I agree with that. I don't think I remember hearing that proposed. > If we're really lucky, ma

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Peter Geoghegan
On Thu, Sep 26, 2013 at 4:43 AM, Bruce Momjian wrote: > So, I guess my question is if we are only bloating on a contended > operation, do we expect that to happen so much that bloat is a problem? Maybe I could have done a better job of explaining the nature of my concerns around bloat. I am spec

Re: [HACKERS] Minmax indexes

2013-09-26 Thread Jim Nasby
On 9/26/13 12:00 PM, Robert Haas wrote: More generally, I fear we really opened a bag of worms with this relation fork stuff. Every time I turn around I run into a problem that could be solved by adding another relation fork. I'm not terribly sure that it was a good idea to go that way to begin

Re: [HACKERS] pgbench progress report improvements - split 3 v2 - A

2013-09-26 Thread Fabien COELHO
Patch (4): Redefine "latency" as reported by pgbench and report "lag" more. Here is a first partial patch, which focusses on measuring latency and reporting the measure under --progress. ** Improve pgbench measurements & progress report Measure transaction latency instead under --rate an

Re: [HACKERS] logical changeset generation v6

2013-09-26 Thread Steve Singer
On 09/25/2013 01:20 PM, Steve Singer wrote: On 09/25/2013 11:08 AM, Andres Freund wrote: On 2013-09-25 11:01:44 -0400, Steve Singer wrote: On 09/17/2013 10:31 AM, Andres Freund wrote: This patch set now fails to apply because of the commit "Rename various "freeze multixact" variables". And I

Re: [HACKERS] dynamic shared memory

2013-09-26 Thread Jim Nasby
On 9/26/13 8:27 AM, Noah Misch wrote: On Tue, Sep 24, 2013 at 12:19:51PM -0400, Robert Haas wrote: >On Fri, Sep 20, 2013 at 7:44 AM, Andres Freund wrote: > >>Actually, I was wondering if we ought > >>to have a directory under pgdata whose explicit charter it was to > >>contain files that shoul

Re: [HACKERS] Minmax indexes

2013-09-26 Thread Alvaro Herrera
Erik Rijkers wrote: > I have the impression it's not quite working correctly. > > The attached program returns different results for different values of > enable_bitmapscan (consistently). Clearly there's some bug somewhere. I'll investigate it more. > ( Btw, I had to make the max_locks_per_t

Re: [HACKERS] Minmax indexes

2013-09-26 Thread Alvaro Herrera
Robert Haas escribió: > On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera > wrote: > > Here's an updated version of this patch, with fixes to all the bugs > > reported so far. Thanks to Thom Brown, Jaime Casanova, Erik Rijkers and > > Amit Kapila for the reports. > > I'm not very happy with the us

Re: [HACKERS] backup.sgml-cmd-v003.patch

2013-09-26 Thread Ivan Lezhnjov IV
On Sep 3, 2013, at 6:56 AM, Karl O. Pinc wrote: > On 07/31/2013 12:08:12 PM, Ivan Lezhnjov IV wrote: > >> Patch filename: backup.sgml-cmd-v003.patch >> >> The third version of this patch takes into consideration feedback >> received after original submission (it can be read starting from this

[HACKERS] Extra functionality to createuser

2013-09-26 Thread Christopher Browne
Sitting on my todo list for a while has been to consider the idea of adding a bit of additional functionality to createuser. One of the functions of CREATE ROLE is to associate the role with other roles, thus... create role my_new_user nosuperuser nocreatedb login IN ROLE app_readonly_role

Re: [HACKERS] Minmax indexes

2013-09-26 Thread Robert Haas
On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera wrote: > Here's an updated version of this patch, with fixes to all the bugs > reported so far. Thanks to Thom Brown, Jaime Casanova, Erik Rijkers and > Amit Kapila for the reports. I'm not very happy with the use of a separate relation fork for st

Re: [HACKERS] FW: REVIEW: Allow formatting in log_line_prefix

2013-09-26 Thread Robert Haas
On Wed, Sep 25, 2013 at 4:46 AM, David Rowley wrote: > Ok, I think I've managed to narrow the performance gap to just about nothing > but noise, though to do this the code is now a bit bigger. I've added a > series of tests to see if the padding is > 0 and if it's not then I'm doing > things the o

Re: [HACKERS] [PATCH] bitmap indexes

2013-09-26 Thread Jeff Janes
On Wed, Sep 25, 2013 at 3:20 PM, Alvaro Herrera wrote: > Hi, > > Here are some quick items while skimming this patch. I am looking at > commit 6448de29d from your github repo, branch bmi. > > What's with the pg_bitmapindex stuff in pg_namespace.h? It doesn't seem > to be used anywhere. > > This

Re: [HACKERS] record identical operator

2013-09-26 Thread Robert Haas
On Tue, Sep 24, 2013 at 3:22 PM, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: >> Now I admit that's an arguable point. We could certainly define an >> intermediate notion of equality that is more equal than whatever = >> does, but not as equal as exact binary equality. > >

Re: [HACKERS] pgbench filler columns

2013-09-26 Thread Noah Misch
On Thu, Sep 26, 2013 at 03:23:30PM +0530, Pavan Deolasee wrote: > On Thu, Sep 26, 2013 at 2:05 PM, Pavan Deolasee > wrote: > > > While looking at the compressibility of WAL files generated by pgbench, > > which is close to 90%, I first thought its because of the "filler" column > > in the account

Re: [HACKERS] dynamic shared memory

2013-09-26 Thread Noah Misch
On Tue, Sep 24, 2013 at 12:19:51PM -0400, Robert Haas wrote: > On Fri, Sep 20, 2013 at 7:44 AM, Andres Freund wrote: > >> Actually, I was wondering if we ought > >> to have a directory under pgdata whose explicit charter it was to > >> contain files that shouldn't be copied as part of a base backu

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Bruce Momjian
On Thu, Sep 26, 2013 at 07:43:15AM -0400, Bruce Momjian wrote: > On Wed, Sep 25, 2013 at 08:48:11PM -0700, Peter Geoghegan wrote: > > On Wed, Sep 25, 2013 at 8:19 PM, Bruce Momjian wrote: > > > This thread had a lot of discussion about bloating. I wonder, does the > > > code check to see if there

Re: [HACKERS] Patch for fast gin cache performance improvement

2013-09-26 Thread Etsuro Fujita
Hi Ian, > This patch contains a performance improvement for the fast gin cache. As you > may know, the performance of the fast gin cache decreases with its size. > Currently, the size of the fast gin cache is tied to work_mem. The size of > work_mem can often be quite high. The large size of work_

[HACKERS] setting separate values of replication parameters to each standby to provide more granularity

2013-09-26 Thread Samrat Revagade
Hi, How about providing more granularity to replication, by setting separate values of replication parameters to each standby for example: standby1.wal_sender_timeout= 50s standby2.wal_sender_timeout= 40s The idea is to allow configuration of standby servers such that they have there own set of

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Andres Freund
On 2013-09-26 20:47:33 +0900, Michael Paquier wrote: > On Thu, Sep 26, 2013 at 8:43 PM, Andres Freund wrote: > > On 2013-09-26 20:40:40 +0900, Michael Paquier wrote: > >> On Thu, Sep 26, 2013 at 7:34 PM, Andres Freund > >> wrote: > >> > On 2013-09-26 12:13:30 +0900, Michael Paquier wrote: > >> >

Re: [HACKERS] Patch for fail-back without fresh backup

2013-09-26 Thread Pavan Deolasee
On Thu, Sep 19, 2013 at 4:02 PM, Fujii Masao wrote: > > > Hmm... when synchronous_transfer is set to data_flush, > IMO the intuitive behaviors are > > (1) synchronous_commit = on > A data flush should wait for the corresponding WAL to be > flushed in the standby > > (2) synchronous_commit = remot

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Michael Paquier
On Thu, Sep 26, 2013 at 8:43 PM, Andres Freund wrote: > On 2013-09-26 20:40:40 +0900, Michael Paquier wrote: >> On Thu, Sep 26, 2013 at 7:34 PM, Andres Freund >> wrote: >> > On 2013-09-26 12:13:30 +0900, Michael Paquier wrote: >> >> > 2) I don't think the drop algorithm used now is correct. Your

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Bruce Momjian
On Wed, Sep 25, 2013 at 08:48:11PM -0700, Peter Geoghegan wrote: > On Wed, Sep 25, 2013 at 8:19 PM, Bruce Momjian wrote: > > This thread had a lot of discussion about bloating. I wonder, does the > > code check to see if there is a matching row _before_ adding any data? > > That's pretty much wh

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Andres Freund
On 2013-09-26 20:40:40 +0900, Michael Paquier wrote: > On Thu, Sep 26, 2013 at 7:34 PM, Andres Freund wrote: > > On 2013-09-26 12:13:30 +0900, Michael Paquier wrote: > >> > 2) I don't think the drop algorithm used now is correct. Your > >> > index_concurrent_set_dead() sets both indisvalid = false

Re: [HACKERS] pgbench - exclude pthread_create() from connection start timing

2013-09-26 Thread Fabien COELHO
pgbench changes, when adding the throttling stuff. Having the start time taken when the thread really starts is just sanity, and I needed that just to rule out that it was the source of the "strange" measures. I don't get it; why is taking the time just after pthread_create() more sane than ta

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Michael Paquier
On Thu, Sep 26, 2013 at 7:34 PM, Andres Freund wrote: > On 2013-09-26 12:13:30 +0900, Michael Paquier wrote: >> > 2) I don't think the drop algorithm used now is correct. Your >> > index_concurrent_set_dead() sets both indisvalid = false and indislive = >> > false at the same time. It does so afte

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Andres Freund
On 2013-09-26 12:13:30 +0900, Michael Paquier wrote: > > 2) I don't think the drop algorithm used now is correct. Your > > index_concurrent_set_dead() sets both indisvalid = false and indislive = > > false at the same time. It does so after doing a WaitForVirtualLocks() - > > but that's not suffici

Re: [HACKERS] pgbench filler columns

2013-09-26 Thread Pavan Deolasee
On Thu, Sep 26, 2013 at 2:05 PM, Pavan Deolasee wrote: > While looking at the compressibility of WAL files generated by pgbench, > which is close to 90%, I first thought its because of the "filler" column > in the accounts table. But a comment in pgbench.c says: > > /* > * Note: TPC-B req

Re: [HACKERS] SSI freezing bug

2013-09-26 Thread Heikki Linnakangas
On 23.09.2013 01:07, Hannu Krosing wrote: On 09/20/2013 12:55 PM, Heikki Linnakangas wrote: Hi, Prompted by Andres Freund's comments on my Freezing without Write I/O patch, I realized that there's there's an existing bug in the way predicate locking handles freezing (or rather, it doesn't handl

Re: [HACKERS] [PATCH] bitmap indexes

2013-09-26 Thread Abhijit Menon-Sen
At 2013-09-25 19:20:17 -0300, alvhe...@2ndquadrant.com wrote: > > Here are some quick items while skimming this patch. Great, that gives me plenty to work on. At this point, I think it's appropriate to mark this patch as returned with feedback (which I will do), since the changes needed seem fair

[HACKERS] pgbench filler columns

2013-09-26 Thread Pavan Deolasee
While looking at the compressibility of WAL files generated by pgbench, which is close to 90%, I first thought its because of the "filler" column in the accounts table. But a comment in pgbench.c says: /* * Note: TPC-B requires at least 100 bytes per row, and the "filler" * fields in

Re: [HACKERS] pgbench progress report improvements - split 3 v2

2013-09-26 Thread Fabien COELHO
My feelings on the patch split haven't changed; your three bullet points call for four separate patches. Conflicting patches are bad, but dependent patches are okay; Indeed, this is the only solution if you do not want one patch. Note that it will not possible to backtrack one of the patch b

Re: [HACKERS] [PATCH] bitmap indexes

2013-09-26 Thread Antonin Houska
On 09/26/2013 12:20 AM, Alvaro Herrera wrote: > This led me to research how these indexes are stored. I note that what > we're doing here is to create another regular table and a btree index on > top of it, and those are the ones that actually store the index data. > This seems grotty and complet