Re: [HACKERS] Extension Templates S03E11

2013-12-03 Thread Jeff Davis
On Tue, 2013-12-03 at 14:31 -0500, Tom Lane wrote: > Stephen Frost writes: > > When it comes to dump/reload, I'd much rather see a mechanism which uses > > our deep understanding of the extension's objects (as database objects) > > to implement the dump/reload than a text blob which is carried for

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Claudio Freire
On Wed, Dec 4, 2013 at 4:28 AM, Tatsuo Ishii wrote: >>> Can we avoid the Linux kernel problem by simply increasing our shared >>> buffer size, say up to 80% of memory? >> It will be swap more easier. > > Is that the case? If the system has not enough memory, the kernel > buffer will be used for ot

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Tatsuo Ishii
>> Can we avoid the Linux kernel problem by simply increasing our shared >> buffer size, say up to 80% of memory? > It will be swap more easier. Is that the case? If the system has not enough memory, the kernel buffer will be used for other purpose, and the kernel cache will not work very well any

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2013-12-03 Thread Shigeru Hanada
2013/11/29 Kohei KaiGai : > I merged all the propositions from Jim. Thanks, it made the documentation > quality better. Also, I fixed up cosmetic stuff around whitespace <-> tab. I found some typos in documents and comments. Please see attached patch for detail. -- Shigeru HANADA fix_typo.pat

Re: [HACKERS] Extension Templates S03E11

2013-12-03 Thread Jeff Davis
On Tue, 2013-12-03 at 14:41 -0500, Stephen Frost wrote: > * Jeff Davis (pg...@j-davis.com) wrote: > The extra catalog tables which store SQL scripts in text columns is one > of my main objections to the as-proposed Extension Templates. OK, that's what I thought. This seems like the root of your

Re: [HACKERS] Problem with displaying "wide" tables in psql

2013-12-03 Thread Sergey Muraviov
Thank you for this trick. It would be nice if this trick was documented. However, with the pager I can't see wide value on one screen, select and copy it entirely. And I have to press many keys to find the necessary part of the value. There is no such problems with the patch. 2013/12/3 Pavel Ste

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Stefan Kaltenbrunner
On 12/04/2013 05:40 AM, Peter Eisentraut wrote: > On Tue, 2013-12-03 at 14:44 -0800, Josh Berkus wrote: >> Would certainly be nice. Realistically, getting good automated >> performace tests will require paying someone like Greg S., Mark or me >> for 6 solid months to develop them, since worthwhile

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2013-12-03 Thread Shigeru Hanada
Hi KaiGai-san, 2013/11/29 Kohei KaiGai : > The attached ones are the revised patches. > > I merged all the propositions from Jim. Thanks, it made the documentation > quality better. Also, I fixed up cosmetic stuff around whitespace <-> tab. > > An actual code changes are to follow the changes in

[HACKERS] Changes in Trigger Firing

2013-12-03 Thread Sameer Kumar
Hi all, I am doing some changes in trigger firing mechanism (as a POC first, I will share my work if people find it useful and if it works). I am going to extend the trigger calls to child tables. So that in trigger definition I can create triggers with CASCADED TRUE | FALSE option. This will en

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-03 Thread Álvaro Hernández Tortosa
Hi Peter, Dimitri, thank you for your comments. On 03/12/13 22:27, Dimitri Fontaine wrote: Peter Eisentraut writes: On 12/1/13, 2:24 PM, Álvaro Hernández Tortosa wrote: IMHO, defining a new syntax for the postgreql.conf file format, that is suitable for writing and parsing,

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-03 Thread Claudio Freire
On Wed, Dec 4, 2013 at 12:57 AM, Amit Kapila wrote: >> As a quick side, we also repeated the same experiment on an EC2 instance >> with 16 CPU cores, and found that the scale out behavior became worse there. >> (We also tried increasing the shared_buffers to 30 GB. This change >> completely solved

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Peter Eisentraut
On Tue, 2013-12-03 at 14:44 -0800, Josh Berkus wrote: > Would certainly be nice. Realistically, getting good automated > performace tests will require paying someone like Greg S., Mark or me > for 6 solid months to develop them, since worthwhile open source > performance test platforms currently d

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread KONDO Mitsumasa
(2013/12/04 11:28), Tatsuo Ishii wrote: Magnus Hagander writes: On Tue, Dec 3, 2013 at 11:44 PM, Josh Berkus wrote: Would certainly be nice. Realistically, getting good automated performace tests will require paying someone like Greg S., Mark or me for 6 solid months to develop them, since w

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-03 Thread Amit Kapila
On Tue, Dec 3, 2013 at 7:11 PM, Metin Doslu wrote: > We have several independent tables on a multi-core machine serving Select > queries. These tables fit into memory; and each Select queries goes over one > table's pages sequentially. In this experiment, there are no indexes or > table joins. > >

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Tatsuo Ishii
> Magnus Hagander writes: >> On Tue, Dec 3, 2013 at 11:44 PM, Josh Berkus wrote: >>> Would certainly be nice. Realistically, getting good automated >>> performace tests will require paying someone like Greg S., Mark or me >>> for 6 solid months to develop them, since worthwhile open source >>> p

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread KONDO Mitsumasa
(2013/12/04 4:00), Andres Freund wrote: On 2013-12-03 13:46:28 -0500, Robert Haas wrote: On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello wrote: On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse wrote: Hi Fabrizio, looks good to me. I did some testing on 9.2.4, 9.2.5 and HEAD. It app

Re: [HACKERS] WITHIN GROUP patch

2013-12-03 Thread Tom Lane
Andrew Gierth writes: > "Tom" == Tom Lane writes: > Tom> 1. I really hate the way you've overloaded the transvalue to do > Tom> something that has approximately nothing to do with transition > Tom> state (and haven't updated catalogs.sgml to explain that, > Tom> either). Seems like it'd be c

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread KONDO Mitsumasa
(2013/11/30 5:34), Fabrízio de Royes Mello wrote: On Fri, Nov 29, 2013 at 5:49 AM, KONDO Mitsumasa mailto:kondo.mitsum...@lab.ntt.co.jp>> wrote: > * Problem1 > Your patch does not code recovery.conf.sample about recovery_time_delay. > Please add it. Fixed. OK. It seems no problem. > * Prob

Re: [HACKERS] Logging WAL when updating hintbit

2013-12-03 Thread Sawada Masahiko
On Tue, Dec 3, 2013 at 5:34 PM, Sawada Masahiko wrote: > On Tue, Dec 3, 2013 at 4:28 PM, Michael Paquier > wrote: >> On Tue, Dec 3, 2013 at 3:30 PM, Sawada Masahiko >> wrote: >> >> After more thinking... >> Before performing a rewind on a node, what we need to know is that >> log_hint_bits was

Re: [HACKERS] logical changeset generation v6.7

2013-12-03 Thread Andres Freund
On 2013-12-03 19:15:53 +0900, Kyotaro HORIGUCHI wrote: > - In heapam.c, it seems to be better replacing t_self only >during logical decoding. I don't see what'd be gained by that except make the test matrix bigger at no gain. > - Before that, In LogicalDecodingAcquireFreeSlot, lock window >

Re: [HACKERS] logical changeset generation v6.7

2013-12-03 Thread Andres Freund
Hi, On 2013-12-03 17:13:05 +0900, Kyotaro HORIGUCHI wrote: > - Some patches have line offset to master. Rebase needed. Will send the rebased version as soon as I've addressed your comments. > = 0001: > > - You assined HeapTupleGetOid(tuple) into relid to read in >several points but no

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Joshua D. Drake
On 12/03/2013 03:15 PM, Josh Berkus wrote: It's *always* much easier to get money for features than for other things. Earlier this year I was really hoping that our new corporate community members, who seemed to be interested in testing, would put some serious resources behind this. When pres

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Andres Freund
On 2013-12-03 19:55:40 -0300, Alvaro Herrera wrote: > I added a new isolation spec to test this specific case, and noticed > something that seems curious to me when that test is run in REPEATABLE > READ mode: when the UPDATE is aborted, the concurrent FOR UPDATE gets a > "can't serialize due to con

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Joshua D. Drake
On 12/03/2013 03:02 PM, Magnus Hagander wrote: On Tue, Dec 3, 2013 at 11:44 PM, Josh Berkus Would certainly be nice. Realistically, getting good automated performace tests will require paying someone like Greg S., Mark or me for 6 solid months to develop them, since worthwhile op

[HACKERS] ruleutils vs. empty targetlists

2013-12-03 Thread Tom Lane
Thinking some more about bug #8648, it occurred to me that ruleutils.c isn't exactly prepared for the case either: regression=# create table nocols(); CREATE TABLE regression=# create view vv1 as select exists (select * from nocols); CREATE VIEW regression=# \d+ vv1 View "public.

[HACKERS] ANALYZE sampling is too good

2013-12-03 Thread Greg Stark
At multiple conferences I've heard about people trying all sorts of gymnastics to avoid ANALYZE which they expect to take too long and consume too much I/O. This is especially a big complain after upgrades when their new database performs poorly until the new statistics are in and they did pg_upgra

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Josh Berkus
Magnus, > So in order to get *testing* we need to pay somebody. But to build a great > database server, we can rely on volunteer efforts or sponsorship from > companies who are interested in moving the project forward? That hardly > seems right... Either it's just not high enough on peoples prior

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Alvaro Herrera
Alvaro Herrera wrote: > Attached is a patch to fix it. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services *** a/src/backend/utils/time/tqual.c --- b/src/backend/utils/time/tqual.c *** *** 789,801 HeapTupleSatis

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Alvaro Herrera
Magnus Hagander wrote: > On Tue, Dec 3, 2013 at 11:44 PM, Josh Berkus wrote: > > Would certainly be nice. Realistically, getting good automated > > performace tests will require paying someone like Greg S., Mark or me > > for 6 solid months to develop them, since worthwhile open source > > perfo

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Andres Freund
On 2013-12-03 15:46:09 -0500, Tom Lane wrote: > Noah Misch writes: > > I'd rather have an readily-verifiable fix that changes WAL format than a > > tricky fix that avoids doing so. So, modulo not having seen the change, +1. > > Yeah, same here. I am afraid it won't be *that* simple. We still nee

Re: [HACKERS] Cube extension kNN support

2013-12-03 Thread Stas Kelvich
Hi. > cube and earthdistance regression tests fail. Code updated to work with current HEAD. Also added tests to cover new functionality. > Do you have any benchmarks ? This patch just introduces functionality of calculating distances between cubes, so this code don't interfere much with kN

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Tom Lane
Magnus Hagander writes: > On Tue, Dec 3, 2013 at 11:44 PM, Josh Berkus wrote: >> Would certainly be nice. Realistically, getting good automated >> performace tests will require paying someone like Greg S., Mark or me >> for 6 solid months to develop them, since worthwhile open source >> performa

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-12-03 Thread Timothy Garnett
On Tue, Dec 3, 2013 at 12:14 PM, Bruce Momjian wrote: > On Wed, Apr 24, 2013 at 03:33:42PM -0400, Andrew Dunstan wrote: > > > > On 04/23/2013 07:53 PM, Timothy Garnett wrote: > ... > > >Attached is two diffs off of the REL9_2_4 tag that I've been > > >using. The first is a simple change that ser

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Magnus Hagander
On Tue, Dec 3, 2013 at 11:44 PM, Josh Berkus wrote: > On 12/03/2013 12:15 PM, Stefan Kaltenbrunner wrote: > > We are in no way different and I would like to note that we do not have > > any form of sensible performance related regression testing either. > > I would even argue that there is ton mo

Re: [HACKERS] Proposed feature: Selective Foreign Keys

2013-12-03 Thread Tom Dunstan
On 4 December 2013 01:24, Robert Haas wrote: > Yeah, more or less, but the key is ensuring that it wouldn't let you > create the constraint in the first place if the partial index > specified *didn't* match the WHERE clause. For example, suppose the > partial index says WHERE parent_entity = 'eve

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Alvaro Herrera
Noah Misch wrote: > On Tue, Dec 03, 2013 at 04:08:23PM +0100, Andres Freund wrote: > > > (For clarity, the other problem demonstrated by the test spec is also a > > > 9.3.2 > > > regression.) > > > > Yea, I just don't see why yet... Looking now. > > Sorry, my original report was rather terse.

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Josh Berkus
On 12/03/2013 12:15 PM, Stefan Kaltenbrunner wrote: > We are in no way different and I would like to note that we do not have > any form of sensible performance related regression testing either. > I would even argue that there is ton more regression testing (be it > performance or otherwise) going

Re: [HACKERS] [bug fix] "pg_ctl stop" times out when it should respond quickly

2013-12-03 Thread Tom Lane
"MauMau" writes: > The problem occurs in the sequence below: > 1. postmaster creates $PGDATA/postmaster.pid. > 2. postmaster tries to resolve the value of listen_addresses to IP > addresses. This took about 15 seconds in my failure scenario. > 3. During 2, pg_ctl sends SIGTERM to postmaster. >

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Tom Lane
Alvaro Herrera writes: > Tom Lane wrote: >> After some discussion, the core committee has concluded that we should go >> ahead with the already-wrapped releases. 9.2.6 and below are good anyway, >> and despite this issue 9.3.2 is an improvement over 9.3.1. We'll plan to >> do a 9.3.3 as soon as

Re: [HACKERS] Recovery bug in GIN, missing full page image

2013-12-03 Thread Heikki Linnakangas
On 12/03/2013 04:39 PM, Heikki Linnakangas wrote: While looking at Alexander's GIN patch, I noticed an ancient bug in the WAL-logging of GIN entry-tree insertions. entryPlaceToPage and dataPlacetoPage functions don't make a full-page image of the page, when inserting a downlink on a non-leaf page

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Joshua D. Drake
On 12/03/2013 12:35 PM, Tom Lane wrote: Stefan Kaltenbrunner writes: If we care about our performance on various operating systems it is _OUR_ responsibility to track that closely and automated and report back and only if that feedback loop fails to work we are actually in a real position to c

[HACKERS] [patch] libpq: Support TLSv1.1+ (was: fe-secure.c and SSL/TLS)

2013-12-03 Thread Marko Kreen
As pointed out by Jeffrey Walton, only SSLv23_method() will negotiate higher TLS versions than immediately requested. All other _method() functions will negotiate only one specific version. Attached are two patches: * libpq.tls11plus.diff Use SSLv23_method() instead TLSv1_method in fe-secure.c.

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-03 Thread Dimitri Fontaine
Peter Eisentraut writes: > On 12/1/13, 2:24 PM, Álvaro Hernández Tortosa wrote: >> IMHO, defining a new syntax for the postgreql.conf file format, >> that is suitable for writing and parsing, or using an already existing, >> well-known, programmatic syntax, could offer a solution for all t

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-03 Thread Peter Eisentraut
On 12/1/13, 2:24 PM, Álvaro Hernández Tortosa wrote: > IMHO, defining a new syntax for the postgreql.conf file format, > that is suitable for writing and parsing, or using an already existing, > well-known, programmatic syntax, could offer a solution for all the > problems/limitations above

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Alvaro Herrera
Tom Lane wrote: > After some discussion, the core committee has concluded that we should go > ahead with the already-wrapped releases. 9.2.6 and below are good anyway, > and despite this issue 9.3.2 is an improvement over 9.3.1. We'll plan to > do a 9.3.3 as soon as the multixact situation can b

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Tom Lane
Noah Misch writes: > On Tue, Dec 03, 2013 at 07:26:38PM +0100, Andres Freund wrote: >> On 2013-12-03 13:14:38 -0500, Noah Misch wrote: >>> On Tue, Dec 03, 2013 at 04:37:58PM +0100, Andres Freund wrote: I currently don't see fixing the errorneous freezing of lockers (not the updater thoug

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Tom Lane
Stefan Kaltenbrunner writes: > If we care about our performance on various operating systems it is > _OUR_ responsibility to track that closely and automated and report back > and only if that feedback loop fails to work we are actually in a real > position to consider something as drastical as co

Re: [HACKERS] logical changeset generation v6.7

2013-12-03 Thread Robert Haas
On Tue, Dec 3, 2013 at 8:18 AM, Andres Freund wrote: >> I've taken a crack at rewriting >> this logic, and the result looks cleaner and simpler to me, but I >> haven't tested it beyond the fact that it passes make check. See what >> you think. > > Hm. I think it actually will not abort early in

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Stefan Kaltenbrunner
On 12/03/2013 08:23 PM, Josh Berkus wrote: > On 12/03/2013 10:59 AM, Joshua D. Drake wrote: >> This seems rather half cocked. I read the article. They found a problem, >> that really will only affect a reasonably small percentage of users, >> created a test case, reported it, and a patch was produc

Re: [HACKERS] logical changeset generation v6.7

2013-12-03 Thread Robert Haas
On Tue, Dec 3, 2013 at 8:24 AM, Andres Freund wrote: > On 2013-11-28 21:15:18 -0500, Robert Haas wrote: >> OK, I've committed the patch to adjust the definition of >> IsSystemRelation()/IsSystemClass() and add >> IsCatalogRelation()/IsCatalogClass(). > > Thanks for taking care of this! > >> I kib

Re: [HACKERS] UNNEST with multiple args, and TABLE with multiple funcs

2013-12-03 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Another issue is that if you are used to the Oracle syntax, in which an > UNNEST() is presumed, it's not exactly clear that TABLE ROWS, or any other > phrase including TABLE, *doesn't* also imply an UNNEST. So to me that's > kind of a strike against Stephen

Re: [HACKERS] Extension Templates S03E11

2013-12-03 Thread Stephen Frost
* Jeff Davis (pg...@j-davis.com) wrote: > On Tue, 2013-12-03 at 09:20 -0500, Stephen Frost wrote: > > * Jeff Davis (pg...@j-davis.com) wrote: > > > Stephen mentioned using external tools and/or metadata, but to me that > > > sounds like it would require porting the extension away from what's on > >

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread Simon Riggs
On 18 October 2013 19:03, Fabrízio de Royes Mello wrote: > The attached patch is a continuation of Robert's work [1]. Reviewing v2... > I made some changes: > - use of Latches instead of pg_usleep, so we don't have to wakeup regularly. OK > - call HandleStartupProcInterrupts() before CheckFo

Re: [HACKERS] 9.2.1 & index-only scans : abnormal heap fetches after VACUUM FULL

2013-12-03 Thread Bruce Momjian
On Tue, Dec 3, 2013 at 02:01:52PM -0500, Robert Haas wrote: > On Tue, Dec 3, 2013 at 11:25 AM, Bruce Momjian wrote: > > On Thu, Nov 28, 2013 at 05:38:05PM -0500, Bruce Momjian wrote: > >> > I wonder if we ought to mark each page as all-visible in > >> > raw_heap_insert() when we first initialize

Re: [HACKERS] Extension Templates S03E11

2013-12-03 Thread Tom Lane
Stephen Frost writes: > When it comes to dump/reload, I'd much rather see a mechanism which uses > our deep understanding of the extension's objects (as database objects) > to implement the dump/reload than a text blob which is carried forward > from major version to major version and may even fai

Re: [HACKERS] UNNEST with multiple args, and TABLE with multiple funcs

2013-12-03 Thread Tom Lane
Noah Misch writes: > On Tue, Dec 03, 2013 at 10:03:32AM -0500, Stephen Frost wrote: >> Alright, for my 2c, I like having this syntax include 'TABLE' simply >> because it's what folks coming from Oracle might be looking for. >> Following from that, to keep it distinct from the spec's notion of >> '

Re: [HACKERS] Extension Templates S03E11

2013-12-03 Thread Stephen Frost
* Jeff Davis (pg...@j-davis.com) wrote: > This sounds like Inline Extensions to me, which was previously proposed. I've not looked at that proposal very carefully, but I agree that what we're describing is a lot closer to 'inline extensions' than 'extension templates'. > If I recall, that proposa

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Josh Berkus
On 12/03/2013 10:59 AM, Joshua D. Drake wrote: > This seems rather half cocked. I read the article. They found a problem, > that really will only affect a reasonably small percentage of users, > created a test case, reported it, and a patch was produced. "Users with at least one file bigger than 5

Re: [HACKERS] Problem with displaying "wide" tables in psql

2013-12-03 Thread Pavel Stehule
Hello do you know a pager less trick http://merlinmoncure.blogspot.cz/2007/10/better-psql-with-less.html Regards Pavel Stehule 2013/12/3 Sergey Muraviov > Hi. > > Psql definitely have a problem with displaying "wide" tables. > Even in expanded mode, they look horrible. > So I tried to solve

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Alvaro Herrera
Andres Freund wrote: > On 2013-12-03 13:49:49 -0500, Noah Misch wrote: > > On Tue, Dec 03, 2013 at 07:26:38PM +0100, Andres Freund wrote: > > > On 2013-12-03 13:14:38 -0500, Noah Misch wrote: > > > > Not fixing it at all is the real no-go. We'd take both of those > > > > undesirables > > > > befo

[HACKERS] Problem with displaying "wide" tables in psql

2013-12-03 Thread Sergey Muraviov
Hi. Psql definitely have a problem with displaying "wide" tables. Even in expanded mode, they look horrible. So I tried to solve this problem. Before the patch: postgres=# \x 1 Expanded display (expanded) is on. postgres=# \pset border 2 Border style (border) is 2. postgres=# select * from pg_sta

[HACKERS] Parallel Select query performance and shared buffers

2013-12-03 Thread Metin Doslu
We have several independent tables on a multi-core machine serving Select queries. These tables fit into memory; and each Select queries goes over one table's pages sequentially. In this experiment, there are no indexes or table joins. When we send concurrent Select queries to these tables, query

Re: [HACKERS] 9.2.1 & index-only scans : abnormal heap fetches after VACUUM FULL

2013-12-03 Thread Robert Haas
On Tue, Dec 3, 2013 at 11:25 AM, Bruce Momjian wrote: > On Thu, Nov 28, 2013 at 05:38:05PM -0500, Bruce Momjian wrote: >> > I wonder if we ought to mark each page as all-visible in >> > raw_heap_insert() when we first initialize it, and then clear the flag >> > when we come across a tuple that isn

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Joshua D. Drake
On 12/03/2013 10:44 AM, Josh Berkus wrote: All, https://lkml.org/lkml/2013/11/24/133 What this means for us: http://citusdata.com/blog/72-linux-memory-manager-and-your-big-data It seems clear that Kernel.org, since 2.6, has been in the business of pushing major, hackish, changes to the IO s

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread Andres Freund
On 2013-12-03 13:46:28 -0500, Robert Haas wrote: > On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello > wrote: > > On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse > > wrote: > >> > >> Hi Fabrizio, > >> > >> looks good to me. I did some testing on 9.2.4, 9.2.5 and HEAD. It > >> applies and c

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Andres Freund
On 2013-12-03 15:40:44 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > > I wondered about that myself. How would you suggest the format to look > > like? > > ISTM per tuple we'd need: > > > > * OffsetNumber off > > * uint16 infomask > > * TransactionId xmin > > * TransactionId xmax > > >

Re: [HACKERS] UNNEST with multiple args, and TABLE with multiple funcs

2013-12-03 Thread Noah Misch
On Tue, Dec 03, 2013 at 10:03:32AM -0500, Stephen Frost wrote: > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > After sleeping on it, your other suggestion of TABLE OF, or possibly > > TABLE FROM, is starting to grow on me. > > > > Who else has an opinion? > > Alright, for my 2c, I like having this s

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Andres Freund
On 2013-12-03 13:49:49 -0500, Noah Misch wrote: > On Tue, Dec 03, 2013 at 07:26:38PM +0100, Andres Freund wrote: > > On 2013-12-03 13:14:38 -0500, Noah Misch wrote: > > > Not fixing it at all is the real no-go. We'd take both of those > > > undesirables > > > before just tolerating the lost locks

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread Joshua D. Drake
On 12/03/2013 10:46 AM, Robert Haas wrote: On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello wrote: On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse wrote: Hi Fabrizio, looks good to me. I did some testing on 9.2.4, 9.2.5 and HEAD. It applies and compiles w/o errors or warnings. I s

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Robert Haas
On Tue, Dec 3, 2013 at 1:44 PM, Josh Berkus wrote: > All, > > https://lkml.org/lkml/2013/11/24/133 > > What this means for us: > > http://citusdata.com/blog/72-linux-memory-manager-and-your-big-data > > It seems clear that Kernel.org, since 2.6, has been in the business of > pushing major, hackish

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Noah Misch
On Tue, Dec 03, 2013 at 07:26:38PM +0100, Andres Freund wrote: > On 2013-12-03 13:14:38 -0500, Noah Misch wrote: > > On Tue, Dec 03, 2013 at 04:37:58PM +0100, Andres Freund wrote: > > > I currently don't see fixing the errorneous freezing of lockers (not the > > > updater though) without changing t

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread Robert Haas
On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello wrote: > On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse > wrote: >> >> Hi Fabrizio, >> >> looks good to me. I did some testing on 9.2.4, 9.2.5 and HEAD. It >> applies and compiles w/o errors or warnings. I set up a master and two >> hot st

[HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread Josh Berkus
All, https://lkml.org/lkml/2013/11/24/133 What this means for us: http://citusdata.com/blog/72-linux-memory-manager-and-your-big-data It seems clear that Kernel.org, since 2.6, has been in the business of pushing major, hackish, changes to the IO stack without testing them or even thinking too

Re: [HACKERS] Extension Templates S03E11

2013-12-03 Thread Jeff Davis
On Tue, 2013-12-03 at 09:20 -0500, Stephen Frost wrote: > * Jeff Davis (pg...@j-davis.com) wrote: > > Stephen mentioned using external tools and/or metadata, but to me that > > sounds like it would require porting the extension away from what's on > > PGXN today. > > Not at all- and that'd be the

Re: [HACKERS] WITHIN GROUP patch

2013-12-03 Thread Andrew Gierth
> "Tom" == Tom Lane writes: Tom> 1. I really hate the way you've overloaded the transvalue to do Tom> something that has approximately nothing to do with transition Tom> state (and haven't updated catalogs.sgml to explain that, Tom> either). Seems like it'd be cleaner to just hardwire a

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Alvaro Herrera
Andres Freund wrote: > I wondered about that myself. How would you suggest the format to look > like? > ISTM per tuple we'd need: > > * OffsetNumber off > * uint16 infomask > * TransactionId xmin > * TransactionId xmax > > I don't see why we'd need infomask2, but so far being overly skimpy in >

Re: [HACKERS] Extension Templates S03E11

2013-12-03 Thread Jeff Davis
On Tue, 2013-12-03 at 10:08 +0200, Heikki Linnakangas wrote: > Another perspective is that that's already a situation we'd rather not > have. Let's not make it worse by introducing a third way to install an > extension, which again requires the extension author to package the > extension differe

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Andres Freund
On 2013-12-03 13:11:13 -0500, Tom Lane wrote: > Andres Freund writes: > > Any idea how to cheat our way out of that one given the current way > > heap_freeze_tuple() works (running on both primary and standby)? My only > > idea was to MultiXactIdWait() if !InRecovery but that's extremly grotty. >

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Alvaro Herrera
Noah Misch wrote: > > On 2013-12-03 10:29:54 -0500, Noah Misch wrote: > > > Sorry, my original report was rather terse. I speak of the scenario > > > exercised > > > by the second permutation in that isolationtester spec. The multixact is > > > later than VACUUM's cutoff_multi, so 9.3.1 does no

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Andres Freund
On 2013-12-03 13:14:38 -0500, Noah Misch wrote: > On Tue, Dec 03, 2013 at 04:37:58PM +0100, Andres Freund wrote: > > On 2013-12-03 10:29:54 -0500, Noah Misch wrote: > > > Sorry, my original report was rather terse. I speak of the scenario > > > exercised > > > by the second permutation in that is

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Magnus Hagander
On Tue, Dec 3, 2013 at 7:20 PM, Tom Lane wrote: > Magnus Hagander writes: > > On Tue, Dec 3, 2013 at 7:11 PM, Tom Lane wrote: > >> Maybe we should just bite the bullet and change the WAL format for > >> heap_freeze (inventing an all-new record type, not repurposing the old > >> one, and allowin

Re: [HACKERS] Dynamic configuration via LDAP in postmaster

2013-12-03 Thread Vasily Soshnikov
Thank you for the advice, nowadays we(company where I work) use such scheme but that scheme is not always useful at the stage of development of the back-end infrastructure. Also we have found a better solution : have ldap for dynamic configuraion out of the box. So question is following: if you go

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Tom Lane
Magnus Hagander writes: > On Tue, Dec 3, 2013 at 7:11 PM, Tom Lane wrote: >> Maybe we should just bite the bullet and change the WAL format for >> heap_freeze (inventing an all-new record type, not repurposing the old >> one, and allowing WAL replay to continue to accept the old one). The >> imp

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Alvaro Herrera
Tom Lane wrote: > Andres Freund writes: > > Any idea how to cheat our way out of that one given the current way > > heap_freeze_tuple() works (running on both primary and standby)? My only > > idea was to MultiXactIdWait() if !InRecovery but that's extremly grotty. > > We can't even realistically

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Magnus Hagander
On Tue, Dec 3, 2013 at 7:11 PM, Tom Lane wrote: > Andres Freund writes: > > Any idea how to cheat our way out of that one given the current way > > heap_freeze_tuple() works (running on both primary and standby)? My only > > idea was to MultiXactIdWait() if !InRecovery but that's extremly grotty

Re: [HACKERS] Extension Templates S03E11

2013-12-03 Thread David E. Wheeler
On Dec 3, 2013, at 9:14 AM, Dimitri Fontaine wrote: > I understand that it can happen, it still really sucks when it does. > > I have not followed this project closely, Dimitri, but I for one have appreciated your tenacity in following through on it. Extensions are awesome, thanks to you, a

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Noah Misch
On Tue, Dec 03, 2013 at 04:37:58PM +0100, Andres Freund wrote: > On 2013-12-03 10:29:54 -0500, Noah Misch wrote: > > Sorry, my original report was rather terse. I speak of the scenario > > exercised > > by the second permutation in that isolationtester spec. The multixact is > > later than VACUU

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Tom Lane
Andres Freund writes: > Any idea how to cheat our way out of that one given the current way > heap_freeze_tuple() works (running on both primary and standby)? My only > idea was to MultiXactIdWait() if !InRecovery but that's extremly grotty. > We can't even realistically create a new multixact wit

Re: [HACKERS] WITHIN GROUP patch

2013-12-03 Thread Tom Lane
Atri Sharma writes: > Please find attached the latest patch for WITHIN GROUP. This patch is > after fixing the merge conflicts. I've started to look at this patch now. I have a couple of immediate reactions to the catalog changes: 1. I really hate the way you've overloaded the transvalue to do

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Andres Freund
On 2013-12-03 12:22:33 -0500, Tom Lane wrote: > Andres Freund writes: > >> On 2013-12-03 09:48:23 -0500, Tom Lane wrote: > >>> Is this bad enough that we need to re-wrap the release? > > > After looking, I think I am revising my opinion. The broken locking > > behaviour (outside of freeze, which

Re: [HACKERS] UNNEST with multiple args, and TABLE with multiple funcs

2013-12-03 Thread Pavel Stehule
2013/12/3 Stephen Frost > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > After sleeping on it, your other suggestion of TABLE OF, or possibly > > TABLE FROM, is starting to grow on me. > > > > Who else has an opinion? > > Alright, for my 2c, I like having this syntax include 'TABLE' simply > because

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread Fabrízio de Royes Mello
On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse wrote: > Hi Fabrizio, > > looks good to me. I did some testing on 9.2.4, 9.2.5 and HEAD. It > applies and compiles w/o errors or warnings. I set up a master and two > hot standbys replicating from the master, one with 5 minutes delay and > one withou

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Tom Lane
Andres Freund writes: >> On 2013-12-03 09:48:23 -0500, Tom Lane wrote: >>> Is this bad enough that we need to re-wrap the release? > After looking, I think I am revising my opinion. The broken locking > behaviour (outside of freeze, which I don't see how we can fix in time), > is actually bad. >

Re: [HACKERS] Extension Templates S03E11

2013-12-03 Thread Dimitri Fontaine
Stephen Frost writes: > I understand that you once proposed that and it was shot down but I > think we need to move past that now that we've seen what the alternative > is.. That isn't to say anything about the code or about you > specifically, but, for my part, I really don't like nor see the va

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-12-03 Thread Bruce Momjian
On Wed, Apr 24, 2013 at 03:33:42PM -0400, Andrew Dunstan wrote: > > On 04/23/2013 07:53 PM, Timothy Garnett wrote: > >Hi All, > > > >Currently the -j option to pg_restore, which allows for > >parallelization in the restore, can only be used if the input file > >is a regular file and not, for ex.,

Re: [HACKERS] pgsql: Fix a couple of bugs in MultiXactId freezing

2013-12-03 Thread Andres Freund
Hi Alvaro, Noah, On 2013-12-03 15:57:10 +0100, Andres Freund wrote: > On 2013-12-03 09:48:23 -0500, Tom Lane wrote: > > Andres Freund writes: > > > On 2013-12-03 00:47:07 -0500, Noah Misch wrote: > > >> The test spec additionally > > >> covers a (probably-related) assertion failure, new in 9.3.2

Re: Review: [HACKERS] ECPG infrastructure changes part 1, was: Re: ECPG fixes

2013-12-03 Thread Boszormenyi Zoltan
Thanks for the review. 2013-12-03 16:48 keltezéssel, Antonin Houska írta: The changes are very well divided into logical units, so I think I could understand the ideas. I'm not too familiar with the ecpg internals, so consider this at least a coding review. git apply: Clean, except for on

Re: [HACKERS] Dynamic configuration via LDAP in postmaster

2013-12-03 Thread Heikki Linnakangas
On 12/03/2013 05:44 PM, Vasily Soshnikov wrote: I need advise about where is best place for adding such features. Currently I found that 'postmaster' have event loop(including handling SIGHUP) inside PostgressMain(postgress.c) for realoding configuration file, based on my investigation my plan

Re: [HACKERS] Extension Templates S03E11

2013-12-03 Thread Dimitri Fontaine
Robert Haas writes: > In more normal cases, however, the system can (and probably should) > figure out what was intended by choosing the *shortest* path to get to > the intended version. For example, if someone ships 1.0, 1.0--1.1, > 1.1, and 1.1--1.2, the system should choose to run 1.1 and then

Re: [HACKERS] 9.2.1 & index-only scans : abnormal heap fetches after VACUUM FULL

2013-12-03 Thread Bruce Momjian
On Thu, Nov 28, 2013 at 05:38:05PM -0500, Bruce Momjian wrote: > > I wonder if we ought to mark each page as all-visible in > > raw_heap_insert() when we first initialize it, and then clear the flag > > when we come across a tuple that isn't all-visible. We could try to > > set hint bits on the tu

  1   2   >