Re: [HACKERS] Using pg_rewind for differential backup

2014-11-27 Thread Heikki Linnakangas
On 11/28/2014 09:30 AM, Michael Paquier wrote: On Thu, Nov 27, 2014 at 9:39 PM, Sameer Kumar wrote: Can we tweak pg_rewind to take differential backups in PostgreSQL? I was wondering can we hack the pg_rewind code to print the details of the file which have been modified compared to a target s

Re: [HACKERS] Allocation in critical section after node exits archive recovery

2014-11-27 Thread Heikki Linnakangas
On 11/28/2014 08:35 AM, Michael Paquier wrote: Hi all, While performing some archive recovery tests I found the following assertion failure once standby gets out of archive recovery (HEAD at a5eb85e): (lldb) * thread #1: tid = 0x, 0x7fff8faaf282 libsystem_kernel.dylib`__pthread_kill + 10

Re: [HACKERS] Using pg_rewind for differential backup

2014-11-27 Thread Michael Paquier
On Thu, Nov 27, 2014 at 9:39 PM, Sameer Kumar wrote: > Can we tweak pg_rewind to take differential backups in PostgreSQL? > I was wondering can we hack the pg_rewind code to print the details of the > file which have been modified compared to a target server. The list output > can then be used fo

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-11-27 Thread Michael Paquier
On Fri, Nov 28, 2014 at 1:30 PM, Rahila Syed wrote: > I have attached the changes separately as changes.patch. Yes thanks. FWIW, I noticed those things as well when going through the code again this morning for my tests. Note as well that the declaration of doPageCompression at the top of xlog.c

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-11-27 Thread Michael Paquier
On Fri, Nov 28, 2014 at 3:48 PM, Michael Paquier wrote: > Configuration > == > 3) HEAD + fdw = on > start LSN: 0/1BC8 > stop LSN: > difference: Wrong copy/paste: stop LSN = 0/8FA8 difference = 1872MB tps = 2057.344827 (including connections establishing) tps = 2057.468800 (excludin

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-11-27 Thread Michael Paquier
So, I have been doing some more tests with this patch. I think the compression numbers are in line with the previous tests. Configuration == 3 sets are tested: - HEAD (a5eb85e) + fpw = on - patch + fpw = on - patch + fpw = compress With the following configuration: shared_buffers=512MB ch

[HACKERS] Allocation in critical section after node exits archive recovery

2014-11-27 Thread Michael Paquier
Hi all, While performing some archive recovery tests I found the following assertion failure once standby gets out of archive recovery (HEAD at a5eb85e): (lldb) * thread #1: tid = 0x, 0x7fff8faaf282 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP * frame #0: 0x00

[HACKERS] Testing DDL deparsing support

2014-11-27 Thread Ian Barwick
DDL deparsing is a feature that allows collection of DDL commands as they are executed in a server, in some flexible, complete, and fully-contained format that allows manipulation, storage, and transmission. This feature has several use cases; the two best known ones are DDL replication and DDL a

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-11-27 Thread Rahila Syed
>if (!fullPageWrites) >{ > WALInsertLockAcquireExclusive(); >Insert->fullPageWrites = fullPageWrites; >WALInsertLockRelease(); >} > As fullPageWrites is not a boolean isnt it better to change the if condition as fullPageWrites =

Re: [HACKERS] why is PG_AUTOCONF_FILENAME is pg_config_manual.h?

2014-11-27 Thread Amit Kapila
On Thu, Nov 27, 2014 at 8:28 PM, Peter Eisentraut wrote: > > Surely that's not a value that we expect users to be able to edit. Is > pg_config_manual.h just abused as a place that's included everywhere? > Changing PG_AUTOCONF_FILENAME even by developer would lead to loss of useful configuration

Re: [HACKERS] no test programs in contrib

2014-11-27 Thread Kouhei Kaigai
> -Original Message- > From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Josh Berkus > Sent: Friday, November 28, 2014 5:48 AM > To: Alvaro Herrera; Pg Hackers > Subject: Re: [HACKERS] no test programs in contrib > > On 11/24/2014 05:49 AM,

Re: [HACKERS] pg_regress and --dbname option / multiple databases

2014-11-27 Thread Ian Barwick
On 28/11/14 00:02, Andrew Dunstan wrote: On 11/27/2014 04:12 AM, Ian Barwick wrote: Hi pg_regress provides the command line option "--dbname", which is described in the help output thusly: --dbname=DBuse database DB (default "regression") It does however accept multiple comma separa

Re: [HACKERS] [pgsql-packagers] Palle Girgensohn's ICU patch

2014-11-27 Thread Tatsuo Ishii
> Another issue is that (AFAIK) ICU doesn't support any non-Unicode > encodings, which means that a build supporting *only* ICU collations is a > nonstarter IMO. So we really need a way to deal with both system and ICU > collations, and treating the latter as a separate subset of pg_collation > se

Re: [HACKERS] [pgsql-packagers] Palle Girgensohn's ICU patch

2014-11-27 Thread Peter Geoghegan
On Thu, Nov 27, 2014 at 7:03 AM, Tom Lane wrote: > +1 ... this seems like a nice end-run around the backwards compatibility > problem. > > Another issue is that (AFAIK) ICU doesn't support any non-Unicode > encodings, which means that a build supporting *only* ICU collations is a > nonstarter IMO.

Re: [HACKERS] no test programs in contrib

2014-11-27 Thread Alvaro Herrera
Tom Lane wrote: > I'm not too happy with that approach, because packagers are going to > tend to think they should package any files installed by install-world. > The entire point of this change, IMO, is that the test modules should > *not* get installed, certainly not by normal install targets.

Re: [HACKERS] no test programs in contrib

2014-11-27 Thread Tom Lane
Josh Berkus writes: > So test_decoding is fairly useful for users demonstrating that decoding > works, especially if they're also testing an external decoding module > and are unsure of where their replication problem is located, or what's > wrong with their HBA settings. For that reason it's imp

Re: [HACKERS] no test programs in contrib

2014-11-27 Thread Josh Berkus
On 11/24/2014 05:49 AM, Alvaro Herrera wrote: > test_parser (a toy text search parser, added in 2007) > dummy_seclabel (for SECURITY LABEL regression testing, added Sept 2010) > worker_spi (for bgworkers, added Dec 2012) > test_shm_mq (test program for shared memory queues, added Jan 2014) > test_d

[HACKERS] OverrideSearchPathMatchesCurrent performance improvement

2014-11-27 Thread Tom Lane
The other thing that emerged from http://www.postgresql.org/message-id/flat/CAOR=d=3j1U_q-zf8+jUx1hkx8ps+N8pm=EUTqyFdJ5ov=+f...@mail.gmail.com is that for workloads like this, OverrideSearchPathMatchesCurrent is a bottleneck: it's part of the code path needed to re-use a cached plan, and this examp

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-11-27 Thread Michael Paquier
On Thu, Nov 27, 2014 at 11:59 PM, Michael Paquier wrote: > On Thu, Nov 27, 2014 at 11:42 PM, Andres Freund > wrote: >> One thing Heikki brought up somewhere, which I thought to be a good >> point, was that it might be worthwile to forget about compressing FDWs >> themselves, and instead compress

Re: [HACKERS] no test programs in contrib

2014-11-27 Thread Tom Lane
Alvaro Herrera writes: > I have also changed things so that: > 1. test modules are not installed by "make install", not checked by > "make installcheck", not checked by "make check". > 2. test modules are checked by "make check-world" (this is consistent > with handling of contrib). > 3. test m

Re: [HACKERS] no test programs in contrib

2014-11-27 Thread Alvaro Herrera
Peter Eisentraut wrote: > On 11/26/14 9:27 AM, Alvaro Herrera wrote: > > I haven't done anything about documentation. I thought a new chapter > > after "Additional Supplied Modules", perhaps entitled "Additional Sample > > Modules" would be appropriate. > > I would remove the SGML files and put s

[HACKERS] Marginal performance improvement: replace bms_first_member loops

2014-11-27 Thread Tom Lane
Another thing that came out of the discussion at http://www.postgresql.org/message-id/flat/CAOR=d=3j1U_q-zf8+jUx1hkx8ps+N8pm=EUTqyFdJ5ov=+f...@mail.gmail.com was that there was a significant amount of palloc/pfree traffic blamable on the bms_first_member() loop in plpgsql's setup_param_list(). I'v

Re: [HACKERS] The problems of PQhost()

2014-11-27 Thread Noah Misch
On Fri, Nov 28, 2014 at 03:11:06AM +0900, Fujii Masao wrote: > On Thu, Nov 27, 2014 at 12:38 PM, Noah Misch wrote: > > Sure. I'll first issue "git revert 9f80f48", then apply the attached patch. > > Since libpq ignores a hostaddr parameter equal to the empty string, this > > implementation does l

[HACKERS] Re: [BUGS] BUG #12071: Stat collector went crasy (50MB/s constant writes)

2014-11-27 Thread Tomas Vondra
Moving to pgsql-hackers, as that's a more appropriate place for this discussion. On 27.11.2014 11:26, Maxim Boguk wrote: > > > FWIW, I got curious and checked why we decided not to implement this > while reworking the stats in 9.3, as keeping an is_dirty flag seems as a > rather stra

Re: [HACKERS] The problems of PQhost()

2014-11-27 Thread Fujii Masao
On Thu, Nov 27, 2014 at 12:38 PM, Noah Misch wrote: > On Tue, Nov 25, 2014 at 09:53:10PM +0900, Fujii Masao wrote: >> On Tue, Nov 25, 2014 at 12:37 PM, Noah Misch wrote: >> > On Wed, Jan 22, 2014 at 11:48:26PM +0900, Fujii Masao wrote: >> >> (3) PQhost() cannot return the hostaddr. >> > >> >> We

Re: [HACKERS] why is PG_AUTOCONF_FILENAME is pg_config_manual.h?

2014-11-27 Thread Fujii Masao
On Thu, Nov 27, 2014 at 11:58 PM, Peter Eisentraut wrote: > Surely that's not a value that we expect users to be able to edit. Is > pg_config_manual.h just abused as a place that's included everywhere? > > (I suggest utils/guc.h as a better place.) +1 Regards, -- Fujii Masao -- Sent via pg

Re: [HACKERS] psql \watch always ignores \pset null

2014-11-27 Thread Fujii Masao
On Wed, Nov 19, 2014 at 3:24 PM, Tom Lane wrote: > Will Leinweber writes: >> On Tue, Nov 18, 2014 at 9:54 PM, Tom Lane wrote: >>> Fujii Masao writes: Is there any reason why \watch must ignore \pset null setting? > >>> Hmmm ... the comment offers a reasonable argument for forcing pager = 0

Re: [HACKERS] pg_isready: Missing translation macros.

2014-11-27 Thread Fujii Masao
On Thu, Nov 27, 2014 at 9:32 PM, Mats Erik Andersson wrote: > Hello there, > > the text response of pg_isready is hard coded in English. > These short snippets really ought to be localized as well. Thanks for the patch! Committed. Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing l

Re: [HACKERS] pg_dump/pg_restore seem broken on hamerkop

2014-11-27 Thread Bruce Momjian
On Sun, Oct 26, 2014 at 08:29:19PM -0400, Tom Lane wrote: > I wrote: > > Hm. %z ought not be locale-dependent ... however, it has a bigger > > problem, which is that it's a C99-ism. It's not there in SUSv2, > > which is our normal baseline for what's portable. I think we need > > to get rid of t

Re: [HACKERS] [pgsql-packagers] Palle Girgensohn's ICU patch

2014-11-27 Thread Tom Lane
Greg Stark writes: > Hm. Actually the pg_collation catalog might give a handy way out for the > issue of being inconsistent with the system collation. We could support > both sets of collations and let the user select an ICU collation or system > collation at runtime. +1 ... this seems like a nic

Re: [HACKERS] pg_regress and --dbname option / multiple databases

2014-11-27 Thread Andrew Dunstan
On 11/27/2014 04:12 AM, Ian Barwick wrote: Hi pg_regress provides the command line option "--dbname", which is described in the help output thusly: --dbname=DBuse database DB (default "regression") It does however accept multiple comma separated names and will create a database for eac

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-11-27 Thread Michael Paquier
On Thu, Nov 27, 2014 at 11:42 PM, Andres Freund wrote: > On 2014-11-27 13:00:57 +0900, Michael Paquier wrote: >> This is backward-incompatible in the fact that forcibly-written FPWs >> would be compressed all the time, even if FPW is set to off. The >> documentation of the previous patches also me

[HACKERS] why is PG_AUTOCONF_FILENAME is pg_config_manual.h?

2014-11-27 Thread Peter Eisentraut
Surely that's not a value that we expect users to be able to edit. Is pg_config_manual.h just abused as a place that's included everywhere? (I suggest utils/guc.h as a better place.) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: htt

Re: [HACKERS] Fillfactor for GIN indexes

2014-11-27 Thread Cédric Villemain
Le jeudi 27 novembre 2014 23:33:11 Michael Paquier a écrit : > On Fri, Nov 21, 2014 at 2:12 PM, Michael Paquier > > wrote: > > I am adding that to the commit fest of December. > > Here are updated patches. Alvaro notified me about an inconsistent > comment. what are the benefits of this patch ?

Re: [HACKERS] Proposal : REINDEX SCHEMA

2014-11-27 Thread Michael Paquier
On Thu, Nov 27, 2014 at 8:18 PM, Sawada Masahiko wrote: > On Thu, Nov 27, 2014 at 1:07 PM, Michael Paquier > wrote: >> On Thu, Nov 27, 2014 at 12:55 AM, Sawada Masahiko >> wrote: >>> +1 to define new something object type and remove do_user and do_system. >>> But if we add OBJECT_SYSTEM to Obje

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-11-27 Thread Andres Freund
On 2014-11-27 13:00:57 +0900, Michael Paquier wrote: > On Wed, Nov 26, 2014 at 8:27 PM, Syed, Rahila wrote: > > Don't we need to initialize doPageCompression similar to doPageWrites in > > InitXLOGAccess? > Yep, you're right. I missed this code path. > > > Also , in the earlier patches compress

Re: [HACKERS] Fillfactor for GIN indexes

2014-11-27 Thread Michael Paquier
On Fri, Nov 21, 2014 at 2:12 PM, Michael Paquier wrote: > I am adding that to the commit fest of December. Here are updated patches. Alvaro notified me about an inconsistent comment. -- Michael From eda0730d991f8b4dfbacc4d7a953ec5bff8b2ffe Mon Sep 17 00:00:00 2001 From: Michael Paquier Date: Fri

Re: [HACKERS] [PATCH] HINT: pg_hba.conf changed since last config reload

2014-11-27 Thread Bruce Momjian
On Thu, Nov 6, 2014 at 05:46:42PM -0500, Peter Eisentraut wrote: > Finally, the fact that a configuration change is in progress is > privileged information. Unprivileged users can deduct from the presence > of this message that administrators are doing something, and possibly > that they have don

Re: [HACKERS] Add CREATE support to event triggers

2014-11-27 Thread Bruce Momjian
On Wed, Nov 26, 2014 at 09:01:13PM -0500, Stephen Frost wrote: > * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > > Bruce Momjian wrote: > > > How would replicating DDL handle cases where the master and slave > > > servers have different major versions and the DDL is only supported by > > > the

Re: [HACKERS] What exactly is our CRC algorithm?

2014-11-27 Thread Bruce Momjian
On Thu, Nov 27, 2014 at 11:19:51AM +0530, Abhijit Menon-Sen wrote: > At 2014-11-26 18:56:52 -0500, br...@momjian.us wrote: > > > > I would test it with fsync=off to remove the fsync delay. That will > > simulate an SSD or BBU controller. > > The earlier tests were run with fsync=off. > > I ran a

[HACKERS] Using pg_rewind for differential backup

2014-11-27 Thread Sameer Kumar
Can we tweak pg_rewind to take differential backups in PostgreSQL? I was wondering can we hack the pg_rewind code to print the details of the file which have been modified compared to a target server. The list output can then be used for taking differential backups. Or perhaps we can add an optio

[HACKERS] pg_isready: Missing translation macros.

2014-11-27 Thread Mats Erik Andersson
Hello there, the text response of pg_isready is hard coded in English. These short snippets really ought to be localized as well. Best regards, Mats Erik Andersson >From eca5f0d8b72b7ae5508af2850977d999a0f2bca4 Mon Sep 17 00:00:00 2001 From: Mats Erik Andersson Date: Thu, 27 Nov 2014 11:58:3

[HACKERS] initdb: Improve error recovery.

2014-11-27 Thread Mats Erik Andersson
Hello there, I would like to improve error recovery of initdb when the password file is empty. The present code declares "Error 0" as the cause of failure. It suffices to use ferrer() since fgets() returns NULL also at a premature EOF. In addition, a minor case is the need of a line feed in order

Re: [HACKERS] [PATCH] add ssl_protocols configuration option

2014-11-27 Thread Alex Shulgin
Dag-Erling Smørgrav writes: > Alex Shulgin writes: >> OK, looks like I've come up with something workable: I've added >> sslprotocol connection string keyword similar to pre-existing >> sslcompression, etc. Please see attached v2 of the original patch. >> I'm having doubts about the name of op

Re: [HACKERS] [v9.5] Custom Plan API

2014-11-27 Thread Simon Riggs
On 27 November 2014 at 10:33, Kouhei Kaigai wrote: > The reason why documentation portion was not yet committed is, sorry, it > is due to quality of documentation from the standpoint of native English > speaker. > Now, I'm writing up a documentation stuff according to the latest code base, > plea

Re: [HACKERS] Proposal : REINDEX SCHEMA

2014-11-27 Thread Sawada Masahiko
On Thu, Nov 27, 2014 at 1:07 PM, Michael Paquier wrote: > On Thu, Nov 27, 2014 at 12:55 AM, Sawada Masahiko > wrote: >> +1 to define new something object type and remove do_user and do_system. >> But if we add OBJECT_SYSTEM to ObjectType data type, >> system catalogs are OBJECT_SYSTEM as well as

Re: [HACKERS] [PATCH] add ssl_protocols configuration option

2014-11-27 Thread Dag-Erling Smørgrav
Alex Shulgin writes: > OK, looks like I've come up with something workable: I've added > sslprotocol connection string keyword similar to pre-existing > sslcompression, etc. Please see attached v2 of the original patch. > I'm having doubts about the name of openssl.h header though, > libpq-openss

Re: [HACKERS] [v9.5] Custom Plan API

2014-11-27 Thread Kouhei Kaigai
> On 7 November 2014 at 22:46, Robert Haas wrote: > > On Mon, Oct 27, 2014 at 2:35 AM, Kouhei Kaigai > wrote: > >>> FYI, patch v12 part 2 no longer applies cleanly. > >>> > >> Thanks. I rebased the patch set according to the latest master branch. > >> The attached v13 can be applied to the master

Re: [HACKERS] inherit support for foreign tables

2014-11-27 Thread Etsuro Fujita
(2014/11/17 17:55), Ashutosh Bapat wrote: Here are my review comments for patch fdw-inh-3.patch. Thanks for the review! Tests --- 1. It seems like you have copied from testcase inherit.sql to postgres_fdw testcase. That's a good thing, but it makes the test quite long. May be we should ha

Re: [HACKERS] [pgsql-packagers] Palle Girgensohn's ICU patch

2014-11-27 Thread Greg Stark
On 27 Nov 2014 09:09, "Jakob Egger" wrote: > > ICU offers a lot more collations than the OS. For example, besides "de_CH" it also offers "de_CH@collation=phonebook". Adding support for these is a bit more involved. > > * initdb would need to be extended to also look for collations offered by ICU a

Re: [HACKERS] [v9.5] Custom Plan API

2014-11-27 Thread Simon Riggs
On 7 November 2014 at 22:46, Robert Haas wrote: > On Mon, Oct 27, 2014 at 2:35 AM, Kouhei Kaigai wrote: >>> FYI, patch v12 part 2 no longer applies cleanly. >>> >> Thanks. I rebased the patch set according to the latest master branch. >> The attached v13 can be applied to the master. > > I've com

Re: [HACKERS] [pgsql-packagers] Palle Girgensohn's ICU patch

2014-11-27 Thread Dave Page
On Thu, Nov 27, 2014 at 9:09 AM, Jakob Egger wrote: > Am 26.11.2014 um 17:46 schrieb Geoff Montee : > > This topic reminds me of a thread from a couple months ago: > > > > > http://www.postgresql.org/message-id/f8268db6-b50f-429f-8289-da8ffa5f2...@tripadvisor.com > > > > It sounds like adding ICU

[HACKERS] pg_regress and --dbname option / multiple databases

2014-11-27 Thread Ian Barwick
Hi pg_regress provides the command line option "--dbname", which is described in the help output thusly: --dbname=DBuse database DB (default "regression") It does however accept multiple comma separated names and will create a database for each name provided, but AFAICS only ever uses the

Re: [HACKERS] [pgsql-packagers] Palle Girgensohn's ICU patch

2014-11-27 Thread Jakob Egger
Am 26.11.2014 um 17:46 schrieb Geoff Montee : > This topic reminds me of a thread from a couple months ago: > > http://www.postgresql.org/message-id/f8268db6-b50f-429f-8289-da8ffa5f2...@tripadvisor.com > > It sounds like adding ICU support to core may also allow for adding > collation versioning