Re: [HACKERS] WAL consistency check facility

2016-11-12 Thread Michael Paquier
On Sun, Nov 13, 2016 at 12:06 PM, Peter Eisentraut wrote: > Could we name this something like wal_consistency_checking? > > Otherwise it sounds like you use this to select the amount of > consistency in the WAL (similar to, say, wal_level). Or wal_check? Or wal_consistency_check? -- Michael --

Re: [HACKERS] Logical Replication WIP

2016-11-12 Thread Peter Eisentraut
On 11/12/16 2:18 PM, Andres Freund wrote: >>> I also wonder if we want an easier to >>> > > extend form of pubinsert/update/delete (say to add pubddl, pubtruncate, >>> > > pub ... without changing the schema). >>> > > >> > >> > So like, text array that's then parsed everywhere (I am not doing >>

Re: [HACKERS] Do we need use more meaningful variables to replace 0 in catalog head files?

2016-11-12 Thread Peter Eisentraut
On 11/11/16 11:10 AM, Tom Lane wrote: > boolin: OID=1242 proname=boolin proargtypes="cstring" prorettype=bool > boolin: prosrc=boolin provolatile=i proparallel=s Then we're not very far away from just using CREATE FUNCTION SQL commands. -- Peter Eisentraut http://www.2ndQuadrant.com

Re: [HACKERS] Proposal: scan key push down to heap [WIP]

2016-11-12 Thread Dilip Kumar
I have done performance analysis for TPCH queries, I saw visible gain in 5 queries (10-25%). Performance Data: Benchmark : TPCH (S.F. 10) shared_buffer : 20GB work_mem : 50MB Machine : POWER Results are median of three run (explain analyze results for both head/patch are attached in T

Re: [HACKERS] Logical Replication WIP

2016-11-12 Thread Petr Jelinek
On 12/11/16 20:19, Andres Freund wrote: > On 2016-11-10 23:31:27 +0100, Petr Jelinek wrote: >> On 04/11/16 13:15, Andres Freund wrote: >>> >>> /* Prototypes for private functions */ >>> -static bool libpq_select(int timeout_ms); >>> +static bool libpq_select(PGconn *streamConn, >>> +

Re: [HACKERS] WAL consistency check facility

2016-11-12 Thread Peter Eisentraut
On 11/9/16 11:55 PM, Michael Paquier wrote: > + > + wal_consistency (string) > + > + wal_consistency configuration parameter > + > + > + > + > +This parameter is used to check the consistency of WAL records, i.e, > +whether the WAL reco

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-12 Thread Peter Eisentraut
On 11/8/16 3:48 PM, Robert Haas wrote: > First question: Do we even want this? Generally, when a program > writes to a file, we rely on the operating system to decide when that > data should be written back to disk. We have to override that > distinction for things internal to PostgreSQL because

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-12 Thread Michael Paquier
On Sun, Nov 13, 2016 at 12:17 AM, Dilip Kumar wrote: > On Sat, Nov 12, 2016 at 10:16 AM, Michael Paquier > wrote: >>> - For pg_dumpall, a short option "-N" is added for "--no-sync", but not for >>> pg_dump (because -N is already taken there). >>> I'd opt for either using the same short option

Re: [HACKERS] More snapshot-management fun

2016-11-12 Thread Tom Lane
I wrote: > * ProcArrayInstallImportedXmin and ProcArrayInstallRestoredXmin will > forcibly overwrite MyPgXact->xmin with some random xmin or other. > While they take pains to ensure that doesn't cause the global xmin > horizon to go backwards, and while we're on the subject, I doubt that those pai

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-11-12 Thread Andres Freund
Hi, Subject: [PATCH 1/2] slab allocator diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 6ad7e7d..520f295 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c I'd ra

Re: [HACKERS] Logical Replication WIP

2016-11-12 Thread Andres Freund
Hi, On 2016-11-11 12:04:27 +0100, Petr Jelinek wrote: > On 04/11/16 14:00, Andres Freund wrote: > > Hi, > > > > + > > + pg_publication_rel > > + > > + > > + pg_publication_rel > > + > > + > > + > > + The pg_publication_rel catalog contains > > + mapping between tables and publicatio

Re: [HACKERS] Logical Replication WIP

2016-11-12 Thread Andres Freund
On 2016-11-10 23:31:27 +0100, Petr Jelinek wrote: > On 04/11/16 13:15, Andres Freund wrote: > > > > /* Prototypes for private functions */ > > -static bool libpq_select(int timeout_ms); > > +static bool libpq_select(PGconn *streamConn, > > +int timeout_

Re: [HACKERS] proposal: psql \setfileref

2016-11-12 Thread Pavel Stehule
2016-11-10 18:56 GMT+01:00 Tom Lane : > Pavel Stehule writes: > > 2016-11-09 22:47 GMT+01:00 Tom Lane : > >> * I really dislike the notion of keying the behavior to a special type > of > >> psql variable. > > > still I am thinking so some differencing can be nice, because we can use > > psql file

Re: [HACKERS] Remove the comment on the countereffectiveness of large shared_buffers on Windows

2016-11-12 Thread Jeff Janes
On Mon, Nov 7, 2016 at 7:03 PM, Amit Kapila wrote: > On Tue, Nov 8, 2016 at 5:12 AM, Jeff Janes wrote: > > On Mon, Nov 7, 2016 at 5:55 AM, Amit Kapila > wrote: > >> > >> > >> Isn't it somewhat strange that writes are showing big win whereas > >> reads doesn't show much win? > > > > > > I don't

Re: [HACKERS] xlogreader.c fails with FATAL on a cluster with 4kB block size

2016-11-12 Thread Tomas Vondra
Meh, ignore this report - I've just realized I've been running the pg_xlogdump binary built for 8kB pages, so the failures are kinda expected. Sorry about the confusion. regards On 11/12/2016 07:52 PM, Tomas Vondra wrote: Hi, I'm running some tests on a cluster with 4kB blocks, and it seems

[HACKERS] More snapshot-management fun

2016-11-12 Thread Tom Lane
While pursuing the unprotected-use-of-CatalogSnapshot problem I noted earlier, I came across a couple of related issues: * ProcArrayInstallImportedXmin and ProcArrayInstallRestoredXmin will forcibly overwrite MyPgXact->xmin with some random xmin or other. While they take pains to ensure that doesn

Re: [HACKERS] Patch to implement pg_current_logfile() function

2016-11-12 Thread Karl O. Pinc
On Mon, 7 Nov 2016 23:29:28 +0100 Gilles Darold wrote: > Here is the v13 of the patch, > - I've reverted the patch removing the call to logfile_writename in > open_csvlogfile function, it is required to write log filename at > startup when log_destination is set to csvlog. I could not find a

[HACKERS] xlogreader.c fails with FATAL on a cluster with 4kB block size

2016-11-12 Thread Tomas Vondra
Hi, I'm running some tests on a cluster with 4kB blocks, and it seems there's a bug in xlogreader.c, causing FATAL errors for example when running pg_xlogdump: pg_xlogdump: FATAL: error in WAL record at 48/63970258: BKPIMAGE_HAS_HOLE not set, but hole offset 0 length 4096 at 48/63970258 T

Re: [HACKERS] Remove the comment on the countereffectiveness of large shared_buffers on Windows

2016-11-12 Thread Magnus Hagander
On Sat, Nov 12, 2016 at 4:03 AM, Amit Kapila wrote: > On Fri, Nov 11, 2016 at 3:01 PM, Magnus Hagander > wrote: > >> > Based on this optimization we might want to keep the text that says > >> > large > >> > shared buffers on Windows aren't as effective perhaps, > > Sounds sensible or may add a l

Re: [HACKERS] Indirect indexes

2016-11-12 Thread Andres Freund
Hi, On 2016-11-01 01:43:31 -0300, Alvaro Herrera wrote: > Alvaro Herrera wrote: > > I propose we introduce the concept of "indirect indexes". > > This is a WIP non-functional patch for indirect indexes. I've been > distracted from working on it for some time already and will be off-line > for ha

Re: [HACKERS] Re: [COMMITTERS] pgsql: Change the way that LWLocks for extensions are allocated.

2016-11-12 Thread Tom Lane
Andres Freund writes: > On 2016-11-12 11:42:12 -0500, Tom Lane wrote: >> Hm. What about inventing an additional macro NO_INSTALLCHECK that >> prevents pgxs.mk from generating an installcheck rule? > That'd work. I'd also be ok with living with the warning. I have to say > I find it hard to be c

Re: [HACKERS] Re: [COMMITTERS] pgsql: Change the way that LWLocks for extensions are allocated.

2016-11-12 Thread Andres Freund
On 2016-11-12 11:42:12 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2016-11-12 11:30:42 -0500, Tom Lane wrote: > >> which is a rather blatant waste of cycles. I would suggest an explicit > >> do-nothing installcheck rule rather than the hack you came up with here. > > > I had that at fir

Re: [HACKERS] Re: [COMMITTERS] pgsql: Change the way that LWLocks for extensions are allocated.

2016-11-12 Thread Tom Lane
Andres Freund writes: > On 2016-11-12 11:30:42 -0500, Tom Lane wrote: >> which is a rather blatant waste of cycles. I would suggest an explicit >> do-nothing installcheck rule rather than the hack you came up with here. > I had that at first, but that generates a warning about overwriting the > m

Re: [HACKERS] Re: [COMMITTERS] pgsql: Change the way that LWLocks for extensions are allocated.

2016-11-12 Thread Andres Freund
On 2016-11-12 11:30:42 -0500, Tom Lane wrote: > Andres Freund writes: > > Committed after simplifying the Makefile. > > While I have no particular objection to adding these tests, the > commit log's claim that this will improve buildfarm testing is > quite wrong. The buildfarm runs "make install

Re: [HACKERS] Re: [COMMITTERS] pgsql: Change the way that LWLocks for extensions are allocated.

2016-11-12 Thread Tom Lane
Andres Freund writes: > Committed after simplifying the Makefile. While I have no particular objection to adding these tests, the commit log's claim that this will improve buildfarm testing is quite wrong. The buildfarm runs "make installcheck" not "make check" in contrib. What I'm actually see

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-11-12 Thread Andres Freund
Hi, On 2016-11-01 05:14:58 +0530, Abhijit Menon-Sen wrote: > Subject: Convert recovery.conf settings to GUCs I really want to get this in, we've been back and forth about this for far too long. > >- Create a recovery command file recovery.conf in the cluster >- data directory (see )

Re: [HACKERS] move collation import to backend

2016-11-12 Thread Andres Freund
Hi, On 2016-10-27 21:56:53 -0400, Peter Eisentraut wrote: > Currently, initdb parses locale -a output to populate pg_collation. If > additional collations are installed in the operating system, it is not > possible to repeat this process, only by doing each step manually. So I > propose to move

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-11-12 Thread Andres Freund
On 2016-08-30 07:38:10 -0400, Tom Lane wrote: > Heikki Linnakangas writes: > > While profiling some queries and looking at executor overhead, I > > realized that we're not making much use of TupleTableSlot's ability to > > hold a buffer pin. In a SeqScan, the buffer is held pinned by the > > under

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-12 Thread Dilip Kumar
On Sat, Nov 12, 2016 at 10:16 AM, Michael Paquier wrote: >> - For pg_dumpall, a short option "-N" is added for "--no-sync", but not for >> pg_dump (because -N is already taken there). >> I'd opt for either using the same short option for both or (IMO better) >> only offering a long option fo

Re: [HACKERS] [PATCH] Logical decoding timeline following take II

2016-11-12 Thread Andres Freund
On 2016-10-24 17:49:13 +0200, Petr Jelinek wrote: > + * Determine which timeline to read an xlog page from and set the > + * XLogReaderState's state->currTLI to that timeline ID. "XLogReaderState's state->currTLI" - the state's a bit redundant. > + * The caller must also make sure it doesn't read

Re: [HACKERS] Patch to implement pg_current_logfile() function

2016-11-12 Thread Gilles Darold
Le 09/11/2016 à 02:51, Robert Haas a écrit : > On Thu, Nov 3, 2016 at 7:34 PM, Karl O. Pinc wrote: >> To me, the CURRENT_LOG_FILENAME symbol should contain the name >> of the current log file. It does not. The CURRENT_LOG_FILENAME symbol >> holds the name of the file which itself contains the na

Re: [HACKERS] Re: [COMMITTERS] pgsql: Change the way that LWLocks for extensions are allocated.

2016-11-12 Thread Andres Freund
Hi, On 2016-11-08 10:25:11 +0530, Amit Kapila wrote: > ifdef USE_PGXS > PG_CONFIG = pg_config > PGXS := $(shell $(PG_CONFIG) --pgxs) > @@ -21,3 +25,29 @@ top_builddir = ../.. > include $(top_builddir)/src/Makefile.global > include $(top_srcdir)/contrib/contrib-global.mk > endif > + > +# Disa

Re: [HACKERS] [bug fix] Cascading standby cannot catch up and get stuck emitting the same message repeatedly

2016-11-12 Thread Amit Kapila
On Thu, Nov 10, 2016 at 10:43 AM, Tsunakawa, Takayuki wrote: > From: pgsql-hackers-ow...@postgresql.org >> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas >> OK. I agree that's a problem. However, your patch adds zero new comment >> text while removing some existing comments

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-11-12 Thread Andres Freund
Hi, On 2016-11-11 16:42:43 +0900, Michael Paquier wrote: > + * This takes also > + * advantage to avoid 8-byte torn reads on some platforms by using the > + * fact that each insert lock is located on the same cache line. Something residing on the same cache line doens't provide that guarantee on

Re: [HACKERS] pg_dump, pg_dumpall and data durability

2016-11-12 Thread Michael Paquier
On Sat, Nov 12, 2016 at 1:46 PM, Michael Paquier wrote: > On Fri, Nov 11, 2016 at 11:03 PM, Albe Laurenz > wrote: >> This would avoid confusion, and we expect that few people will want to use >> this option anyway, right? > > Definitely a good point. Meh. I forgot docs and --help output upd

Re: [HACKERS] Shared memory estimation for postgres

2016-11-12 Thread Andres Freund
On 2016-11-11 09:56:21 +0900, Michael Paquier wrote: > On Fri, Nov 11, 2016 at 8:40 AM, leoaaryan wrote: > > The easiest way to find the value for the shared memory computation is to > > change the logging level to DEBUG3 and start postgres DB engine and it will > > give the calculated value in th