Re: [HACKERS] Patch to improve reliability of postgresql on linux nfs

2011-09-09 Thread Thom Brown
On 9 September 2011 01:04, George Barnett wrote: > After looking through the code I found that when postgres calls write() it > doesn't retry.  In order to address the issue with the PANIC in the WAL > writer I set the sync method to o_sync which solved the issue in that part of > the code, how

Re: [HACKERS] Patch to improve reliability of postgresql on linux nfs

2011-09-09 Thread Peter Eisentraut
On fre, 2011-09-09 at 10:04 +1000, George Barnett wrote: > After looking through the code I found that when postgres calls > write() it doesn't retry. In order to address the issue with the > PANIC in the WAL writer I set the sync method to o_sync which solved > the issue in that part of the code,

Re: [HACKERS] [PATCH] Log crashed backend's query (activity string)

2011-09-09 Thread Marti Raudsepp
On Thu, Sep 8, 2011 at 03:22, Robert Haas wrote: > On Wed, Sep 7, 2011 at 5:24 PM, Alvaro Herrera > wrote: >> I remember we had bugs whereby an encoding conversion would fail, >> leading to elog trying to report this problem, but this attempt would >> also incur a conversion step, failing recursi

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-09 Thread Magnus Hagander
On Fri, Sep 9, 2011 at 11:56, Fujii Masao wrote: > Hi, > > http://archives.postgresql.org/pgsql-hackers/2010-12/msg02343.php > > In previous discussion, we've reached the consensus that we should unite > recovery.conf and postgresql.conf. The attached patch does that. The > patch is WIP, I'll have

Re: [HACKERS] WAL "low watermark" during base backup

2011-09-09 Thread Magnus Hagander
On Tue, Sep 6, 2011 at 22:35, Simon Riggs wrote: > On Mon, Sep 5, 2011 at 11:38 AM, Magnus Hagander wrote: >> On Sun, Sep 4, 2011 at 19:02, Simon Riggs wrote: >>> On Fri, Sep 2, 2011 at 6:52 PM, Magnus Hagander wrote: >>> Attached patch implements a "low watermark wal location" in the

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-09 Thread Fujii Masao
On Fri, Sep 9, 2011 at 7:21 PM, Magnus Hagander wrote: > On Fri, Sep 9, 2011 at 11:56, Fujii Masao wrote: >> Hi, >> >> http://archives.postgresql.org/pgsql-hackers/2010-12/msg02343.php >> >> In previous discussion, we've reached the consensus that we should unite >> recovery.conf and postgresql.c

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-09 Thread Magnus Hagander
On Fri, Sep 9, 2011 at 13:15, Fujii Masao wrote: > On Fri, Sep 9, 2011 at 7:21 PM, Magnus Hagander wrote: >> On Fri, Sep 9, 2011 at 11:56, Fujii Masao wrote: >>> Hi, >>> >>> http://archives.postgresql.org/pgsql-hackers/2010-12/msg02343.php >>> >>> In previous discussion, we've reached the consen

Re: [HACKERS] WAL "low watermark" during base backup

2011-09-09 Thread Dimitri Fontaine
Magnus Hagander writes: >> If you must have this then make pg_basebackup copy xlog files >> regularly during the backup. That way your backup can take forever and >> your primary disk won't fill up. In many cases it actually will take >> forever, but at least we don't take down the primary. > > Th

Re: [HACKERS] WAL "low watermark" during base backup

2011-09-09 Thread Magnus Hagander
On Fri, Sep 9, 2011 at 13:40, Dimitri Fontaine wrote: > Magnus Hagander writes: >>> If you must have this then make pg_basebackup copy xlog files >>> regularly during the backup. That way your backup can take forever and >>> your primary disk won't fill up. In many cases it actually will take >>>

Re: [HACKERS] regress test failed

2011-09-09 Thread Peter Eisentraut
On sön, 2011-09-04 at 12:06 -0400, Andrew Dunstan wrote: > Maybe we need a few members that test a large number of locales. > (Anyone feel like donating resources? I'm currently providing > resources for seven, which I think is sufficient :-) ) If we're just testing different configuration combin

Re: [HACKERS] WAL "low watermark" during base backup

2011-09-09 Thread Florian Pflug
On Sep9, 2011, at 13:48 , Magnus Hagander wrote: > On Fri, Sep 9, 2011 at 13:40, Dimitri Fontaine wrote: >> Magnus Hagander writes: If you must have this then make pg_basebackup copy xlog files regularly during the backup. That way your backup can take forever and your primary disk

Re: [HACKERS] WAL "low watermark" during base backup

2011-09-09 Thread Dimitri Fontaine
Florian Pflug writes: > Couldn't we send all available WAL after each single data-file instead > of waiting for all data files to be transferred before sending WAL? +1 (or maybe not at the file boundary but rather driven by archive command with some internal hooking, as the backend needs some new

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-09 Thread Tom Lane
Magnus Hagander writes: > I have to wonder though, if it wouldn't be less confusing to just get > rid of recovery.conf and use a *different* file for this. Just to make > it clear it's not a config file, but just a boolean exists/notexists > state. +1. If it's not a configuration file anymore, i

Re: [HACKERS] WAL "low watermark" during base backup

2011-09-09 Thread Tom Lane
Magnus Hagander writes: > On Fri, Sep 9, 2011 at 13:40, Dimitri Fontaine wrote: >> I'm not getting why we need the later one when we have this older one? > One of them is for the simple case. It requires a single connection to > the server, and it supports things like writing to tarfiles and > c

Re: [HACKERS] Patch to improve reliability of postgresql on linux nfs

2011-09-09 Thread Tom Lane
George Barnett writes: > [ patch to retry writes on NFS ] I'm having a hard time getting excited about this idea, because IMO NFS is insufficiently reliable to run a database on, and no patch like this can fix that. However, some concrete points: 1. If writes need to be retried, why not reads?

Re: [HACKERS] REVIEW proposal: a validator for configuration files

2011-09-09 Thread Alexey Klyukin
Hello, On Sep 7, 2011, at 5:00 PM, Tom Lane wrote: > Andy Colson writes: >> Where did the other warnings go? Its right though, line 570 is bad. It >> also seems to have killed the server. I have not gotten through the history >> of messages regarding this patch, but is it supposed to kill t

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-09 Thread Simon Riggs
On Fri, Sep 9, 2011 at 3:05 PM, Tom Lane wrote: > Magnus Hagander writes: >> I have to wonder though, if it wouldn't be less confusing to just get >> rid of recovery.conf and use a *different* file for this. Just to make >> it clear it's not a config file, but just a boolean exists/notexists >> s

Re: [HACKERS] Patch to improve reliability of postgresql on linux nfs

2011-09-09 Thread Bernd Helmle
--On 9. September 2011 10:27:22 -0400 Tom Lane wrote: On the whole I think you'd be better off lobbying your NFS implementors to provide something closer to the behavior of every other filesystem on the planet. Or checking to see if you need to adjust your NFS configuration, as the other res

Re: [HACKERS] pg_last_xact_insert_timestamp

2011-09-09 Thread Robert Haas
On Thu, Sep 8, 2011 at 8:42 PM, Fujii Masao wrote: > Another idea to avoid spinlock contention is save the timestamp in > PgBackendStatus (which contains information for pg_stat_activity). > This enables us to write and read the timestamp without spinlock. > Comments? That seems like a possibly p

Re: [HACKERS] force_not_null option support for file_fdw

2011-09-09 Thread Kohei Kaigai
I marked this patch as "Ready for Committer", and hope this patch being committed. Thanks, -- NEC Europe Ltd, SAP Global Competence Center KaiGai Kohei > -Original Message- > From: Shigeru Hanada [mailto:shigeru.han...@gmail.com] > Sent: 9. September 2011 06:03 > To: Kohei Kaigai > Cc:

Re: [HACKERS] Moving core timestamp typedefs/macros somewhere else

2011-09-09 Thread Tom Lane
I wrote: > I propose moving the Timestamp/Interval typedefs, as well as some basic > associated macros such as the ones mentioned above (basically, lines > 25-100 of utils/timestamp.h, plus the DT_NOBEGIN/DT_NOEND stuff, plus > the Julian-date macros in datetime.h), into a separate header file that

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-09 Thread Josh Berkus
On 9/9/11 7:05 AM, Tom Lane wrote: > Magnus Hagander writes: >> I have to wonder though, if it wouldn't be less confusing to just get >> rid of recovery.conf and use a *different* file for this. Just to make >> it clear it's not a config file, but just a boolean exists/notexists >> state. > > +1.

Re: [HACKERS] Patch to improve reliability of postgresql on linux nfs

2011-09-09 Thread Noah Misch
On Fri, Sep 09, 2011 at 10:27:22AM -0400, Tom Lane wrote: > George Barnett writes: > > [ patch to retry writes on NFS ] > > I'm having a hard time getting excited about this idea, because IMO > NFS is insufficiently reliable to run a database on, and no patch like > this can fix that. However, s

[HACKERS] fsyncing data to disk

2011-09-09 Thread Nulik Nol
Hi, this is not exactly a Postgresql question, but an input from hackers list like this would be invaluable for me. I am coding my own database engine, and I decided to do not implement transaction engine because it implies too much code. But to achieve the Durability of ACID I need a 100% reliable

Re: [HACKERS] Patch to improve reliability of postgresql on linux nfs

2011-09-09 Thread Kevin Grittner
Noah Misch wrote: > We shouldn't complain when a kernel provides a conforming write(), > even if it appears that the kernel achieved little by using some > freedom afforded it. I remember we had some compiler warnings in the logging area because we were making the assumption that no implementa

Re: [HACKERS] fsyncing data to disk

2011-09-09 Thread Florian Pflug
On Sep9, 2011, at 20:15 , Nulik Nol wrote: > this is not exactly a Postgresql question, but an input from hackers > list like this would be invaluable for me. > I am coding my own database engine, and I decided to do not implement > transaction engine because it implies too much code. > But to achi

Re: [HACKERS] postgresql.conf archive_command example

2011-09-09 Thread Florian Pflug
On Sep8, 2011, at 15:09 , Aidan Van Dyk wrote: > Personally, I think both of these show examples of why PG should be > looking hard at either providing a simple robust local directory based > archive_command, or very seriously pointing users at properly written > tools like omniptr, or ptrtools, wa

Re: [HACKERS] postgresql.conf archive_command example

2011-09-09 Thread Martijn van Oosterhout
On Fri, Sep 09, 2011 at 08:59:43PM +0200, Florian Pflug wrote: > Archiving WAL should be done by copying to a temp file and moving it > into place. Before returning success, one should probably also do the > fsync incantations the linux kernel guys argued are necessary to prevent > the file from ap

Re: [HACKERS] memory-related bugs

2011-09-09 Thread Daniel Farina
On Thu, Sep 8, 2011 at 1:56 PM, Tom Lane wrote: > Daniel Farina writes: >> On Tue, Sep 6, 2011 at 12:00 PM, Tom Lane wrote: >>> I'm still of the opinion that there's no real need to avoid memcpy with >>> identical source and destination, so I didn't apply this first patch. > >> I am leery of mem

Re: [HACKERS] augmenting MultiXacts to improve foreign keys

2011-09-09 Thread Alvaro Herrera
Excerpts from Alvaro Herrera's message of mar ago 09 13:01:04 -0400 2011: > To implement this, we need to augment MultiXact to store the lock type > that each comprising Xid holds on the tuple. Two bits per Xid are > needed. My thinking is that we could simply extend the "members" to add > a by

Re: [HACKERS] Should I implement DROP INDEX CONCURRENTLY?

2011-09-09 Thread Daniel Farina
On Wed, Aug 24, 2011 at 1:04 PM, Daniel Farina wrote: > On Wed, Aug 24, 2011 at 12:38 PM, Tom Lane wrote: >> Assuming the issue really is the physical unlinks (which I agree I'd >> like to see some evidence for), I wonder whether the problem could be >> addressed by moving smgrDoPendingDeletes()

Re: [HACKERS] Protecting against multiple instances per cluster

2011-09-09 Thread Greg Stark
On Thu, Sep 8, 2011 at 10:03 PM, Magnus Hagander wrote: >> Would there be a way to prevent this abhorrent scenario from coming >> into existence? > There are plenty of clustering products out there that are really > designed for one thing pimarily, and that's dealing with this kind of > fencing.

Re: [HACKERS] Large C files

2011-09-09 Thread Greg Stark
On Fri, Sep 9, 2011 at 6:57 AM, Heikki Linnakangas wrote: >> In particular, I'd like to know what >> boundaries it is envisaged that the code should be refactored along to >> increase its conceptual integrity, or to better separate concerns. I >> assume that that's the idea, since each new .c file

Re: [HACKERS] Patch to improve reliability of postgresql on linux nfs

2011-09-09 Thread Bruce Momjian
Tom Lane wrote: > George Barnett writes: > > [ patch to retry writes on NFS ] > > I'm having a hard time getting excited about this idea, because IMO > NFS is insufficiently reliable to run a database on, and no patch like > this can fix that. However, some concrete points: > > 1. If writes nee

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-09 Thread Greg Stark
On Fri, Sep 9, 2011 at 6:40 PM, Josh Berkus wrote: > I'm in favor of this.  People are sufficiently confused by the existing > behavior that we're not going to confuse them further by changing it. > Fwiw as someone who *was* confused previously, it now makes perfect sense to me. "We have postgres

Re: [HACKERS] fsyncing data to disk

2011-09-09 Thread Greg Stark
On Fri, Sep 9, 2011 at 7:46 PM, Florian Pflug wrote: >> I am going to use the whole partition device for the DB (like /dev/sda1) >> , so no filesystem code will be used. Also I am using asynchronous IO >> (the aio_read and aio_write) and I don't know if they can be combined >> with the fdatasync()

[HACKERS] nonempty default log_line_prefix (was [COMMITTERS] pgsql: Simplify handling of the timezone GUC)

2011-09-09 Thread Alvaro Herrera
Excerpts from Tom Lane's message of vie sep 09 18:59:45 -0300 2011: > Simplify handling of the timezone GUC by making initdb choose the default. > > We were doing some amazingly complicated things in order to avoid running > the very expensive identify_system_timezone() procedure during GUC > ini

Re: [HACKERS] nonempty default log_line_prefix (was [COMMITTERS] pgsql: Simplify handling of the timezone GUC)

2011-09-09 Thread Tom Lane
Alvaro Herrera writes: > Excerpts from Tom Lane's message of vie sep 09 18:59:45 -0300 2011: >> Simplify handling of the timezone GUC by making initdb choose the default. > Hmm, I was hoping that this might open the door for setting a nonempty > default log_line_prefix, but apparently not :-( Se

Re: [HACKERS] Large C files

2011-09-09 Thread Peter Geoghegan
I've produced the refinement of my little shell script anticipated by my last e-mail (using sed to remove irrelevant variations in __func__.12345 type symbol names). I decided to test it for: 1. Detecting behavioural changes when removing existing "pgrminclude ignore" files (Basically headers that

Re: [HACKERS] superusers are members of all roles?

2011-09-09 Thread Bruce Momjian
Robert Haas wrote: > On Sat, May 7, 2011 at 11:42 PM, Bruce Momjian wrote: > > Is this a TODO? > > I think so. Added to TODO: Address problem where superusers are assumed to be members of all groups http://archives.postgresql.org/pgsql-hackers/2011-04/msg00337.php

[HACKERS] new createuser option for replication role

2011-09-09 Thread Fujii Masao
Hi, Currently createuser cannot create a role with REPLICATION privilege because it doesn't have any option to do that. Which sometimes annoys me when setting up replication. I'd like to propose to add new options "-x (--replication)" and "-X (--no-replication)" into createuser. "-x" allows the ne

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-09 Thread Magnus Hagander
On Sat, Sep 10, 2011 at 01:07, Greg Stark wrote: > On Fri, Sep 9, 2011 at 6:40 PM, Josh Berkus wrote: >> I'm in favor of this.  People are sufficiently confused by the existing >> behavior that we're not going to confuse them further by changing it. >> > > Fwiw as someone who *was* confused previ

Re: [HACKERS] new createuser option for replication role

2011-09-09 Thread Fujii Masao
On Sat, Sep 10, 2011 at 2:08 PM, Fujii Masao wrote: > Currently createuser cannot create a role with REPLICATION privilege > because it doesn't have any option to do that. Which sometimes annoys > me when setting up replication. I'd like to propose to add new options > "-x (--replication)" and "-X