Re: [HACKERS] Large C files

2011-09-08 Thread Heikki Linnakangas
On 08.09.2011 23:45, Peter Geoghegan wrote: On 8 September 2011 15:43, Robert Haas wrote: I wouldn't be too enthusiastic about starting a project like this in January, but now seems fine. A bigger problem is that I don't hear anyone volunteering to do the work. You seem to have a fairly stro

Re: [HACKERS] pg_dump.c

2011-09-08 Thread Pavel Golub
Hello, Andrew. You wrote: AD> In the "refactoring Large C files" discussion one of the biggest files AD> Bruce mentioned is pg_dump.c. There has been discussion in the past of AD> turning lots of the knowledge currently embedded in this file into a AD> library, which would make it available to

Re: [HACKERS] force_not_null option support for file_fdw

2011-09-08 Thread Shigeru Hanada
Thanks for the review, Kaigai-san. (2011/09/09 0:47), Kohei Kaigai wrote: > I found one other point to be fixed: > On get_force_not_null(), it makes a list of attribute names with > force_not_null option. > > + foreach (cell, options) > + { > + DefElem*def = (DefElem *)

Re: [HACKERS] pg_last_xact_insert_timestamp

2011-09-08 Thread Fujii Masao
On Thu, Sep 8, 2011 at 10:03 PM, Robert Haas wrote: > On Thu, Sep 8, 2011 at 6:14 AM, Fujii Masao wrote: >> OTOH, new function enables users to monitor the delay as a timestamp. >> For users, a timestamp is obviously easier to handle than LSN, and the delay >> as a timestamp is more intuitive. So

Re: [HACKERS] Large C files

2011-09-08 Thread Robert Haas
On Thu, Sep 8, 2011 at 4:45 PM, Peter Geoghegan wrote: > On 8 September 2011 15:43, Robert Haas wrote: >> I wouldn't be too enthusiastic about >> starting a project like this in January, but now seems fine.  A bigger >> problem is that I don't hear anyone volunteering to do the work. > > You seem

Re: [HACKERS] Patch to improve reliability of postgresql on linux nfs

2011-09-08 Thread Josh Berkus
George, I'm quoting you here because in the version of your email which got posted to the list your whole explanation got put below the patch text, making it hard to find the justification for the patch. Follows: > I run a number of postgresql installations on NFS and on the whole I find > this

[HACKERS] Patch to improve reliability of postgresql on linux nfs

2011-09-08 Thread George Barnett
Hi Hackers, postgresql-writeall.patch Description: Binary data I run a number of postgresql installations on NFS and on the whole I find this to be very reliable. I have however run into a few issues when there is concurrent writes from multiple processes. I see errors such as the following:

Re: [HACKERS] pg_last_xact_insert_timestamp

2011-09-08 Thread Chris Redekop
Thanks for all the feedback guys. Just to throw another monkey wrench in here - I've been playing with Simon's proposed solution of returning 0 when the WAL positions match, and I've come to the realizatiion that even if using pg_last_xact_insert_timestamp, although it would help, we still wouldn'

Re: [HACKERS] Large C files

2011-09-08 Thread Josh Berkus
Simon, Robert, Bruce, Tom, >>> Say what? When else would you have us do it? >> > >>> >> When else would you have us develop? >> > >> > In my eyes that sort of activity *is* development. I find the >> > distinction you are drawing entirely artificial, and more calculated to >> > make sure re

Re: [HACKERS] concurrent snapshots

2011-09-08 Thread Ants Aasma
On Thu, Sep 8, 2011 at 6:46 PM, Robert Haas wrote: > I'm not convinced it's anywhere near that easy.  For one thing, on at > least one big server I'm playing with, memory latency on shared memory > is vastly higher (like >10x!) than on backend-local memory due to NUMA > effects. I wonder if both

Re: [HACKERS] Protecting against multiple instances per cluster

2011-09-08 Thread Magnus Hagander
On Thu, Sep 8, 2011 at 20:40, Thom Brown wrote: > Hi all, > > I've come across a PostgreSQL set up where there are 2 servers, each > with the same version of PostgreSQL on, both mounting the same SAN > onto their respective file systems.  It was intended that only 1 of > the servers would be runni

Re: [HACKERS] memory-related bugs

2011-09-08 Thread Tom Lane
Daniel Farina writes: > On Tue, Sep 6, 2011 at 12:00 PM, Tom Lane wrote: >> I'm still of the opinion that there's no real need to avoid memcpy with >> identical source and destination, so I didn't apply this first patch. > I am leery of memcpy with overlapping regions. I know that it can > caus

Re: [HACKERS] Large C files

2011-09-08 Thread Peter Geoghegan
On 8 September 2011 15:43, Robert Haas wrote: > I wouldn't be too enthusiastic about > starting a project like this in January, but now seems fine.  A bigger > problem is that I don't hear anyone volunteering to do the work. You seem to have a fairly strong opinion on the xlog.c code. It would be

Re: [HACKERS] memory-related bugs

2011-09-08 Thread Daniel Farina
On Tue, Sep 6, 2011 at 12:00 PM, Tom Lane wrote: > [ Sorry for letting this slip through the cracks ... I think I got >  distracted by collation bugs :-( ] > > Noah Misch writes: >> On Sat, Mar 12, 2011 at 12:44:29PM -0500, Tom Lane wrote: >>> Noah Misch writes: A suitably-instrumented run

[HACKERS] Re: [COMMITTERS] pgsql: Add missing format argument to ecpg_log() call

2011-09-08 Thread Peter Eisentraut
On tor, 2011-09-08 at 15:32 -0400, Tom Lane wrote: > Peter Eisentraut writes: > > Add missing format argument to ecpg_log() call > > Oh, fun. So why isn't there an __attribute__((format...)) decoration > on ecpg_log()? I have a larger patch for that. I just wanted to get the one piece of obvio

Re: [HACKERS] WIP: Fast GiST index build

2011-09-08 Thread Alexander Korotkov
Thanks for congratulations! Tnanks to Heikki for mentoring and his work on patch! -- With best regards, Alexander Korotkov.

[HACKERS] pg_dump.c

2011-09-08 Thread Andrew Dunstan
In the "refactoring Large C files" discussion one of the biggest files Bruce mentioned is pg_dump.c. There has been discussion in the past of turning lots of the knowledge currently embedded in this file into a library, which would make it available to other clients (e.g. psql). I'm not sure

Re: [HACKERS] Large C files

2011-09-08 Thread Tom Lane
Simon Riggs writes: > You clearly have the bit between your teeth on this. Personally, I'm neither intending to break up xlog.c right now, nor asking you to do it. I'm just objecting to your claim that there should be some project-policy restriction on when refactoring gets done. I do have other

Re: [HACKERS] Protecting against multiple instances per cluster

2011-09-08 Thread Tom Lane
Thom Brown writes: > I've come across a PostgreSQL set up where there are 2 servers, each > with the same version of PostgreSQL on, both mounting the same SAN > onto their respective file systems. It was intended that only 1 of > the servers would be running an instance of PostgreSQL at a time as

[HACKERS] Protecting against multiple instances per cluster

2011-09-08 Thread Thom Brown
Hi all, I've come across a PostgreSQL set up where there are 2 servers, each with the same version of PostgreSQL on, both mounting the same SAN onto their respective file systems. It was intended that only 1 of the servers would be running an instance of PostgreSQL at a time as they both point to

Re: [HACKERS] WIP: Fast GiST index build

2011-09-08 Thread Oleg Bartunov
My congratulations too, Alexander ! Hope to work on SP-GiST together ! Oleg On Thu, 8 Sep 2011, Heikki Linnakangas wrote: On 06.09.2011 01:18, Alexander Korotkov wrote: Small bugfix: in gistBufferingFindCorrectParent check that downlinkoffnum doesn't exceed maximal offset number. I've commi

Re: [HACKERS] Large C files

2011-09-08 Thread Bruce Momjian
Simon Riggs wrote: > On Thu, Sep 8, 2011 at 3:25 PM, Tom Lane wrote: > > Simon Riggs writes: > >> On Wed, Sep 7, 2011 at 9:02 PM, Tom Lane wrote: > >>> Simon Riggs writes: > Please lets not waste effort on refactoring efforts in mid dev cycle. > > > >>> Say what? ?When else would you have

Re: [HACKERS] Large C files

2011-09-08 Thread Simon Riggs
On Thu, Sep 8, 2011 at 3:25 PM, Tom Lane wrote: > Simon Riggs writes: >> On Wed, Sep 7, 2011 at 9:02 PM, Tom Lane wrote: >>> Simon Riggs writes: Please lets not waste effort on refactoring efforts in mid dev cycle. > >>> Say what?  When else would you have us do it? > >> When else would yo

[HACKERS] Fast GiST index build - further improvements

2011-09-08 Thread Heikki Linnakangas
Now that the main GiST index build patch has been committed, there's a few further improvements that could make it much faster still: Better management of the buffer pages on disk. At the moment, the temporary file is used as a heap of pages belonging to all the buffers in random order. I thin

Re: [HACKERS] force_not_null option support for file_fdw

2011-09-08 Thread Kohei Kaigai
Hi Hanada-san. > ISTM that your results are reasonable for each collation setting. > Former ordering is same as C locale, and in latter case alphabetical order > has priority over case > distinctions. Do you mean that ordering used in file_fdw is affected from > something unexpected, without >

Re: [HACKERS] concurrent snapshots

2011-09-08 Thread Tom Lane
Ants Aasma writes: > On Thu, Sep 8, 2011 at 5:28 PM, Robert Haas wrote: >> 1. A backend can have lots of snapshots, potentially requiring an >> unbounded amount of shared memory.  We can't accommodate that. > If PostgreSQL gets POSIX shared memory capability at some point in the > future, would

Re: [HACKERS] concurrent snapshots

2011-09-08 Thread Robert Haas
On Thu, Sep 8, 2011 at 11:33 AM, Ants Aasma wrote: > On Thu, Sep 8, 2011 at 5:28 PM, Robert Haas wrote: >> On Thu, Sep 8, 2011 at 9:26 AM, Ants Aasma wrote: >>> When go try to find the new csnmin >>> and discover that a backend has a csnmin that is too old, we go through >>> the snapshots of tha

Re: [HACKERS] concurrent snapshots

2011-09-08 Thread Ants Aasma
On Thu, Sep 8, 2011 at 5:28 PM, Robert Haas wrote: > On Thu, Sep 8, 2011 at 9:26 AM, Ants Aasma wrote: >> When go try to find the new csnmin >> and discover that a backend has a csnmin that is too old, we go through >> the snapshots of that backend and convert every snapshot under the >> desired

Re: [HACKERS] FATAL: lock AccessShareLock on object 0/1260/0 is already held

2011-09-08 Thread Tom Lane
daveg writes: > On Wed, Sep 07, 2011 at 09:02:04PM -0400, Tom Lane wrote: >> daveg writes: >>> The first version we saw it on was 8.4.7. >> Yeah, you said that. I was wondering what you'd last run before 8.4.7. > Sorry, misunderstood. We were previously running 8.4.4, but have been on 8.4.7 >

Re: [HACKERS] WIP: Fast GiST index build

2011-09-08 Thread Robert Haas
On Thu, Sep 8, 2011 at 10:59 AM, Heikki Linnakangas wrote: > On 06.09.2011 01:18, Alexander Korotkov wrote: >> >> Small bugfix: in gistBufferingFindCorrectParent check that downlinkoffnum >> doesn't exceed maximal offset number. > > I've committed the patch now, including that fix. Thanks for a gr

Re: [HACKERS] WIP: Fast GiST index build

2011-09-08 Thread Heikki Linnakangas
On 06.09.2011 01:18, Alexander Korotkov wrote: Small bugfix: in gistBufferingFindCorrectParent check that downlinkoffnum doesn't exceed maximal offset number. I've committed the patch now, including that fix. Thanks for a great GSoC project! -- Heikki Linnakangas EnterpriseDB http://ww

Re: [HACKERS] Large C files

2011-09-08 Thread Robert Haas
On Thu, Sep 8, 2011 at 10:29 AM, Bruce Momjian wrote: > Tom Lane wrote: >> Simon Riggs writes: >> > On Wed, Sep 7, 2011 at 9:02 PM, Tom Lane wrote: >> >> Simon Riggs writes: >> >>> Please lets not waste effort on refactoring efforts in mid dev cycle. >> >> >> Say what?  When else would you have

Re: [HACKERS] Large C files

2011-09-08 Thread Bruce Momjian
Tom Lane wrote: > Simon Riggs writes: > > On Wed, Sep 7, 2011 at 9:02 PM, Tom Lane wrote: > >> Simon Riggs writes: > >>> Please lets not waste effort on refactoring efforts in mid dev cycle. > > >> Say what? ?When else would you have us do it? > > > When else would you have us develop? > > In

Re: [HACKERS] concurrent snapshots

2011-09-08 Thread Robert Haas
On Thu, Sep 8, 2011 at 9:26 AM, Ants Aasma wrote: > When go try to find the new csnmin > and discover that a backend has a csnmin that is too old, we go through > the snapshots of that backend and convert every snapshot under the > desired csnmin to a traditional snapshot. I thought about somethi

Re: [HACKERS] Large C files

2011-09-08 Thread Tom Lane
Simon Riggs writes: > On Wed, Sep 7, 2011 at 9:02 PM, Tom Lane wrote: >> Simon Riggs writes: >>> Please lets not waste effort on refactoring efforts in mid dev cycle. >> Say what?  When else would you have us do it? > When else would you have us develop? In my eyes that sort of activity *is*

Re: [HACKERS] cheaper snapshots redux

2011-09-08 Thread Robert Haas
On Tue, Sep 6, 2011 at 11:06 PM, Amit Kapila wrote: > 1. With the above, you want to reduce/remove the concurrency issue between > the GetSnapshotData() [used at begining of sql command execution] and > ProcArrayEndTransaction() [used at end transaction]. The concurrency issue > is mainly ProcArra

Re: [HACKERS] postgresql.conf archive_command example

2011-09-08 Thread Kevin Grittner
Aidan Van Dyk wrote: > If you're copying a file into the archive, and making it appear > non-atomically in your archive, your doing something wrong. > > Period. > > No excuses. +1 -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscri

[HACKERS] concurrent snapshots

2011-09-08 Thread Ants Aasma
Hi, I have been thinking about how to handle long running transactions with Robert’s commit sequence number (CSN) idea. http://archives.postgresql.org/message-id/CA%2BTgmoaAjiq%3Dd%3DkYt3qNj%2BUvi%2BMB-aRovCwr75Ca9egx-Ks9Ag%40mail.gmail.com I just started to go through transaction management code

Re: [HACKERS] postgresql.conf archive_command example

2011-09-08 Thread Aidan Van Dyk
On Thu, Sep 8, 2011 at 2:05 AM, Fujii Masao wrote: > That's an option. But I don't think that finding an existing file is so > serious > problem. The most common cases which cause a partially-filled archived > file are; > > 1. The server crashes while WAL file is being archived, and then the ser

Re: [HACKERS] pg_last_xact_insert_timestamp

2011-09-08 Thread Robert Haas
On Thu, Sep 8, 2011 at 6:14 AM, Fujii Masao wrote: > OTOH, new function enables users to monitor the delay as a timestamp. > For users, a timestamp is obviously easier to handle than LSN, and the delay > as a timestamp is more intuitive. So, I think that it's worth adding > something like pg_last_

Re: [HACKERS] [PATCH] Don't truncate integer part in to_char for 'FM99.'

2011-09-08 Thread Marti Raudsepp
On Wed, Sep 7, 2011 at 23:48, Tom Lane wrote: > Also, the way yours is set up, I'm dubious that it > does the right thing when the last '0' specifier is to the left of the > decimal point. When the last '0' is left of the decimal point, Num->zero_end is set to 0, so the branch dealing with that i

[HACKERS] EXPLAIN and nfiltered, take two

2011-09-08 Thread Marko Tiikkaja
Hi, Here's a patch for $SUBJECT based on the feedback here: http://archives.postgresql.org/message-id/9053.1295888...@sss.pgh.pa.us I intentionally decided to omit the information for Join Filter, since the information can already be deduced from EXPLAIN ANALYZE output, and for Left Joins and

Re: [HACKERS] Back branch update releases this week; beta postponed

2011-09-08 Thread Simon Riggs
On Wed, Sep 7, 2011 at 2:43 AM, Bruce Momjian wrote: > Tom Lane wrote: >> Bernd Helmle writes: >> > --On 10. April 2011 13:53:52 -0400 Tom Lane wrote: >> >> The core team has therefore decided to wrap back-branch >> >> update releases this Thursday for release Monday 4/18. >> >> > Hmm, I would l

Re: [HACKERS] pg_last_xact_insert_timestamp

2011-09-08 Thread Fujii Masao
On Thu, Sep 8, 2011 at 5:55 PM, Simon Riggs wrote: > On Thu, Sep 8, 2011 at 9:36 AM, Fujii Masao wrote: >> The above has been posted in pgsql-general. The reason why Chris thinks >> a counterpart of pg_last_xact_replay_timestamp() is required sounds >> reasonable to me. So I'd like to propose new

Re: [HACKERS] pg_last_xact_insert_timestamp

2011-09-08 Thread Simon Riggs
On Thu, Sep 8, 2011 at 9:36 AM, Fujii Masao wrote: > On Thu, Sep 8, 2011 at 7:06 AM, Chris Redekop wrote: >> Is there anything available to get the last time a transaction >> occurred?like say "pg_last_xact_timestamp"?  In order to accurately >> calculate how far behind my slave is I need to

[HACKERS] pg_last_xact_insert_timestamp

2011-09-08 Thread Fujii Masao
On Thu, Sep 8, 2011 at 7:06 AM, Chris Redekop wrote: > Is there anything available to get the last time a transaction > occurred?like say "pg_last_xact_timestamp"?  In order to accurately > calculate how far behind my slave is I need to do something like > master::pg_last_xact_timestamp() - >

Re: [HACKERS] postgresql.conf archive_command example

2011-09-08 Thread Fujii Masao
On Thu, Sep 8, 2011 at 3:26 PM, Simon Riggs wrote: > On Thu, Sep 8, 2011 at 7:05 AM, Fujii Masao wrote: >> That's an option. But I don't think that finding an existing file is so >> serious >> problem. > > The recommendation should be that the archived files are never > overwritten because that