Re: [HACKERS] Spreading full-page writes

2014-05-26 Thread Simon Riggs
On 25 May 2014 17:52, Heikki Linnakangas wrote: > Here's an idea I tried to explain to Andres and Simon at the pub last night, > on how to reduce the spikes in the amount of WAL written at beginning of a > checkpoint that full-page writes cause. I'm just writing this down for the > sake of the ar

[HACKERS] Race condition within _bt_findinsertloc()? (new page split code)

2014-05-26 Thread Peter Geoghegan
While speccing out a new B-Tree verification tool, I had the opportunity to revisit a thought I had during review concerning _bt_findinsertloc(): that the new coding is unsafe in the event of deferred split completion of a leaf page of a unique index. To recap, we now do the following in _bt_findin

Re: [HACKERS] postgresql.auto.conf read from wrong directory

2014-05-26 Thread Amit Kapila
On Sun, May 11, 2014 at 11:23 PM, Tom Lane wrote: > I think it's clearly *necessary* to forbid setting data_directory in > postgresql.auto.conf. The file is defined to be found in the data > directory, so any such setting is circular logic by definition; > no good can come of not rejecting it. >

Re: [HACKERS] [PATCH] Replacement for OSSP-UUID for Linux and BSD

2014-05-26 Thread Tom Lane
"David E. Wheeler" writes: > On May 26, 2014, at 9:30 PM, Tom Lane wrote: >>> How about --with-unix-uuid? Or --with-ext2-uuid? >> Meh. "Unix" certainly subsumes BSD, so that doesn't seem like a very >> useful distinction. I guess we could use "ext2" but that would just >> confuse most people.

Re: [HACKERS] [PATCH] Replacement for OSSP-UUID for Linux and BSD

2014-05-26 Thread David E. Wheeler
On May 26, 2014, at 9:30 PM, Tom Lane wrote: >> How about --with-unix-uuid? Or --with-ext2-uuid? > > Meh. "Unix" certainly subsumes BSD, so that doesn't seem like a very > useful distinction. I guess we could use "ext2" but that would just > confuse most people. --with-uuid? >> Which one is

Re: [HACKERS] [PATCH] Replacement for OSSP-UUID for Linux and BSD

2014-05-26 Thread Tom Lane
"David E. Wheeler" writes: > On May 26, 2014, at 6:07 PM, Tom Lane wrote: >> This means that if we want to give users control over which implementation >> gets selected, we actually need *three* configure switches. In the >> attached revision of Matteo's patch, I called them --with-ossp-uuid >>

Re: [HACKERS] Re-create dependent views on ALTER TABLE ALTER COLUMN ... TYPE?

2014-05-26 Thread David Fetter
On Tue, May 27, 2014 at 12:37:32AM +0400, ash wrote: > David Fetter writes: > > On Mon, May 26, 2014 at 06:25:09PM +0400, ash wrote: > >> Hi Hackers, > >> > >> This came up recently on general list (and I've just hit the same > >> issue today): > >> http://www.postgresql.org/message-id/cab7npqtlm

Re: [HACKERS] [PATCH] Replacement for OSSP-UUID for Linux and BSD

2014-05-26 Thread David E. Wheeler
On May 26, 2014, at 6:07 PM, Tom Lane wrote: > This means that if we want to give users control over which implementation > gets selected, we actually need *three* configure switches. In the > attached revision of Matteo's patch, I called them --with-ossp-uuid > (the existing switch name), --wit

Re: [HACKERS] fix worker_spi to run as non-dynamic background worker

2014-05-26 Thread Michael Paquier
On Tue, May 27, 2014 at 12:09 PM, Robert Haas wrote: > On Fri, May 23, 2014 at 10:38 AM, Michael Paquier > wrote: >> On Fri, May 23, 2014 at 6:29 PM, Shigeru Hanada >> wrote: >>> I noticed that contrib/worker_spi can't run as non-dynamic background >>> worker (IOW, load via shared_preload_librar

Re: [HACKERS] Compression of full-page-writes

2014-05-26 Thread Rahila Syed
Hello All, 0001-CompressBackupBlock_snappy_lz4_pglz extends patch on compression of full page writes to include LZ4 and Snappy . Changes include making "compress_backup_block" GUC from boolean to enum. Value of the GUC can be OFF, pglz, snappy or lz4 which can be used to turn off compression or

Re: [HACKERS] postgres_fdw and connection management

2014-05-26 Thread Fabrízio de Royes Mello
On Mon, May 26, 2014 at 11:47 PM, Shigeru Hanada wrote: > > 2014-05-24 0:09 GMT+09:00 Sandro Santilli : > > Indeed I tried "DISCARD ALL" in hope it would have helped, so I find > > good your idea of allowing extensions to register an hook there. > > > > Still, I'd like the FDW handler itself to po

Re: [HACKERS] Spreading full-page writes

2014-05-26 Thread Robert Haas
On Mon, May 26, 2014 at 1:22 PM, Heikki Linnakangas wrote: >>I don't think we know that. The server might have crashed before that >>second record got generated. (This appears to be an unfixable flaw in >>this proposal.) > > The second record is generated before the checkpoint is finished and the

Re: [HACKERS] fix worker_spi to run as non-dynamic background worker

2014-05-26 Thread Robert Haas
On Fri, May 23, 2014 at 10:38 AM, Michael Paquier wrote: > On Fri, May 23, 2014 at 6:29 PM, Shigeru Hanada > wrote: >> I noticed that contrib/worker_spi can't run as non-dynamic background >> worker (IOW, load via shared_preload_libraries), because of >> uninitialized bgw_notify_pid. > That's act

Re: [HACKERS] fix worker_spi to run as non-dynamic background worker

2014-05-26 Thread Robert Haas
On Fri, May 23, 2014 at 5:29 AM, Shigeru Hanada wrote: > I noticed that contrib/worker_spi can't run as non-dynamic background > worker (IOW, load via shared_preload_libraries), because of > uninitialized bgw_notify_pid. > > Attached patch fixes this issue. Please apply onto HEAD and 9.4. Good c

Re: [HACKERS] postgres_fdw and connection management

2014-05-26 Thread Shigeru Hanada
2014-05-24 0:09 GMT+09:00 Sandro Santilli : > Indeed I tried "DISCARD ALL" in hope it would have helped, so I find > good your idea of allowing extensions to register an hook there. > > Still, I'd like the FDW handler itself to possibly be configured > to disable the pool completely as a server-spe

Re: [HACKERS] Re: popen and pclose redefinitions causing many warning in Windows build

2014-05-26 Thread Tom Lane
Michael Paquier writes: > On Tue, May 27, 2014 at 1:39 AM, Tom Lane wrote: >> The best alternative I can think of is to use strncmp() to check for >> whether the head of the string matches "COPY ", and then perform the >> integer conversion using strtoull() #ifdef HAVE_STRTOULL and strtoul() >> o

Re: [HACKERS] Re: popen and pclose redefinitions causing many warning in Windows build

2014-05-26 Thread Michael Paquier
On Tue, May 27, 2014 at 1:39 AM, Tom Lane wrote: > The best alternative I can think of is to use strncmp() to check for > whether the head of the string matches "COPY ", and then perform the > integer conversion using strtoull() #ifdef HAVE_STRTOULL and strtoul() > otherwise. What about the attach

Re: [HACKERS] [PATCH] Replacement for OSSP-UUID for Linux and BSD

2014-05-26 Thread Tom Lane
I wrote: > Matteo Beccati writes: >> On 26/05/2014 19:31, Andres Freund wrote: >>> On 2014-05-26 13:25:49 -0400, Tom Lane wrote: Matteo Beccati writes: > * Restored --with-ossp-uuid. Configure tries ossp support first, then > falls back to Linux and BSD variants >>> Imo should be th

[HACKERS] Extending MSVC scripts to support --with-extra-version

2014-05-26 Thread Michael Paquier
Hi all, Please find attached a patch extending support of --with-extra-version in the MSVC scripts. This is something I have been using internally, and I believe that it is useful for Windows packagers. Similarly to the other options, a new field called extraver is added in config_default.pl and i

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Amit Langote
On Tue, May 27, 2014 at 6:19 AM, Josh Berkus wrote: > On 05/26/2014 08:52 AM, Tom Lane wrote: >> Alvaro Herrera writes: >>> Amit Langote wrote: On Mon, May 26, 2014 at 10:59 PM, Fujii Masao wrote: > IMO it's better if we can write SQL in multiples line *without* a tailing > es

Re: [HACKERS] [PATCH] Replacement for OSSP-UUID for Linux and BSD

2014-05-26 Thread Tom Lane
Christoph Berg writes: > Re: Tom Lane 2014-05-25 <12508.1401045...@sss.pgh.pa.us> >> Hmm ... this is not actually what I had in mind. Unless I'm misreading >> the patch, this nukes the "uuid-ossp" extension entirely in favor of a >> new extension "uuid" (providing the same SQL functions with a di

Re: [HACKERS] [PATCH] Replacement for OSSP-UUID for Linux and BSD

2014-05-26 Thread Christoph Berg
Re: Tom Lane 2014-05-25 <12508.1401045...@sss.pgh.pa.us> > Matteo Beccati writes: > > here's the latest version of my uuid changes patch, according to > > proposal (2) from Tom in the thread about OSSP-UUID[1]. > > Hmm ... this is not actually what I had in mind. Unless I'm misreading > the patc

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Christoph Berg
Re: Tom Lane 2014-05-26 <26629.1401119...@sss.pgh.pa.us> > >> Yeah, that would be much cleaner. > > > But that would require duplicating the lexing stuff to determine where > > quotes are and where commands end. There are already some cases where > > pgbench itself is the bottleneck; adding a lex

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Josh Berkus
On 05/26/2014 08:52 AM, Tom Lane wrote: > Alvaro Herrera writes: >> Amit Langote wrote: >>> On Mon, May 26, 2014 at 10:59 PM, Fujii Masao wrote: IMO it's better if we can write SQL in multiples line *without* a tailing escape character, like psql's input file. > >>> Yeah, that would be

Re: [HACKERS] Re-create dependent views on ALTER TABLE ALTER COLUMN ... TYPE?

2014-05-26 Thread ash
David Fetter writes: > On Mon, May 26, 2014 at 06:25:09PM +0400, ash wrote: >> Hi Hackers, >> >> This came up recently on general list (and I've just hit the same issue >> today): >> >> http://www.postgresql.org/message-id/cab7npqtlmmn1ltb5we0v0do57ip0u73ykwzbzytaxdf1caw...@mail.gmail.com >

[HACKERS] Stating the significance of Lehman & Yao in the nbtree README

2014-05-26 Thread Peter Geoghegan
While talking to various people during pgCon, I was reminded that the nbtree README does a poor job of explaining the actual practical advantages of L&Y from a high level. The "move right" trick is initially mentioned only as an adjunct to a discussion of the special-case handling of root page spli

Re: [HACKERS] Re-create dependent views on ALTER TABLE ALTER COLUMN ... TYPE?

2014-05-26 Thread David Fetter
On Mon, May 26, 2014 at 06:25:09PM +0400, ash wrote: > Hi Hackers, > > This came up recently on general list (and I've just hit the same issue > today): > > http://www.postgresql.org/message-id/cab7npqtlmmn1ltb5we0v0do57ip0u73ykwzbzytaxdf1caw...@mail.gmail.com > > Why couldn't postgres re-cre

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Alvaro Herrera
Andres Freund wrote: > I've more than once wished for the capability. +1 -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscr

Re: [HACKERS] Spreading full-page writes

2014-05-26 Thread Greg Stark
On Mon, May 26, 2014 at 1:22 PM, Heikki Linnakangas wrote: > The second record is generated before the checkpoint is finished and the > checkpoint record is written. So it will be there. > > (if you crash before the checkpoint is finished, the in-progress > checkpoint is no good for recovery any

Re: [HACKERS] SQL access to database attributes

2014-05-26 Thread Vik Fearing
On 05/26/2014 07:10 AM, Stephen Frost wrote: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> I don't really object to doing an unlocked check for another such >> database, but I'm not convinced that additional locking to try to >> prevent a race is worth its keep. > +1 on the nannyism, and +1 to ignori

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Tom Lane
Jeff Janes writes: > On Monday, May 26, 2014, Tom Lane wrote: >> This seems pretty likely to break existing scripts that happen to contain >> backslashes. Is it really worth the compatibility risk? > Do you mean due to the bugs you point out, or in general? Is it really at > all likely that so

Re: [HACKERS] [PATCH] Replacement for OSSP-UUID for Linux and BSD

2014-05-26 Thread Tom Lane
Matteo Beccati writes: > On 26/05/2014 19:31, Andres Freund wrote: >> On 2014-05-26 13:25:49 -0400, Tom Lane wrote: >>> Matteo Beccati writes: * Restored --with-ossp-uuid. Configure tries ossp support first, then falls back to Linux and BSD variants >> Imo should be the other way round

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Andres Freund
On 2014-05-26 11:44:31 -0400, Tom Lane wrote: > Amit Langote writes: > > In a custom pgbench script, it seems convenient to be able to split a > > really long query to span multiple lines using an escape character > > (bash-style). Attached adds that capability to read_line_from_file() > > in pgbe

Re: [HACKERS] [PATCH] Replacement for OSSP-UUID for Linux and BSD

2014-05-26 Thread Matteo Beccati
On 26/05/2014 19:31, Andres Freund wrote: > On 2014-05-26 13:25:49 -0400, Tom Lane wrote: >> Matteo Beccati writes: >>> I'm attaching v2 of the patch. Here's a list of changes from v1: >> >>> * Restored --with-ossp-uuid. Configure tries ossp support first, then >>> falls back to Linux and BSD vari

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Jeff Janes
On Monday, May 26, 2014, Tom Lane wrote: > Amit Langote > writes: > > In a custom pgbench script, it seems convenient to be able to split a > > really long query to span multiple lines using an escape character > > (bash-style). Attached adds that capability to read_line_from_file() > > in pgbenc

Re: [HACKERS] [PATCH] Replacement for OSSP-UUID for Linux and BSD

2014-05-26 Thread Andres Freund
On 2014-05-26 13:25:49 -0400, Tom Lane wrote: > Matteo Beccati writes: > > I'm attaching v2 of the patch. Here's a list of changes from v1: > > > * Restored --with-ossp-uuid. Configure tries ossp support first, then > > falls back to Linux and BSD variants Imo should be the other way round. > >

Re: [HACKERS] [PATCH] Replacement for OSSP-UUID for Linux and BSD

2014-05-26 Thread Tom Lane
Matteo Beccati writes: > I'm attaching v2 of the patch. Here's a list of changes from v1: > * Restored --with-ossp-uuid. Configure tries ossp support first, then > falls back to Linux and BSD variants > * md5.o and sha1.o are linked only when not using the ossp library > * fixed a bug in the v1mc

Re: [HACKERS] Spreading full-page writes

2014-05-26 Thread Heikki Linnakangas
On 26 May 2014 20:16:33 EEST, Robert Haas wrote: >On May 25, 2014, at 5:52 PM, Heikki Linnakangas > wrote: >> Here's how this works out during replay: >> >> a) You start WAL replay from the latest checkpoint's Redo-pointer. >> >> When you see a WAL record that's been marked with XLR_FPW_SKIPPED,

Re: [HACKERS] Spreading full-page writes

2014-05-26 Thread Robert Haas
On May 25, 2014, at 5:52 PM, Heikki Linnakangas wrote: > Here's how this works out during replay: > > a) You start WAL replay from the latest checkpoint's Redo-pointer. > > When you see a WAL record that's been marked with XLR_FPW_SKIPPED, don't > replay that record at all. It's OK because we k

Re: [HACKERS] Re: popen and pclose redefinitions causing many warning in Windows build

2014-05-26 Thread Tom Lane
Michael Paquier writes: > On Fri, May 23, 2014 at 10:43 PM, Alvaro Herrera > wrote: >> c:/mingw/msys/1.0/home/pgrunner/bf/root/HEAD/pgsql.5100/../pgsql/contrib/pg_stat_statements/pg_stat_statements.c: >> In function 'pgss_ProcessUtility': >> c:/mingw/msys/1.0/home/pgrunner/bf/root/HEAD/pgsql.510

Re: [HACKERS] IMPORT FOREIGN SCHEMA statement

2014-05-26 Thread Tom Lane
Albe Laurenz writes: > In addition to data type mapping questions (which David already raised) > I have one problem when I think of the Oracle FDW: > Oracle follows the SQL standard in folding table names to upper case. > So this would normally result in a lot of PostgreSQL foreign tables > with

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Amit Langote
On Tue, May 27, 2014 at 12:44 AM, Tom Lane wrote: > Amit Langote writes: >> In a custom pgbench script, it seems convenient to be able to split a >> really long query to span multiple lines using an escape character >> (bash-style). Attached adds that capability to read_line_from_file() >> in pgb

Re: [HACKERS] Priority table or Cache table

2014-05-26 Thread Hannu Krosing
On 05/26/2014 04:16 PM, Fujii Masao wrote: > On Sun, May 25, 2014 at 6:52 PM, Hannu Krosing wrote: >> On 05/20/2014 01:46 PM, Fujii Masao wrote: >>> On Mon, Mar 17, 2014 at 1:16 PM, Haribabu Kommi >>> wrote: ... I Implemented a proof of concept patch to see whether the buffer pool

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Tom Lane
Alvaro Herrera writes: > Amit Langote wrote: >> On Mon, May 26, 2014 at 10:59 PM, Fujii Masao wrote: >>> IMO it's better if we can write SQL in multiples line *without* a tailing >>> escape character, like psql's input file. >> Yeah, that would be much cleaner. > But that would require duplicat

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Tom Lane
Amit Langote writes: > In a custom pgbench script, it seems convenient to be able to split a > really long query to span multiple lines using an escape character > (bash-style). Attached adds that capability to read_line_from_file() > in pgbench.c This seems pretty likely to break existing script

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Alvaro Herrera
Amit Langote wrote: > On Mon, May 26, 2014 at 10:59 PM, Fujii Masao wrote: > > On Mon, May 26, 2014 at 6:50 PM, Amit Langote > > wrote: > >> Thoughts? > > > > IMO it's better if we can write SQL in multiples line *without* a tailing > > escape character, like psql's input file. > > Yeah, that w

[HACKERS] pg_llog not mentioned in "Database File Layout"

2014-05-26 Thread Amit Langote
Hi, Just noticed pg_llog is not mentioned in the "Database File Layout" section. Wonder if it's an oversight? -- Amit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Amit Langote
On Mon, May 26, 2014 at 10:59 PM, Fujii Masao wrote: > On Mon, May 26, 2014 at 6:50 PM, Amit Langote wrote: >> Thoughts? > > IMO it's better if we can write SQL in multiples line *without* a tailing > escape character, like psql's input file. > Yeah, that would be much cleaner. -- Amit -- Se

Re: [HACKERS] Re-create dependent views on ALTER TABLE ALTER COLUMN ... TYPE?

2014-05-26 Thread Stephen Frost
ash, * ash (a...@commandprompt.com) wrote: > This came up recently on general list (and I've just hit the same issue > today): > > http://www.postgresql.org/message-id/cab7npqtlmmn1ltb5we0v0do57ip0u73ykwzbzytaxdf1caw...@mail.gmail.com > > Why couldn't postgres re-create the dependent views au

Re: [HACKERS] Spreading full-page writes

2014-05-26 Thread Fujii Masao
On Mon, May 26, 2014 at 6:52 AM, Heikki Linnakangas wrote: > Here's an idea I tried to explain to Andres and Simon at the pub last night, > on how to reduce the spikes in the amount of WAL written at beginning of a > checkpoint that full-page writes cause. I'm just writing this down for the > sake

[HACKERS] Re-create dependent views on ALTER TABLE ALTER COLUMN ... TYPE?

2014-05-26 Thread ash
Hi Hackers, This came up recently on general list (and I've just hit the same issue today): http://www.postgresql.org/message-id/cab7npqtlmmn1ltb5we0v0do57ip0u73ykwzbzytaxdf1caw...@mail.gmail.com Why couldn't postgres re-create the dependent views automatically? I mean it knows which views d

Re: [HACKERS] Priority table or Cache table

2014-05-26 Thread Fujii Masao
On Sun, May 25, 2014 at 6:52 PM, Hannu Krosing wrote: > On 05/20/2014 01:46 PM, Fujii Masao wrote: >> On Mon, Mar 17, 2014 at 1:16 PM, Haribabu Kommi >> wrote: >>> ... >>> I Implemented a proof of concept patch to see whether the buffer pool >>> split can improve the performance or not. >>> >>> S

Re: [HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Fujii Masao
On Mon, May 26, 2014 at 6:50 PM, Amit Langote wrote: > Hi, > > In a custom pgbench script, it seems convenient to be able to split a > really long query to span multiple lines using an escape character > (bash-style). Attached adds that capability to read_line_from_file() > in pgbench.c > > For ex

Re: [HACKERS] Re: popen and pclose redefinitions causing many warning in Windows build

2014-05-26 Thread Michael Paquier
On Fri, May 23, 2014 at 10:43 PM, Alvaro Herrera wrote: > x86_64-w64-mingw32-gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith > -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute > -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g > -I../../../

Re: [HACKERS] IMPORT FOREIGN SCHEMA statement

2014-05-26 Thread Albe Laurenz
Ronan Dunklau wrote: > Since my last proposal didn't get any strong rebuttal, please find attached a > more complete version of the IMPORT FOREIGN SCHEMA statement. > > I tried to follow the SQL-MED specification as closely as possible. > > This adds discoverability to foreign servers. The struct

[HACKERS] Allowing line-continuation in pgbench custom scripts

2014-05-26 Thread Amit Langote
Hi, In a custom pgbench script, it seems convenient to be able to split a really long query to span multiple lines using an escape character (bash-style). Attached adds that capability to read_line_from_file() in pgbench.c For example, BEGIN; \setrandom 1 1650 UPDATE table \ SET col2 =