Re: [HACKERS] PATCH: adaptive ndistinct estimator v4

2015-04-14 Thread Jeff Janes
On Tue, Mar 31, 2015 at 12:02 PM, Tomas Vondra wrote: > Hi all, > > attached is v4 of the patch implementing adaptive ndistinct estimator. > Hi Tomas, I have a case here where the adaptive algorithm underestimates ndistinct by a factor of 7 while the default estimator is pretty close. 5MB file

Re: [HACKERS] initdb -S and tablespaces

2015-04-14 Thread Abhijit Menon-Sen
At 2015-04-15 11:40:34 +0530, a...@2ndquadrant.com wrote: > > Still, thanks to the example in basebackup.c, I've got most of a patch > that should follow any symlinks in PGDATA. I notice that src/bin/pg_rewind/copy_fetch.c has a traverse_datadir() function that does what we want (but it recurses i

Re: [HACKERS] initdb -S and tablespaces

2015-04-14 Thread Abhijit Menon-Sen
At 2015-04-06 10:16:36 -0300, alvhe...@2ndquadrant.com wrote: > > Well, we have many things that can be set as symlinks; some you can > have initdb create the links for you (initdb --xlogdir), others you > can move manually. Looking at sendDir() in backend/replication/basebackup.c, I notice that t

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Michael Paquier
On Wed, Apr 15, 2015 at 2:38 PM, Noah Misch wrote: > Solaris 10 ships Perl 5.8.4, and RHEL 5.11 ships Perl 5.8.8. Therefore, Perl > installations lacking this File::Path feature will receive vendor support for > years to come. Replacing the use of keep_root with rmtree+mkdir will add 2-10 > lines

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Noah Misch
On Tue, Apr 14, 2015 at 05:29:36PM -0300, Alvaro Herrera wrote: > David E. Wheeler wrote: > > On Apr 14, 2015, at 1:21 PM, Alvaro Herrera > > wrote: > > > > > Castoroides has 5.8.4. Oops. > > > > WUT. > > Yeah, eh? Anyway I don't think it matters much: just don't enable TAP > tests on machin

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Fujii Masao
On Wed, Apr 15, 2015 at 11:55 AM, Michael Paquier wrote: > On Wed, Apr 15, 2015 at 11:10 AM, Fujii Masao wrote: >> On Wed, Apr 15, 2015 at 12:00 AM, Magnus Hagander >> wrote: >>> On Tue, Apr 14, 2015 at 4:50 PM, Heikki Linnakangas wrote: On 04/14/2015 05:42 AM, Robert Haas wrote: >>>

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Robert Haas
On Wed, Apr 15, 2015 at 12:15 AM, Amit Kapila wrote: > IIUC, this will allow us to increase usage count only when the buffer > is touched by clocksweep to decrement the usage count. Yes. > I think such a solution will be good for the cases when many evictions > needs to be performed to satisfy t

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Robert Haas
On Tue, Apr 14, 2015 at 7:10 PM, Greg Stark wrote: > The way the clock sweep algorithm is meant to be thought about is that it's > an approximate lru. Each usage count corresponds to an ntile of the lru. So > we don't know which buffer is least recently used but it must be in the set > of buffers

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Amit Kapila
On Wed, Apr 15, 2015 at 2:55 AM, Robert Haas wrote: > > On Wed, Apr 16, 2014 at 2:44 PM, Tom Lane wrote: > > Merlin Moncure writes: > >> Anyways, I'm still curious if you can post similar numbers basing the > >> throttling on gross allocation counts instead of time. Meaning: some > >> number of

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Michael Paquier
On Wed, Apr 15, 2015 at 12:15 AM, Stephen Frost wrote: > We need a proper "hardening" > guide anyway, this would just need to be included in that documentation. +1. I am sure that many users would like a hardening manual in the official documentation. -- Michael -- Sent via pgsql-hackers maili

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Michael Paquier
On Wed, Apr 15, 2015 at 11:55 AM, Michael Paquier wrote: > OK. I am fine to implement anything required here if needed, meaning > the following: > 1) Doc patch to mention that it is possible that compression can give > hints to attackers when working on sensible fields that have a > non-fixed size.

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Michael Paquier
On Wed, Apr 15, 2015 at 11:10 AM, Fujii Masao wrote: > On Wed, Apr 15, 2015 at 12:00 AM, Magnus Hagander wrote: >> On Tue, Apr 14, 2015 at 4:50 PM, Heikki Linnakangas wrote: >>> >>> On 04/14/2015 05:42 AM, Robert Haas wrote: On Sun, Apr 12, 2015 at 8:38 PM, Heikki Linnakangas wro

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Fujii Masao
On Wed, Apr 15, 2015 at 12:00 AM, Magnus Hagander wrote: > On Tue, Apr 14, 2015 at 4:50 PM, Heikki Linnakangas wrote: >> >> On 04/14/2015 05:42 AM, Robert Haas wrote: >>> >>> On Sun, Apr 12, 2015 at 8:38 PM, Heikki Linnakangas >>> wrote: As to RLS - yeah, that's where I think a lot of

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Robert Haas
On Tue, Apr 14, 2015 at 6:22 PM, Peter Geoghegan wrote: > Why is that good? We did discuss this before. I've recapped some of what I believe to be the most salient points below. > I think that people were all too quick to dismiss the idea of a wall > time interval playing some role here (at lea

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-14 Thread Robert Haas
On Tue, Apr 14, 2015 at 6:07 PM, Simon Riggs wrote: > On 11 March 2015 at 20:55, Peter Eisentraut wrote: >> I don't know how to move forward. We could give users a knob: This >> might make your queries faster or not -- good luck. But of course >> nobody will like that either. > > What is clear

Re: [HACKERS] Auditing extension for PostgreSQL (Take 2)

2015-04-14 Thread Sawada Masahiko
On Wed, Apr 15, 2015 at 8:57 AM, David Steele wrote: > On 4/14/15 7:13 PM, Tatsuo Ishii wrote: >> This patch does not apply cleanly due to the moving of pgbench (patch >> to filelist.sgml failed). > > Thank you for pointing that out! > > Ironic that it was the commit directly after the one I was t

Re: [HACKERS] Auditing extension for PostgreSQL (Take 2)

2015-04-14 Thread Tatsuo Ishii
> Thank you for pointing that out! > > Ironic that it was the commit directly after the one I was testing with > that broke the patch. It appears the end of the last CF is a very bad > time to be behind HEAD. > > Fixed in attached v8 patch. Thank you for your quick response. BTW, in my underst

Re: [HACKERS] inherit support for foreign tables

2015-04-14 Thread Kyotaro HORIGUCHI
Hi, Before suppressing the symptom, I doubt the necessity and/or validity of giving foreign tables an ability to be a parent. Is there any reasonable usage for the ability? I think we should choose to inhibit foreign tables from becoming a parent rather than leaving it allowed then taking measure

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Michael Paquier
On Wed, Apr 15, 2015 at 5:29 AM, Alvaro Herrera wrote: > David E. Wheeler wrote: >> On Apr 14, 2015, at 1:21 PM, Alvaro Herrera wrote: >> >> > Castoroides has 5.8.4. Oops. >> >> WUT. > > Yeah, eh? Anyway I don't think it matters much: just don't enable TAP > tests on machines with obsolete Perl

Re: [HACKERS] Auditing extension for PostgreSQL (Take 2)

2015-04-14 Thread David Steele
On 4/14/15 7:13 PM, Tatsuo Ishii wrote: > This patch does not apply cleanly due to the moving of pgbench (patch > to filelist.sgml failed). Thank you for pointing that out! Ironic that it was the commit directly after the one I was testing with that broke the patch. It appears the end of the las

Re: [HACKERS] Auditing extension for PostgreSQL (Take 2)

2015-04-14 Thread Tatsuo Ishii
This patch does not apply cleanly due to the moving of pgbench (patch to filelist.sgml failed). Best regards, -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese:http://www.sraoss.co.jp > Attached is the v7 pg_audit patch. > > I've tried to address Peter's

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-14 Thread David Steele
On 4/14/15 6:07 PM, Simon Riggs wrote: > On 11 March 2015 at 20:55, Peter Eisentraut wrote: > >> I don't know how to move forward. We could give users a knob: This >> might make your queries faster or not -- good luck. But of course >> nobody will like that either. > > What is clear is that la

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Greg Stark
I've been meaning to write this since PGConf and now isn't a great time since I'm on my phone but I think it's time. The way the clock sweep algorithm is meant to be thought about is that it's an approximate lru. Each usage count corresponds to an ntile of the lru. So we don't know which buffer is

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Jim Nasby
On 4/14/15 5:22 PM, Peter Geoghegan wrote: As long as we're doing random brainstorming, I'd suggest looking at making clocksweep actually approximate LRU-K/LRU-2 (which, again, to be clear, my prototype did not do). The clocksweep could maintain statistics about the recency of the second-to-last

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Peter Geoghegan
On Tue, Apr 14, 2015 at 2:25 PM, Robert Haas wrote: > So, I was thinking about this a little bit more today, prodded by my > coworker John Gorman. I'm wondering if we could drive this off of the > clock sweep; that is, every time the clock sweep touches a buffer, its > usage count goes down by on

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-14 Thread Simon Riggs
On 11 March 2015 at 20:55, Peter Eisentraut wrote: > I don't know how to move forward. We could give users a knob: This > might make your queries faster or not -- good luck. But of course > nobody will like that either. What is clear is that large SELECT queries are doing the work VACUUM shoul

Re: [HACKERS] logical column ordering

2015-04-14 Thread Robert Haas
On Tue, Apr 14, 2015 at 2:38 PM, Alvaro Herrera wrote: > As I said, I'm still writing the first pieces of this so I'm not sure > what other ramifications it will have. If there are any thoughts, I > would appreciate them. (Particularly useful input on whether it is > acceptable to omit lognums/p

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2015-04-14 Thread Robert Haas
On Wed, Apr 16, 2014 at 2:44 PM, Tom Lane wrote: > Merlin Moncure writes: >> Anyways, I'm still curious if you can post similar numbers basing the >> throttling on gross allocation counts instead of time. Meaning: some >> number of buffer allocations has to have occurred before you consider >> e

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Alvaro Herrera
David E. Wheeler wrote: > On Apr 14, 2015, at 1:21 PM, Alvaro Herrera wrote: > > > Castoroides has 5.8.4. Oops. > > WUT. Yeah, eh? Anyway I don't think it matters much: just don't enable TAP tests on machines with obsolete Perl. I think this is fine since 5.8's latest release is supported.

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread David E. Wheeler
On Apr 14, 2015, at 1:21 PM, Alvaro Herrera wrote: > Castoroides has 5.8.4. Oops. WUT. smime.p7s Description: S/MIME cryptographic signature

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Alvaro Herrera
David E. Wheeler wrote: > On Apr 14, 2015, at 9:05 AM, Alvaro Herrera wrote: > > >> http://perldoc.perl.org/File/Path.html > >> With this formulation: > >> remove_tree($tempdir, {keep_root => 1}); > > > > Does Perl 5.8 have this? > > Yes, it does. > > http://cpansearch.perl.org/src/NWCLARK/p

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread David Fetter
On Tue, Apr 14, 2015 at 09:25:33AM -0700, David E. Wheeler wrote: > On Apr 14, 2015, at 9:05 AM, Alvaro Herrera wrote: > > >> http://perldoc.perl.org/File/Path.html > >> With this formulation: > >> remove_tree($tempdir, {keep_root => 1}); > > > > Does Perl 5.8 have this? > > Yes, it does. > >

Re: [HACKERS] inherit support for foreign tables

2015-04-14 Thread Alvaro Herrera
Jim Nasby wrote: > On 4/14/15 5:49 AM, Etsuro Fujita wrote: > > postgres=# create foreign table ft1 (c1 int) server myserver options > > (table_name 't1'); > > CREATE FOREIGN TABLE > > postgres=# create foreign table ft2 (c1 int) server myserver options > > (table_name 't2'); > > CREATE FOREIGN TAB

Re: [HACKERS] logical column ordering

2015-04-14 Thread Alvaro Herrera
I've been looking at this again. It has become apparent to me that what we're doing in parse analysis is wrong, and the underlying node representation is wrong too. Here's a different approach, which I hope will give better fruits. I'm still working on implementing the ideas here (and figuring o

[HACKERS] Re: [COMMITTERS] pgsql: Use Intel SSE 4.2 CRC instructions where available.

2015-04-14 Thread Heikki Linnakangas
On 04/14/2015 07:01 PM, Simon Riggs wrote: On 14 April 2015 at 11:34, Heikki Linnakangas wrote: Or we can punt and always build the version with the runtime check, unless overridden manually at configure command line. That seems safer as the default, which is what the original patch did. We

Re: [HACKERS] inherit support for foreign tables

2015-04-14 Thread Jim Nasby
On 4/14/15 5:49 AM, Etsuro Fujita wrote: > postgres=# create foreign table ft1 (c1 int) server myserver options > (table_name 't1'); > CREATE FOREIGN TABLE > postgres=# create foreign table ft2 (c1 int) server myserver options > (table_name 't2'); > CREATE FOREIGN TABLE > postgres=# alter foreign t

Re: [HACKERS] EvalPlanQual behaves oddly for FDW queries involving system columns

2015-04-14 Thread Jim Nasby
On 4/14/15 1:05 AM, Kyotaro HORIGUCHI wrote: As an example, the following operations cause an "unexpected" result. Ex. 1 Session A=# create table t (a int primary key, b int); Session A=# insert into t (select a, 1 from generate_series(0, 99) a); Session A=# begin; Session A=# select * from t wh

Re: [HACKERS] deparsing utility commands

2015-04-14 Thread David Steele
On 4/9/15 12:14 PM, Alvaro Herrera wrote: > Alvaro Herrera wrote: >> Executive summary: >> >> There is now a CommandDeparse_hook; >> deparse_utility_command is provided as an extension, intended for 9.6; >> rest of patch would be pushed to 9.5. > > Actually here's another approach I like better: u

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread David E. Wheeler
On Apr 14, 2015, at 9:05 AM, Alvaro Herrera wrote: >> http://perldoc.perl.org/File/Path.html >> With this formulation: >> remove_tree($tempdir, {keep_root => 1}); > > Does Perl 5.8 have this? Yes, it does. http://cpansearch.perl.org/src/NWCLARK/perl-5.8.9/lib/File/Path.pm Best, David smim

Re: [HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Alvaro Herrera
Michael Paquier wrote: > Hi all, > > I noticed that src/bin/initdb/t/001_initdb.pl uses directly rm via a > system() call like that: > system_or_bail "rm -rf '$tempdir'/*"; > > This way of doing is not portable, particularly on platforms that do > not have rm like... Windows where the equivalent

Re: [HACKERS] [COMMITTERS] pgsql: Use Intel SSE 4.2 CRC instructions where available.

2015-04-14 Thread Simon Riggs
On 14 April 2015 at 11:34, Heikki Linnakangas wrote: > Or we can punt and always build the version with > the runtime check, unless overridden manually at configure command line. That seems safer as the default, which is what the original patch did. We can optimise that for known good builds la

[HACKERS] Re: [COMMITTERS] pgsql: Use Intel SSE 4.2 CRC instructions where available.

2015-04-14 Thread Heikki Linnakangas
On 04/14/2015 06:28 PM, Simon Riggs wrote: On 14 April 2015 at 10:09, Heikki Linnakangas wrote: Abhijit Menon-Sen, heavily modified by me, reviewed by Andres Freund. Did the heavy modifications have any affect on the patch behaviour, or was this just related to where you would like to put th

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Stephen Frost
* Magnus Hagander (mag...@hagander.net) wrote: > On Tue, Apr 14, 2015 at 4:50 PM, Heikki Linnakangas wrote: > > I'm not a big fan of locking down WAL position information either. If we > > have to treat the current WAL position is security-sensitive information, > > we're doing something wrong. I

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Magnus Hagander
On Tue, Apr 14, 2015 at 4:50 PM, Heikki Linnakangas wrote: > On 04/14/2015 05:42 AM, Robert Haas wrote: > >> On Sun, Apr 12, 2015 at 8:38 PM, Heikki Linnakangas >> wrote: >> >>> As to RLS - yeah, that's where I think a lot of the possible covert >>> >> channel attacks are. But it doesn't have t

Re: [HACKERS] FPW compression leaks information

2015-04-14 Thread Heikki Linnakangas
On 04/14/2015 05:42 AM, Robert Haas wrote: On Sun, Apr 12, 2015 at 8:38 PM, Heikki Linnakangas wrote: Care to name some? This is certainly quite cumbersome to exploit, but it's doable. We've talked a lot about covert channels and timing attacks on RLS, but this makes me more worried because yo

Re: [HACKERS] BRIN range operator class

2015-04-14 Thread Emre Hasegeli
> Judging from a quick look, I think patches 1 and 5 can be committed > quickly; they imply no changes to other parts of BRIN. (Not sure why 1 > and 5 are separate. Any reason for this?) Also patch 2. Not much reason except that 1 includes only functions, but 5 includes operators. > Patch 4 lo

Re: [HACKERS] What exactly is our CRC algorithm?

2015-04-14 Thread Heikki Linnakangas
On 04/03/2015 05:28 AM, Abhijit Menon-Sen wrote: At 2015-04-03 00:33:10 +0300, hlinn...@iki.fi wrote: I came up with the attached. I like it very much. src/port/Makefile has (note src/srv): +# pg_crc32c_sse42.o and its _src.o version need CFLAGS_SSE42 +pg_crc32c_sse42.o: CFLAGS+=$

[HACKERS] Make more portable TAP tests of initdb

2015-04-14 Thread Michael Paquier
Hi all, I noticed that src/bin/initdb/t/001_initdb.pl uses directly rm via a system() call like that: system_or_bail "rm -rf '$tempdir'/*"; This way of doing is not portable, particularly on platforms that do not have rm like... Windows where the equivalent is del. And we could actually use remov

Re: [HACKERS] inherit support for foreign tables

2015-04-14 Thread Etsuro Fujita
On 2015/03/23 2:57, Tom Lane wrote: > Etsuro Fujita writes: >> [ fdw-inh-8.patch ] > > I've committed this with some substantial rearrangements, notably: > > * I thought that if we were doing this at all, we should go all the way > and allow foreign tables to be both inheritance parents and chil

Re: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2015-04-14 Thread Shigeru HANADA
KaiGai-san, 2015/04/14 14:04、Kouhei Kaigai のメール: > >> * Fix typos >> >> Please review the v11 patch, and mark it as “ready for committer” if it’s ok. >> > It's OK for me, and wants to be reviewed by other people to get it committed. > Thanks! >> In addition to essential features, I tried to