Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Pavel Stehule
2017-01-24 21:38 GMT+01:00 Alvaro Herrera : > Pavel Stehule wrote: > > > * SELECT (xmltable(..)).* + regress tests > > * compilation and regress tests without --with-libxml > > Thanks. I just realized that this is doing more work than necessary -- > ?? I don't understand? > I think it would be

Re: [HACKERS] pgbench more operators & functions

2017-01-25 Thread Fabien COELHO
As it stands right now you haven't provided enough context for this patch and only the social difficulty of actually marking a patch rejected has prevented its demise in its current form - because while it has interesting ideas its added maintenance burden for -core without any in-core usage. Bu

Re: [HACKERS] pgbench more operators & functions

2017-01-25 Thread Fabien COELHO
mail addresses> Hello Tom, I concur that this is expanding pgbench's expression language well beyond what anybody has shown a need for. As for the motivation, I'm assuming that pgbench should provide features necessary to implement benchmarks, so I'm adding operators that appear in stand

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-25 Thread Tobias Oberstein
Hi Alvaro, Am 24.01.2017 um 19:36 schrieb Alvaro Herrera: Tobias Oberstein wrote: I am benchmarking IOPS, and while doing so, it becomes apparent that at these scales it does matter _how_ IO is done. The most efficient way is libaio. I get 9.7 million/sec IOPS with low CPU load. Using any syn

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-25 Thread Tobias Oberstein
Hi Andres, Using pread instead of lseek+read halfes the syscalls. I really don't understand what you are fighting here .. Sure, there's some overhead. And as I said upthread, I'm much less against this change than Tom. What I'm saying is that your benchmarks haven't shown a benefit in a mean

Re: [HACKERS] Radix tree for character conversion

2017-01-25 Thread Ishii Ayumi
HI, I patched 4 patchset and run "make", but I got failed. Is this a bug or my mistake ? I'm sorry if I'm wrong. [$(TOP)]$ patch -p1 < ../0001-Add-missing-semicolon.patch [$(TOP)]$ patch -p1 < ../0002-Correct-reference-resolution-syntax.patch [$(TOP)]$ patch -p1 < ../0003-Apply-pgperltidy-on-src-

Re: [HACKERS] Push down more UPDATEs/DELETEs in postgres_fdw

2017-01-25 Thread Etsuro Fujita
On 2016/11/30 17:29, Etsuro Fujita wrote: On 2016/11/23 20:28, Rushabh Lathia wrote: I wrote: How about inserting that before the out param **retrieved_attrs, like this? static void deparseExplicitTargetList(List *tlist, bool is_returning,

Re: [HACKERS] pgbench more operators & functions

2017-01-25 Thread Fabien COELHO
Bonjour Michaƫl, Hello Robert, Let's mark this Returned with Feedback and move on. We've only got a week left in the CommitFest anyhow and there are lots of other things that still need work (and which actually have been revised to match previous feedback). Done as such, let's move on. Hmm

Re: [HACKERS] simplify sequence test

2017-01-25 Thread Petr Jelinek
On 25/01/17 03:48, Peter Eisentraut wrote: > We maintain a separate test output file sequence_1.out because the > log_cnt value can vary if there is a checkpoint happening at the right > time. So we have to maintain two files because of a one character > difference. I propose the attached patch t

Re: [HACKERS] Assignment of valid collation for SET operations on queries with UNKNOWN types.

2017-01-25 Thread Ashutosh Bapat
On Wed, Jan 25, 2017 at 10:54 AM, Michael Paquier wrote: > On Wed, Jan 25, 2017 at 12:46 AM, Tom Lane wrote: >> I wrote: >>> Michael Paquier writes: The table of Pseudo-Types needs to be updated as well with unknown in datatype.sgml. >> >>> Check. >> >> Here's an updated patch with doc

Re: [HACKERS] Floating point comparison inconsistencies of the geometric types

2017-01-25 Thread Emre Hasegeli
I am responding both of your emails together. > Perhaps I don't understand it. Many opclass are defined for > btree. But since (ordinary) btree handles only one-dimentional, > totally-orderable sets, geometric types even point don't fit > it. Even if we relaxed it by removing EPSILON, multidimenti

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Amit Kapila
On Tue, Jan 24, 2017 at 5:07 AM, Jim Nasby wrote: > I took a look at this again, and it doesn't appear to be working for me. The > library is being loaded during startup, but I don't see any further activity > in the log, and I don't see an autoprewarm file in $PGDATA. > > There needs to be some

Re: [HACKERS] Substantial bloat in postgres_fdw regression test runtime

2017-01-25 Thread Ashutosh Bapat
On Thu, Nov 3, 2016 at 1:58 PM, Jeevan Chalke wrote: > > On Wed, Nov 2, 2016 at 10:09 PM, Tom Lane wrote: >> >> In 9.6, "make installcheck" in contrib/postgres_fdw takes a shade >> under 3 seconds on my machine. In HEAD, it's taking 10 seconds. >> I am not happy, especially not since there's no

Re: [HACKERS] Improvements in psql hooks for variables

2017-01-25 Thread Rahila Syed
Hello, The patch works fine on applying on latest master branch and testing it for various variables as listed in PsqlSettings struct. I will post further comments on patch soon. Thank you, Rahila Syed On Wed, Jan 25, 2017 at 1:33 AM, Tom Lane wrote: > "Daniel Verite" writes: > > Here's an up

Re: [HACKERS] macaddr 64 bit (EUI-64) datatype support

2017-01-25 Thread Haribabu Kommi
On Wed, Jan 25, 2017 at 6:43 PM, Vitaly Burovoy wrote: > On 1/23/17, Haribabu Kommi wrote: > > The patch is split into two parts. > > 1. Macaddr8 datatype support > > 2. Contrib module support. > > Hello, > > I'm sorry for the delay. > The patch is almost done, but I have two requests since the

Re: [HACKERS] multivariate statistics (v19)

2017-01-25 Thread Alvaro Herrera
Michael Paquier wrote: > On Wed, Jan 4, 2017 at 11:35 AM, Tomas Vondra > wrote: > > Attached is v22 of the patch series, rebased to current master and fixing > > the reported bug. I haven't made any other changes - the issues reported by > > Petr are mostly minor, so I've decided to wait a bit mo

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Robert Haas
On Mon, Jan 23, 2017 at 6:37 PM, Jim Nasby wrote: > I'm not sure the default GUC setting of 0 makes sense. If you've loaded the > module, presumably you want it to be running. I think it'd be nice if the GUC > had a -1 setting that meant to use checkpoint_timeout. Actually, I think we need to u

Re: [HACKERS] Substantial bloat in postgres_fdw regression test runtime

2017-01-25 Thread Tom Lane
Ashutosh Bapat writes: > On Thu, Nov 3, 2016 at 1:58 PM, Jeevan Chalke > wrote: >> Attached patch with test-case modification. > I verified that this patch indeed bring the time down to 2 to 3 > seconds from 10 seconds. Thanks for working on this, guys. > The additional condition t2.c2 = 6 see

Re: [HACKERS] COPY as a set returning function

2017-01-25 Thread David Fetter
On Wed, Jan 25, 2017 at 02:37:57PM +0900, Michael Paquier wrote: > On Mon, Dec 5, 2016 at 2:10 PM, Haribabu Kommi > wrote: > > On Tue, Nov 1, 2016 at 7:45 AM, Corey Huinker > > wrote: > >> > >> Attached is a patch that implements copy_srf(). > > > > Moved to next CF with "needs review" status. >

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Alvaro Herrera
Tom Lane wrote: > Andres Freund writes: > > On 2017-01-24 21:32:56 -0300, Alvaro Herrera wrote: > >> XMLTABLE is specified by the standard to return multiple rows ... but > >> then as far as my reading goes, it is only supposed to be supported in > >> the range table (FROM clause) not in the targe

Re: [HACKERS] COPY as a set returning function

2017-01-25 Thread David Fetter
On Mon, Oct 31, 2016 at 04:45:40PM -0400, Corey Huinker wrote: > Attached is a patch that implements copy_srf(). > > The function signature reflects cstate more than it reflects the COPY > options (filename+is_program instead of FILENAME or PROGRAM, etc) The patch as it stands needs a rebase. I

Re: [HACKERS] Assignment of valid collation for SET operations on queries with UNKNOWN types.

2017-01-25 Thread Tom Lane
Ashutosh Bapat writes: > On Wed, Jan 25, 2017 at 10:54 AM, Michael Paquier > wrote: >> On Wed, Jan 25, 2017 at 12:46 AM, Tom Lane wrote: >>> Here's an updated patch with doc changes. Aside from that one, >>> I tried to spell "pseudo-type" consistently, and I changed one >>> place where we were

Re: [HACKERS] PostgreSQL 8.2 installation error on Windows 2016 server

2017-01-25 Thread Shruti Rawal
Hi Aaron, Thank you so much for the information. Regards, Shruti Rawal -Original Message- From: Aaron W. Swenson [mailto:titanof...@gentoo.org] Sent: Tuesday, January 24, 2017 8:22 PM To: Shruti Rawal Cc: pgsql-hackers@postgresql.org Subject: Re: [HACKERS] PostgreSQL 8.2 installation err

Re: [HACKERS] pgbench more operators & functions

2017-01-25 Thread Fabien COELHO
Hello Tom, I concur that this is expanding pgbench's expression language well beyond what anybody has shown a need for. As for the motivation, I'm assuming that pgbench should provide features necessary to implement benchmarks, so I'm adding operators that appear in standard benchmark speci

Re: [HACKERS] sequence data type

2017-01-25 Thread Peter Eisentraut
Here is an updated patch that allows changing the sequence type. This was clearly a concern for reviewers, and the presented use cases seemed convincing. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 5aff9

Re: [HACKERS] Speedup twophase transactions

2017-01-25 Thread Nikhil Sontakke
> We are talking about the recovery/promote code path. Specifically this > call to KnownPreparedRecreateFiles() in PrescanPreparedTransactions(). > > We write the files to disk and they get immediately read up in the > following code. We could not write the files to disk and read > KnownPreparedLis

[HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Robert Haas
Commit 1574783b4ced0356fbc626af1a1a469faa6b41e1 gratifyingly removed hard-coded superuser checks from assorted functions, which makes it possible to GRANT EXECUTE ON FUNCTION pg_catalog.whatever() to unprivileged users if the DBA so desires. However, the functions in genfile.c still have hard-code

Re: [HACKERS] PoC: Grouped base relation

2017-01-25 Thread Antonin Houska
David Rowley wrote: > On 20 January 2017 at 00:22, Antonin Houska wrote: > > Sorry, it was my thinko - I somehow confused David's CROSS JOIN example with > > this one. If one side of the join clause is unique and the other becomes > > unique due to aggregation (and if parallel processing is not

Re: [HACKERS] Performance improvement for joins where outer side is unique

2017-01-25 Thread Antonin Houska
David Rowley wrote: > On 19 January 2017 at 11:06, David Rowley > wrote: > > Old patch no longer applies, so I've attached a rebased patch. This > > also re-adds a comment line which I mistakenly removed. > > (meanwhile Andres commits 69f4b9c) > > I should've waited a bit longer. > > Here's a

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Greg Stark
I tend to agree. But in the past when this came up people pointed out you could equally do things this way and still grant all the access you wanted using SECURITY DEFINER. Arguably that's a better approach because then instead of auditing the entire monitor script you only need to audit this one w

Re: [HACKERS] COPY as a set returning function

2017-01-25 Thread David Fetter
On Wed, Jan 25, 2017 at 06:16:16AM -0800, David Fetter wrote: > On Mon, Oct 31, 2016 at 04:45:40PM -0400, Corey Huinker wrote: > > Attached is a patch that implements copy_srf(). > > > > The function signature reflects cstate more than it reflects the COPY > > options (filename+is_program instead

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-01-25 Thread Alvaro Herrera
Reading 0001_track_root_lp_v9.patch again: > +/* > + * We use the same HEAP_LATEST_TUPLE flag to check if the tuple's t_ctid > field > + * contains the root line pointer. We can't use the same > + * HeapTupleHeaderIsHeapLatest macro because that also checks for > TID-equality > + * to decide whe

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 11:30 AM, Greg Stark wrote: > I tend to agree. But in the past when this came up people pointed out > you could equally do things this way and still grant all the access > you wanted using SECURITY DEFINER. Arguably that's a better approach > because then instead of auditin

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2017-01-25 Thread Masahiko Sawada
On Tue, Jan 24, 2017 at 1:49 AM, Claudio Freire wrote: > On Fri, Jan 20, 2017 at 6:24 AM, Masahiko Sawada > wrote: >> On Thu, Jan 19, 2017 at 8:31 PM, Claudio Freire >> wrote: >>> On Thu, Jan 19, 2017 at 6:33 AM, Anastasia Lubennikova >>> wrote: 28.12.2016 23:43, Claudio Freire: >>>

Re: [HACKERS] COPY as a set returning function

2017-01-25 Thread Alvaro Herrera
David Fetter wrote: > @@ -562,7 +563,6 @@ CopyGetData(CopyState cstate, void *databuf, int minread, > int maxread) >errmsg("could not read from > COPY file: %m"))); > break; > case COPY_OLD_FE: > - >

Re: [HACKERS] Logical Replication WIP

2017-01-25 Thread Peter Eisentraut
On 1/22/17 8:11 PM, Petr Jelinek wrote: > 0001 - Changes the libpqrcv_connect to use async libpq api so that it > won't get stuck forever in case of connect is stuck. This is preexisting > bug that also affects walreceiver but it's less visible there as there > is no SQL interface to initiate conne

Re: [HACKERS] COPY as a set returning function

2017-01-25 Thread Corey Huinker
On Wed, Jan 25, 2017 at 11:57 AM, Alvaro Herrera wrote: > David Fetter wrote: > > > @@ -562,7 +563,6 @@ CopyGetData(CopyState cstate, void *databuf, int > minread, int maxread) > >errmsg("could not read > from COPY file: %m"))); > >

Re: [HACKERS] pg_hba_file_settings view patch

2017-01-25 Thread Tom Lane
Ashutosh Bapat writes: > On Wed, Jan 25, 2017 at 9:58 AM, Haribabu Kommi > wrote: >> All the ereport messages of level are LOG, because of this reason, because >> of this reason even if we use the TRY/CATCH, it doesn't work. As the >> messages gets printed to the logfile and continue to process

Re: [HACKERS] Logical Replication WIP

2017-01-25 Thread Peter Eisentraut
On 1/23/17 11:19 AM, Fujii Masao wrote: > The copyright in each file that the commit of logical rep added needs to > be updated. I have fixed that. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pg

[HACKERS] [FEATURE PATCH] pg_stat_statements with plans

2017-01-25 Thread Julian Markwort
Hello psql-hackers! TL:DR; We extended the functionality of pg_stat_statements so it can track worst and best case execution plans. Based on a suggestion of my colleague Arne Scheffer, Marius Timmer and I extended pg_stat_statements so it can also record execution plans, whenever the executi

Re: [HACKERS] [FEATURE PATCH] pg_stat_statements with plans

2017-01-25 Thread Simon Riggs
On 25 January 2017 at 17:34, Julian Markwort wrote: > Analogous to this, a bad_plan is saved, when the time has been exceeded by a > factor greater than 1.1 . ...and the plan differs? Probably best to use some stat math to calculate deviation, rather than fixed %. Sounds good. -- Simon Riggs

Re: [HACKERS] logical-replication.sgml improvements

2017-01-25 Thread Peter Eisentraut
On 1/20/17 11:00 AM, Erik Rijkers wrote: > logical-replication.sgml.diff changes Committed, thanks! -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

Re: [HACKERS] pdf versus single-html

2017-01-25 Thread Peter Eisentraut
On 1/21/17 6:29 AM, Erik Rijkers wrote: > It might even be good to include such a single-file html in the Makefile > as an option. > > I'll give it a try but has anyone done this work already, perhaps? Already exists: doc/src/sgml$ make postgres.html -- Peter Eisentraut http://ww

Re: [HACKERS] Declarative partitioning vs. information_schema

2017-01-25 Thread Peter Eisentraut
On 1/18/17 2:32 PM, Robert Haas wrote: > Unless we can find something official, I suppose we should just > display BASE TABLE in that case as we do in other cases. I wonder if > the schema needs some broader revision; for example, are there > information_schema elements intended to show informatio

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Robert Haas
On Sat, Jan 21, 2017 at 11:57 AM, Andres Freund wrote: > On 2017-01-21 11:39:18 +0100, Magnus Hagander wrote: >> Is it time to enable checksums by default, and give initdb a switch to turn >> it off instead? > > -1 - the WAL overhead is quite massive, and in contrast to the other > GUCs recently c

Re: [HACKERS] COPY as a set returning function

2017-01-25 Thread David Fetter
On Wed, Jan 25, 2017 at 12:23:23PM -0500, Corey Huinker wrote: > > Feel free to mark it returned as feedback. The concept didn't > generate as much enthusiasm as I had hoped, so I think the right > thing to do now is scale it back to a patch that makes > CopyFromRawFields() externally visible so t

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 12:02 AM, Jim Nasby wrote: > I'm not completely grokking your second paragraph, but I would think that an > average user would love got get a heads-up that their hardware is failing. Sure. If the database runs fast enough with checksums enabled, there's basically no reaso

Re: [HACKERS] Declarative partitioning vs. information_schema

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 1:04 PM, Peter Eisentraut wrote: > On 1/18/17 2:32 PM, Robert Haas wrote: >> Unless we can find something official, I suppose we should just >> display BASE TABLE in that case as we do in other cases. I wonder if >> the schema needs some broader revision; for example, are

Re: [HACKERS] COPY as a set returning function

2017-01-25 Thread Corey Huinker
On Wed, Jan 25, 2017 at 1:10 PM, David Fetter wrote: > On Wed, Jan 25, 2017 at 12:23:23PM -0500, Corey Huinker wrote: > > > > Feel free to mark it returned as feedback. The concept didn't > > generate as much enthusiasm as I had hoped, so I think the right > > thing to do now is scale it back to

Re: [HACKERS] PATCH: recursive json_populate_record()

2017-01-25 Thread Tom Lane
Nikita Glukhov writes: > On 22.01.2017 21:58, Tom Lane wrote: >> If you want such macros I think it would be better to submit a separate >> cosmetic patch that tries to hide such bit-tests behind macros throughout >> the jsonb code. > I've attached that patch, but not all the bit-tests were hidde

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Peter Geoghegan
On Wed, Jan 25, 2017 at 10:18 AM, Robert Haas wrote: > Trying to force those people to use checksums is just masterminding; > they've made their own decision that it's not worth bothering with. > When something goes wrong, WE still care about distinguishing hardware > failure from PostgreSQL failu

Re: [HACKERS] COPY as a set returning function

2017-01-25 Thread Corey Huinker
> > > + /* param 7: escape text default null, -- defaults to whatever > quote is */ > > + if (PG_ARGISNULL(7)) > > + { > > + copy_state.escape = copy_state.quote; > > + } > > + else > > + { > > + if (copy_state.csv_mode) > > + { > > +

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 1:37 PM, Peter Geoghegan wrote: > On Wed, Jan 25, 2017 at 10:18 AM, Robert Haas wrote: >> Trying to force those people to use checksums is just masterminding; >> they've made their own decision that it's not worth bothering with. >> When something goes wrong, WE still care

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > The use case I have in mind is > a monitoring tool that needs access to one more of those functions -- > in keeping with the principle of least privilege, it's much better to > give the monitoring user only the privileges which it actually nee

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Greg, * Greg Stark (st...@mit.edu) wrote: > I tend to agree. But in the past when this came up people pointed out > you could equally do things this way and still grant all the access > you wanted using SECURITY DEFINER. Arguably that's a better approach > because then instead of auditing the enti

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > Also, the same argument could be made about removing the built-in > superuser check from ANY function, and we've already rejected that > argument for a bunch of other functions. If we say that argument is > valid for some functions but not ot

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 2:08 PM, Stephen Frost wrote: > That isn't what you're doing with those functions though, you're giving > the monitoring tool superuser-level rights but trying to pretend like > you're not. Uh, how so? None of those functions can be used to escalate to superuser privilege

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 2:13 PM, Stephen Frost wrote: > I went over *every* superuser check in the system when I did that work, > wrote up a long email about why I made the decisions that I did, posted > it here, had follow-on discussions, all of which lead to the patch which > ended up going in.

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 12:02 AM, Jim Nasby wrote: > > I'm not completely grokking your second paragraph, but I would think that an > > average user would love got get a heads-up that their hardware is failing. > > Sure. If the database run

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
* Peter Geoghegan (p...@heroku.com) wrote: > On Wed, Jan 25, 2017 at 10:18 AM, Robert Haas wrote: > > Trying to force those people to use checksums is just masterminding; > > they've made their own decision that it's not worth bothering with. > > When something goes wrong, WE still care about dist

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Tom Lane
Stephen Frost writes: > Would you say that most user's databases run fast enough with checksums > enabled? Or more than most, maybe 70%? 80%? In today's environment, > I'd probably say that it's more like 90+%. It would be nice if there were some actual evidence about this, rather than numbers

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Jim Nasby
On 1/24/17 11:13 PM, Beena Emerson wrote: On Wed, Jan 25, 2017 at 10:36 AM, Jim Nasby mailto:jim.na...@bluetreble.com>> wrote: On 1/24/17 2:26 AM, Mithun Cy wrote: Thanks for looking into this patch, I just downloaded the patch and applied same to the latest code, I can see

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Jim Nasby
On 1/25/17 1:46 PM, Jim Nasby wrote: Based on that and other feedback I'm going to mark this as returned with feedback, though if you're able to get a revised patch in the next few days please do. Actually, based on the message that popped up when I went to do that I guess it's better not to d

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-25 Thread Andres Freund
Hi, On 2017-01-25 10:16:32 +0100, Tobias Oberstein wrote: > > > Using pread instead of lseek+read halfes the syscalls. > > > > > > I really don't understand what you are fighting here .. > > > > Sure, there's some overhead. And as I said upthread, I'm much less > > against this change than Tom.

[HACKERS] Re: [BUGS] Problem in using pgbench's --connect(-C) and --rate=rate(-R rate) options together.

2017-01-25 Thread Fabien COELHO
Repost from bugs. -- Fabien. -- Forwarded message -- Date: Wed, 25 Jan 2017 18:59:45 +0100 (CET) From: Fabien COELHO To: nuko yokohama Cc: PostgreSQL Bugs List Subject: Re: [BUGS] Problem in using pgbench's --connect(-C) and --rate=rate(-R rate) options together. It ope

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 2:13 PM, Stephen Frost wrote: > > I went over *every* superuser check in the system when I did that work, > > wrote up a long email about why I made the decisions that I did, posted > > it here, had follow-on discussio

Re: [HACKERS] PoC plpgsql - possibility to force custom or generic plan

2017-01-25 Thread Jim Nasby
On 1/23/17 11:38 PM, Pavel Stehule wrote: Instead of paralleling all the existing namespace stuff, I wonder if it'd be better to create explicit block infrastructure. AFAIK PRAGMAs are going to have a lot of the same requirements (certainly the nesting is the same), and we might

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2017-01-25 Thread Claudio Freire
On Wed, Jan 25, 2017 at 1:54 PM, Masahiko Sawada wrote: > Thank you for updating the patch! > > + /* > +* Quickly rule out by lower bound (should happen a lot) Upper bound > was > +* already checked by segment search > +*/ > + if (vac_cmp_itemptr((void *) itemp

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Joshua D. Drake
On 01/25/2017 11:41 AM, Tom Lane wrote: Stephen Frost writes: Would you say that most user's databases run fast enough with checksums enabled? Or more than most, maybe 70%? 80%? In today's environment, I'd probably say that it's more like 90+%. It would be nice if there were some actual ev

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 2:08 PM, Stephen Frost wrote: > > That isn't what you're doing with those functions though, you're giving > > the monitoring tool superuser-level rights but trying to pretend like > > you're not. > > Uh, how so? None

[HACKERS] Should buffer of initialization fork have a BM_PERMANENT flag

2017-01-25 Thread Wang Hao
An unlogged table has an initialization fork. The initialization fork does not have an BM_PERMANENT flag when get a buffer. In checkpoint (not shutdown or end of recovery), it will not write to disk. after a crash recovery, the page of initialization fork will not correctly, then make the main fork

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Ants Aasma
On Wed, Jan 25, 2017 at 8:18 PM, Robert Haas wrote: > Also, it's not as if there are no other ways of checking whether your > disks are failing. SMART, for example, is supposed to tell you about > incipient hardware failures before PostgreSQL ever sees a bit flip. > Surely an average user would l

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 2:23 PM, Stephen Frost wrote: >> Sure. If the database runs fast enough with checksums enabled, >> there's basically no reason to have them turned off. The issue is >> when it doesn't. > > I don't believe we're talking about forcing every user to have checksums > enabled.

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Andres Freund
Hi, On 2017-01-25 05:45:24 +0100, Pavel Stehule wrote: > 2017-01-25 1:35 GMT+01:00 Andres Freund : > > > On 2017-01-24 21:32:56 -0300, Alvaro Herrera wrote: > > > Andres Freund wrote: > > > > Hi, > > > > > > > > On 2017-01-24 17:38:49 -0300, Alvaro Herrera wrote: > > > > > +static Datum ExecEvalT

Re: [HACKERS] PATCH: recursive json_populate_record()

2017-01-25 Thread Tom Lane
Nikita Glukhov writes: > On 22.01.2017 21:58, Tom Lane wrote: >> 5. The business with having some arguments that are only for json and >> others that are only for jsonb, eg in populate_scalar(), also makes me >> itch. > I've refactored json/jsonb passing using new struct JsValue which contains >

Re: [HACKERS] PoC plpgsql - possibility to force custom or generic plan

2017-01-25 Thread Pavel Stehule
2017-01-25 21:06 GMT+01:00 Jim Nasby : > On 1/23/17 11:38 PM, Pavel Stehule wrote: > >> >> Instead of paralleling all the existing namespace stuff, I wonder if >> it'd be better to create explicit block infrastructure. AFAIK >> PRAGMAs are going to have a lot of the same requirements (

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-01-25 Thread Alvaro Herrera
Looking at your 0002 patch now. It no longer applies, but the conflicts are trivial to fix. Please rebase and resubmit. I think the way WARM works has been pretty well hammered by now, other than the CREATE INDEX CONCURRENTLY issues, so I'm looking at the code from a maintainability point of vie

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 3:17 PM, Stephen Frost wrote: > Your email is 'pg_ls_dir & friends', which I took to imply at *least* > pg_read_file() and pg_read_binary_file(), and it's not unreasonable to > think you may have meant everything in adminpack, which also includes > pg_file_write(), pg_file_

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-01-25 Thread Robert Haas
On Wed, Jan 25, 2017 at 2:46 PM, Jim Nasby wrote: > I think the two need to be integrated much better than they are right now. > They should certainly be in the same .so, and as others have mentioned the > docs need to be fixed. For consistency, I think the name should just be > pg_prewarm, as wel

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Peter Geoghegan
On Wed, Jan 25, 2017 at 12:23 PM, Robert Haas wrote: > Also, I think that one of the big problems with the way checksums work > is that you don't find problems with your archived data until it's too > late. Suppose that in February bits get flipped in a block. You > don't access the data until J

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-25 Thread Tobias Oberstein
Hi, Synthetic PG workload or real world production workload? Both might work, production-like has bigger pull, but I'd guess synthetic is good enough. Thanks! The box should get PostgreSQL in the not too distant future. It'll get a backup from prod, but will act as new prod, so it might tak

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Pavel Stehule
> > > > > > > If you plan to hold support SRFin target list, then nothing is different. > > In last patch is executed under nodeProjectSet. > > It is, because we suddenly need to call different functions - and I'm > revamping most of execQual to have an opcode dispatch based execution > model (whic

Re: [HACKERS] PATCH: recursive json_populate_record()

2017-01-25 Thread Nikita Glukhov
On 25.01.2017 23:58, Tom Lane wrote: I think you need to take a second look at the code you're producing and realize that it's not so clean either. This extract from populate_record_field, for example, is pretty horrid: But what if we introduce some helper macros like this: #define JsValueIsN

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Andres Freund
Hi, > > > I'll try to explain my motivation. Please, check it and correct me if I > > am > > > wrong. I don't keep on my implementation - just try to implement XMLTABLE > > > be consistent with another behave and be used all time without any > > > surprise. > > > > > > 1. Any function that produce

Re: [HACKERS] multivariate statistics (v19)

2017-01-25 Thread Michael Paquier
On Wed, Jan 25, 2017 at 9:56 PM, Alvaro Herrera wrote: > Michael Paquier wrote: >> And nothing has happened since. Are there people willing to review >> this patch and help it proceed? > > I am going to grab this patch as committer. Thanks, that's good to know. -- Michael -- Sent via pgsql-ha

Re: [HACKERS] PATCH: recursive json_populate_record()

2017-01-25 Thread Tom Lane
Nikita Glukhov writes: > On 25.01.2017 23:58, Tom Lane wrote: >> I think you need to take a second look at the code you're producing >> and realize that it's not so clean either. This extract from >> populate_record_field, for example, is pretty horrid: > But what if we introduce some helper mac

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Pavel Stehule
2017-01-25 22:40 GMT+01:00 Andres Freund : > Hi, > > > > > I'll try to explain my motivation. Please, check it and correct me > if I > > > am > > > > wrong. I don't keep on my implementation - just try to implement > XMLTABLE > > > > be consistent with another behave and be used all time without a

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 3:17 PM, Stephen Frost wrote: > > Your email is 'pg_ls_dir & friends', which I took to imply at *least* > > pg_read_file() and pg_read_binary_file(), and it's not unreasonable to > > think you may have meant everything

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Andres Freund
On 2017-01-25 22:51:37 +0100, Pavel Stehule wrote: > 2017-01-25 22:40 GMT+01:00 Andres Freund : > > > I afraid when I cannot to reuse a SRF infrastructure, I have to > > reimplement > > > it partially :( - mainly for usage in "ROWS FROM ()" > > > > The TableExpr implementation is based on SRF now.

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Andres Freund
Hi, On 2017-01-25 16:52:38 -0500, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: > > Preventing people from calling functions by denying the ability to > > meaningfully GRANT EXECUTE on those functions doesn't actually stop > > them from delegating those functions to non-super

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Pavel Stehule
2017-01-25 23:33 GMT+01:00 Andres Freund : > On 2017-01-25 22:51:37 +0100, Pavel Stehule wrote: > > 2017-01-25 22:40 GMT+01:00 Andres Freund : > > > > I afraid when I cannot to reuse a SRF infrastructure, I have to > > > reimplement > > > > it partially :( - mainly for usage in "ROWS FROM ()" > >

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-01-25 Thread Tom Lane
Peter Geoghegan writes: > I should have already specifically pointed out that the original > discussion on what became 0002-* is here: > postgr.es/m/7256.1476711...@sss.pgh.pa.us > As I said already, the general idea seems uncontroversial. I looked at the 0002 patch, and while the code is probabl

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Stephen Frost
Andres, * Andres Freund (and...@anarazel.de) wrote: > On 2017-01-25 16:52:38 -0500, Stephen Frost wrote: > > * Robert Haas (robertmh...@gmail.com) wrote: > > > Preventing people from calling functions by denying the ability to > > > meaningfully GRANT EXECUTE on those functions doesn't actually st

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-01-25 Thread Peter Geoghegan
On Wed, Jan 25, 2017 at 2:49 PM, Tom Lane wrote: > I looked at the 0002 patch, and while the code is probably OK, I am > dissatisfied with this API spec: > > + * If copy is TRUE, the slot receives a copied tuple that will stay valid > + * regardless of future manipulations of the tuplesort's state

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-01-25 Thread Tom Lane
Peter Geoghegan writes: > It means "another call to tuplesort_gettupleslot", but I believe that > it would be safer (more future-proof) to actually specify "the slot > contents may be invalidated by any subsequent manipulation of the > tuplesort's state" instead. WFM. >> There are several other

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-01-25 Thread Peter Geoghegan
On Wed, Jan 25, 2017 at 3:11 PM, Tom Lane wrote: > Please. You might want to hit the existing ones with a separate patch, > but it doesn't much matter; I'd be just as happy with a patch that did > both things. Got it. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Andres Freund
On 2017-01-25 18:04:09 -0500, Stephen Frost wrote: > Andres, > > * Andres Freund (and...@anarazel.de) wrote: > > On 2017-01-25 16:52:38 -0500, Stephen Frost wrote: > > > * Robert Haas (robertmh...@gmail.com) wrote: > > > > Preventing people from calling functions by denying the ability to > > > >

Re: [HACKERS] tuplesort_gettuple_common() and *should_free argument

2017-01-25 Thread Tom Lane
[ in the service of closing out this thread... ] Peter Geoghegan writes: > Finally, 0003-* is a Valgrind suppression borrowed from my parallel > CREATE INDEX patch. It's self-explanatory. Um, I didn't find it all that self-explanatory. Why wouldn't we want to avoid writing undefined data? I th

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 25, 2017 at 2:23 PM, Stephen Frost wrote: > > Yet, our default is to have them disabled and *really* hard to enable. > > First of all, that could be fixed by further development. I'm certainly all for doing so, but I don't agree

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Peter Geoghegan
On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: > As it is, there are backup solutions which *do* check the checksum when > backing up PG. This is no longer, thankfully, some hypothetical thing, > but something which really exists and will hopefully keep users from > losing data. Wouldn't

  1   2   >