Re: [HACKERS] distinct estimate of a hard-coded VALUES list

2016-08-18 Thread Tom Lane
Jeff Janes writes: > So even though it knows that 6952 values have been shoved in the bottom, it > thinks only 200 are going to come out of the aggregation. This seems like > a really lousy estimate. In more complex queries than the example one > given it leads to poor planning choices. > Is th

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-08-18 Thread Claudio Freire
On Thu, Aug 18, 2016 at 6:15 PM, Peter Geoghegan wrote: > On Thu, Aug 18, 2016 at 1:41 PM, Claudio Freire > wrote: >> In fact, that's why non-leaf index tuples need a different format, >> because while leaf index tuples contain the heap pointer already, >> non-leaf ones contain only the downlink

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-08-18 Thread Tom Lane
Claudio Freire writes: > On Thu, Aug 18, 2016 at 6:04 PM, Kevin Grittner wrote: >> Speaking of performance side effects, does this avoid O(N^2) >> performance on index tuple insertion with duplicate values, for all >> insertion orderings? For example, does it descend directly to the >> right lea

Re: [HACKERS] [PATCH] add option to pg_dumpall to exclude tables from the dump

2016-08-18 Thread Jim Nasby
On 8/18/16 2:40 PM, Tom Lane wrote: This seems pretty dubious to me, in particular that the identical -T option will be passed willy-nilly into the pg_dump runs for every database. That seems more likely to be a foot-gun than something useful. I agree, but I think mandating a database name (wh

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-08-18 Thread Claudio Freire
On Thu, Aug 18, 2016 at 6:27 PM, Tom Lane wrote: > Claudio Freire writes: >> On Thu, Aug 18, 2016 at 6:04 PM, Kevin Grittner wrote: >>> Speaking of performance side effects, does this avoid O(N^2) >>> performance on index tuple insertion with duplicate values, for all >>> insertion orderings? F

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-08-18 Thread Peter Geoghegan
On Thu, Aug 18, 2016 at 2:26 PM, Claudio Freire wrote: > I see that. I could try to measure average depth to measure the impact > this had on fan-in. > > While it should cut it in half for narrow indexes, half of very high > is still high. Wide indexes, which are are the ones that would suffer > f

Re: [HACKERS] Curing plpgsql's memory leaks for statement-lifespan values

2016-08-18 Thread Jim Nasby
On 7/25/16 1:50 PM, Tom Lane wrote: There's a glibc-dependent hack in aset.c that reports any plpgsql-driven palloc or pfree against a context named "SPI Proc", as well as changes in pl_comp.c so that transient junk created during initial parsing of a plpgsql function body doesn't end up in the S

Re: [HACKERS] [PATCH] add option to pg_dumpall to exclude tables from the dump

2016-08-18 Thread Tom Lane
Jim Nasby writes: > On 8/18/16 2:40 PM, Tom Lane wrote: >> This seems pretty dubious to me, in particular that the identical -T >> option will be passed willy-nilly into the pg_dump runs for every >> database. That seems more likely to be a foot-gun than something useful. > I agree, but I think

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

2016-08-18 Thread Claudio Freire
On Thu, Aug 18, 2016 at 6:38 PM, Peter Geoghegan wrote: > On Thu, Aug 18, 2016 at 2:26 PM, Claudio Freire > wrote: >> I see that. I could try to measure average depth to measure the impact >> this had on fan-in. >> >> While it should cut it in half for narrow indexes, half of very high >> is sti

[HACKERS] errno clobbering in reorderbuffer

2016-08-18 Thread Alvaro Herrera
While researching a customer issue with BDR I noticed that one ereport() call happens after clobbering errno, leading to the wrong strerror being reported. This patch fixes it by saving before calling CloseTransientFile and restoring afterwards. I also threw in a missing errcode I noticed while l

Re: [HACKERS] pg_basebackup wish list

2016-08-18 Thread Peter Eisentraut
On 7/12/16 9:55 PM, Masahiko Sawada wrote: > And what I think is pg_baseback never remove the directory specified > by -D option even if execution is failed. initdb command behaves so. > I think it's helpful for backup operation. This has been bothering me as well. How about the attached patch as

Re: [HACKERS] drop src/backend/port/darwin/system.c ?

2016-08-18 Thread Peter Eisentraut
On 8/17/16 12:29 PM, Tom Lane wrote: > Also, the early releases of OS X were rough enough that it's pretty hard > to believe anyone is still using them anywhere (certainly the buildfarm > isn't). So the odds of anyone caring if we remove this file seem > negligible. Let's nuke it. done -- Pete

Re: [HACKERS] errno clobbering in reorderbuffer

2016-08-18 Thread Andres Freund
On 2016-08-18 19:06:02 -0300, Alvaro Herrera wrote: > While researching a customer issue with BDR I noticed that one ereport() > call happens after clobbering errno, leading to the wrong strerror being > reported. This patch fixes it by saving before calling > CloseTransientFile and restoring afte

Re: [HACKERS] errno clobbering in reorderbuffer

2016-08-18 Thread Tom Lane
Alvaro Herrera writes: > While researching a customer issue with BDR I noticed that one ereport() > call happens after clobbering errno, leading to the wrong strerror being > reported. This patch fixes it by saving before calling > CloseTransientFile and restoring afterwards. > I also threw in a

Re: [HACKERS] errno clobbering in reorderbuffer

2016-08-18 Thread Andres Freund
Hi, On 2016-08-18 19:06:02 -0300, Alvaro Herrera wrote: > if (write(fd, rb->outbuf, ondisk->size) != ondisk->size) > { > + int save_errno = errno; > + > CloseTransientFile(fd); > + errno = save_errno; > ereport(ERROR, >

Re: [HACKERS] anyelement -> anyrange

2016-08-18 Thread Corey Huinker
I'd be happy to roll your code into the extension, and make it marked more stable. On Thu, Aug 18, 2016 at 2:49 PM, Jim Nasby wrote: > On 8/18/16 1:06 PM, Corey Huinker wrote: > >> You might also find some gleanable gems in: >> https://github.com/moat/range_type_functions/blob/master/doc >> /ran

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-08-18 Thread Michael Paquier
On Fri, Aug 19, 2016 at 2:40 AM, Peter Geoghegan wrote: > Heroku began a selective roll-out of amcheck yesterday. amcheck > already found a bug in the PostGiS Geography B-Tree opclass: > [...] > I'll go report this to the PostGiS people. Cool. I have been honestly wondering about deploying this t

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-08-18 Thread Peter Geoghegan
On Thu, Aug 18, 2016 at 5:06 PM, Michael Paquier wrote: > Cool. I have been honestly wondering about deploying this tool as well > to allow some of the QE tests to perform live checks of btree indexes > as we use a bunch of them. I'd certainly welcome that. There are Debian packages available fro

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-18 Thread Andres Freund
On 2016-06-30 18:14:15 -0700, Peter Geoghegan wrote: > On Tue, Dec 15, 2015 at 10:04 AM, Andres Freund wrote: > > Took a while. But here we go. The attached version is a significantly > > revised version of my earlier patch. Notably I've pretty much entirely > > revised the code in _mdfd_getseg()

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-18 Thread Peter Geoghegan
On Thu, Aug 18, 2016 at 5:26 PM, Andres Freund wrote: > Rebased version attached. A review would be welcome. Plan to push this > forward otherwise in the not too far away future. I can review this next week. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-18 Thread Andres Freund
On 2016-08-18 17:27:59 -0700, Peter Geoghegan wrote: > On Thu, Aug 18, 2016 at 5:26 PM, Andres Freund wrote: > > Rebased version attached. A review would be welcome. Plan to push this > > forward otherwise in the not too far away future. > > I can review this next week. Thanks -- Sent via pgs

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-08-18 Thread Michael Paquier
On Fri, Aug 19, 2016 at 9:14 AM, Peter Geoghegan wrote: > On Thu, Aug 18, 2016 at 5:06 PM, Michael Paquier > wrote: >> Cool. I have been honestly wondering about deploying this tool as well >> to allow some of the QE tests to perform live checks of btree indexes >> as we use a bunch of them. > >

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-18 Thread Peter Geoghegan
On Thu, Aug 18, 2016 at 5:28 PM, Andres Freund wrote: >> I can review this next week. > > Thanks Given the time frame that you have in mind, I won't revisit the question the parallel CLUSTER CPU bottleneck issue until this is committed. The patch might change things enough that that would be a wa

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-18 Thread Andres Freund
On 2016-08-18 17:35:47 -0700, Peter Geoghegan wrote: > On Thu, Aug 18, 2016 at 5:28 PM, Andres Freund wrote: > >> I can review this next week. > > > > Thanks > > Given the time frame that you have in mind, I won't revisit the > question the parallel CLUSTER CPU bottleneck issue until this is > co

Re: [HACKERS] _mdfd_getseg can be expensive

2016-08-18 Thread Peter Geoghegan
On Thu, Aug 18, 2016 at 5:42 PM, Andres Freund wrote: > How large was the index & table in question? I mean this really only > comes into effect at 100+ segments. Not that big, but I see no reason to take the chance, I suppose. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql

Re: [HACKERS] WIP: About CMake v2

2016-08-18 Thread Michael Paquier
On Fri, Aug 19, 2016 at 4:25 AM, Tom Lane wrote: > BTW, I just noticed that cmake doesn't seem to be supplied as part of > Apple's dev tools, at least not up to current (El Capitan) releases. > That's going to be a rather large minus to be taken into account > whenever we make the go/no-go decisio

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-08-18 Thread Peter Geoghegan
On Thu, Aug 18, 2016 at 5:35 PM, Michael Paquier wrote: > This would be packaged from source in my case, but that's no big deal > :) At least I can see that it is added in the next CF, and that's > marked as ready for committer for a couple of months now... If you consider how the code is written

Re: [HACKERS] Fix comment in ATExecValidateConstraint

2016-08-18 Thread Amit Langote
On 2016/08/19 5:35, Robert Haas wrote: > On Thu, Aug 18, 2016 at 5:15 AM, Amit Langote > wrote: >> On 2016/07/25 17:18, Amit Langote wrote: >>> The comment seems to have been copied from ATExecAddColumn, which says: >>> >>> /* >>> * If we are told not to recurse, there had better not be any >>>

Re: [HACKERS] Most efficient way for libPQ .. PGresult serialization

2016-08-18 Thread Craig Ringer
On 19 August 2016 at 03:08, Joshua Bay wrote: > Thanks, > But I don't think my question was clear enough. > > I already managed the connection pooling, and what I need is to serialize > the result. > > If PGresult was a contiguous block, I could have just create buffer and > call memcpy for seria

Re: [HACKERS] [PATCH] bigint txids vs 'xid' type, new txid_recent(bigint) => xid

2016-08-18 Thread Craig Ringer
On 19 August 2016 at 02:35, Jim Nasby wrote: > On 8/18/16 5:46 AM, Amit Kapila wrote: > >> I think there is a value in exposing such a variant which takes bigint >> and internally converts it to xid. I am not sure the semantics for >> > > I think that's a bad idea because you have the exact same

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-18 Thread Michael Paquier
On Thu, Aug 18, 2016 at 6:12 PM, Heikki Linnakangas wrote: > On 06/22/2016 04:41 AM, Michael Paquier wrote: >> make s >> OK, there is not much that we can do here then. What about the rest? >> Those seem like legit concerns to me. > > > There's also a realloc() and an strdup() call in refint.c. Bu

Re: [HACKERS] errno clobbering in reorderbuffer

2016-08-18 Thread Alvaro Herrera
Andres Freund wrote: > On 2016-08-18 19:06:02 -0300, Alvaro Herrera wrote: > > While researching a customer issue with BDR I noticed that one ereport() > > call happens after clobbering errno, leading to the wrong strerror being > > reported. This patch fixes it by saving before calling > > CloseT

Re: [HACKERS] errno clobbering in reorderbuffer

2016-08-18 Thread Andres Freund
On August 18, 2016 7:21:03 PM PDT, Alvaro Herrera wrote: >Andres Freund wrote: >> On 2016-08-18 19:06:02 -0300, Alvaro Herrera wrote: >> > While researching a customer issue with BDR I noticed that one >ereport() >> > call happens after clobbering errno, leading to the wrong strerror >being >>

Re: [HACKERS] Bug in to_timestamp().

2016-08-18 Thread amul sul
On Friday, August 19, 2016 12:42 AM, Robert Haas wrote: >On Wed, Aug 17, 2016 at 10:35 AM, amul sul wrote: > > >> Hmm. I haven't really looked into the code, but with applying both patches >> it looks precisely imitate Oracle's behaviour. Thanks. > > >This is good to hear, but for us to conside

Re: [HACKERS] pg_basebackup wish list

2016-08-18 Thread Masahiko Sawada
On Fri, Aug 19, 2016 at 7:06 AM, Peter Eisentraut wrote: > On 7/12/16 9:55 PM, Masahiko Sawada wrote: >> And what I think is pg_baseback never remove the directory specified >> by -D option even if execution is failed. initdb command behaves so. >> I think it's helpful for backup operation. > > Th

Re: [HACKERS] Slowness of extended protocol

2016-08-18 Thread Tatsuo Ishii
BTW, there seem to be a room to enhance JDBC driver performance. In my understanding it always uses unnamed portal even if the SQL is like "BEGIN" or "COMMIT" (no parameters). They are too often used. Why not doing like this? Prepare(stmt=S1,query="BEGIN") Bind(stmt=S1,portal=P1) Execute(portal=P1

[HACKERS] Should we cacheline align PGXACT?

2016-08-18 Thread Alexander Korotkov
Hackers, originally this idea was proposed by Andres Freund while experimenting with lockfree Pin/UnpinBuffer [1]. The patch is attached as well as results of pgbench -S on 72-cores machine. As before it shows huge benefit in this case. For sure, we should validate that it doesn't cause performan

[HACKERS] Exporting more function in libpq

2016-08-18 Thread Tatsuo Ishii
I would like to proppse to export these functions in libpq. pqPutMsgStart pqPutMsgEnd pqPutc pqPuts pqPutInt pqPutnchar pqFlush pqHandleSendFailure I think this would be useful to create a tool/library which needs to handle frontend/backend protocol messages in detail. Best regards, -- Tatsuo Is

Re: [HACKERS] Slowness of extended protocol

2016-08-18 Thread Vladimir Sitnikov
Tatsuo>understanding it always uses unnamed portal even if the SQL is like "BEGIN" or "COMMIT" (no parameters). They are too often used. Why not doing like this? Does it actually work? The documentation says named portals last till the end of the transaction: https://www.postgresql.org/docs/9.5/

Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol

2016-08-18 Thread Michael Paquier
On Fri, Aug 19, 2016 at 1:51 AM, Heikki Linnakangas wrote: > On 08/18/2016 03:45 PM, Michael Paquier wrote: >> >> On Thu, Aug 18, 2016 at 9:28 PM, Heikki Linnakangas >> wrote: >> For the current ip.c, I don't have a better idea than putting in >> src/common/ip.c the set of routines used by both t

Re: [HACKERS] Slowness of extended protocol

2016-08-18 Thread Tatsuo Ishii
> Tatsuo>understanding it always uses unnamed portal even if the SQL is like > "BEGIN" or "COMMIT" (no parameters). They are too often used. Why not > doing like this? > > Does it actually work? > > The documentation says named portals last till the end of the transaction: > > https://www.postgr

<    1   2