Re: [HACKERS] UPDATE of partition key

2018-01-24 Thread Robert Haas
On Mon, Jan 22, 2018 at 9:46 AM, Tom Lane wrote: > Robert Haas writes: >> Tom, do you want to double-check that this fixes it for you? > > I can confirm that a valgrind run succeeded for me with the patch > in place. Committed. Sorry for the delay. -- Robert Haas EnterpriseDB: http://www.ente

Re: documentation is now XML

2018-01-24 Thread Peter Eisentraut
On 1/24/18 09:45, Bruce Momjian wrote: > So we are not > using TeX anymore for PG11+ docs? as of PG10 -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Possible performance regression with pg_dump of a large number of relations

2018-01-24 Thread Luke Cowell
Hi Stephen, thank you for putting this together. > If folks get a chance to take a look at the query and/or test, that'd be > great. I'll try to work up an actual patch to pg_dump this weekend to > run it through the regression tests and see if anything breaks. I'm not sure how I can help other

Re: [HACKERS] UPDATE of partition key

2018-01-24 Thread Thomas Munro
On Thu, Jan 25, 2018 at 10:39 AM, Robert Haas wrote: > On Mon, Jan 22, 2018 at 9:46 AM, Tom Lane wrote: >> Robert Haas writes: >>> Tom, do you want to double-check that this fixes it for you? >> >> I can confirm that a valgrind run succeeded for me with the patch >> in place. > > Committed. Sor

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Andres Freund
Hi, On 2018-01-24 22:35:08 +0100, Pierre Ducroquet wrote: > I tried to build on Debian sid, using GCC 7 and LLVM 5. I used the following > to compile, using your branch @3195c2821d : Thanks! > $ export LLVM_CONFIG=/usr/bin/llvm-config-5.0 > $ ./configure --with-llvm > $ make > > And I had the

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Andres Freund
Hi, On 2018-01-24 14:06:30 -0800, Andres Freund wrote: > > In LLVM 5.0, it looks like DebugInfo.h is not available in llvm-c, only as > > a C > > ++ API in llvm/IR/DebugInfo.h. > > Hm, I compiled against 5.0 quite recently, but added the stripping of > debuginfo lateron. I'll add a fallback met

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2018-01-24 Thread Stephen Frost
Tom, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> Fair point, but doesn't it apply equally to non-default ACLs on any > >> other system objects? If you tweaked the permissions on say pg_ls_dir(), > >> then dump, then tweak them so

Re: [HACKERS] parallel.c oblivion of worker-startup failures

2018-01-24 Thread Peter Geoghegan
On Wed, Jan 24, 2018 at 12:05 PM, Robert Haas wrote: > In Thomas's test case, he's using 4 workers, and if even one of those > manages to start, then it'll probably do so after any fork failures > have already occurred, and the error will be noticed when that process > sends its first message to t

Re: Converting plpgsql to use DTYPE_REC for named composite types

2018-01-24 Thread Tom Lane
Here's a version of this rebased up to HEAD, fixing a couple of trivial merge conflicts and incorporating the docs delta I posted separately. (I'd supposed this patch was still OK because the patch tester said so, but I now see that the tester was only testing the docs delta :-(.)

Re: Possible performance regression with pg_dump of a large number of relations

2018-01-24 Thread Stephen Frost
Hi there! * Luke Cowell (lcow...@gmail.com) wrote: > Hi Stephen, thank you for putting this together. Yeah, it needs more work, which I figured out after actually hacking together a patch for it and I've just not gotten back to it yet. > > If folks get a chance to take a look at the query and/or

reducing isolation tests runtime

2018-01-24 Thread Alvaro Herrera
On the subject of test total time, we could paralelize isolation tests. Right now "make check" in src/test/isolation takes 1:16 on my machine. Test "timeouts" takes full 40s of that, with nothing running in parallel -- the machine is completely idle. Seems like we can have a lot of time back just

Re: plpgsql function startup-time improvements

2018-01-24 Thread Tom Lane
Pavel Stehule writes: > please, can you rebase all three patches necessary for patching? Done. These now need to be applied over https://www.postgresql.org/message-id/833.1516834...@sss.pgh.pa.us regards, tom lane diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgs

Re: reducing isolation tests runtime

2018-01-24 Thread Robert Haas
On Wed, Jan 24, 2018 at 6:10 PM, Alvaro Herrera wrote: > On the subject of test total time, we could paralelize isolation tests. > Right now "make check" in src/test/isolation takes 1:16 on my machine. > Test "timeouts" takes full 40s of that, with nothing running in parallel > -- the machine is c

Re: CONSTANT/NOT NULL/initializer properties for plpgsql record variables

2018-01-24 Thread Tom Lane
I wrote: > I said a couple of times in recent threads that it wouldn't be too hard > to implement $SUBJECT given the other patches I've been working on. Here's a version rebased up to HEAD, with a trivial merge conflict fixed. This now needs to be applied over the patches in https://postgr.es/m/8

Re: reducing isolation tests runtime

2018-01-24 Thread Tom Lane
Robert Haas writes: > On Wed, Jan 24, 2018 at 6:10 PM, Alvaro Herrera > wrote: >> On the subject of test total time, we could paralelize isolation tests. > Oh, cool. Yes, the time the isolation tests take to run is quite > annoying. I didn't realize it would be so easy to run it in parallel.

Re: [HACKERS] parallel.c oblivion of worker-startup failures

2018-01-24 Thread Robert Haas
On Wed, Jan 24, 2018 at 5:52 PM, Peter Geoghegan wrote: >> If we made the Gather node wait only for workers that actually reached >> the Gather -- either by using a Barrier or by some other technique -- >> then this would be a lot less fragile. And the same kind of technique >> would work for par

Re: reducing isolation tests runtime

2018-01-24 Thread Tom Lane
Alvaro Herrera writes: > On the subject of test total time, we could paralelize isolation tests. > Right now "make check" in src/test/isolation takes 1:16 on my machine. > Test "timeouts" takes full 40s of that, with nothing running in parallel > -- the machine is completely idle. BTW, one small

Re: [HACKERS] parallel.c oblivion of worker-startup failures

2018-01-24 Thread Peter Geoghegan
On Wed, Jan 24, 2018 at 3:37 PM, Robert Haas wrote: > Well, I've been resisting that approach from the very beginning of > parallel query. Eventually, I hope that we're going to go in the > direction of changing our mind about how many workers parallel > operations use "on the fly". For example,

Re: [HACKERS] SERIALIZABLE with parallel query

2018-01-24 Thread Thomas Munro
On Wed, Dec 13, 2017 at 5:30 PM, Thomas Munro wrote: > On Wed, Dec 13, 2017 at 2:09 PM, Haribabu Kommi > wrote: >> Thanks for explaining the problem in generating an isolation test to >> test the serialize parallel query. >> >> Committer can decide whether existing test is fine to part of the tes

Re: copy.c allocation constant

2018-01-24 Thread Tom Lane
Bruce Momjian writes: > On Thu, Jan 25, 2018 at 09:30:54AM +1300, Thomas Munro wrote: >> On Thu, Jan 25, 2018 at 7:19 AM, Robert Haas wrote: >>> My guess is that a fairly common pattern for larger chunks will be to >>> round the size up to a multiple of 4kB, the usual memory page size. >> >> See

Re: [HACKERS] [PATCH]make pg_rewind to not copy useless WAL files

2018-01-24 Thread Michael Paquier
On Wed, Jan 24, 2018 at 12:43:51PM -0500, Stephen Frost wrote: > * chenhj (chjis...@163.com) wrote: >> At 2018-01-23 09:56:48, "Stephen Frost" wrote: >>> I've only read through the thread to try and understand what's going on >>> and the first thing that comes to mind is that you're changing >>> p

Re: [HACKERS] Optional message to user when terminating/cancelling backend

2018-01-24 Thread Michael Paquier
On Wed, Jan 24, 2018 at 12:45:48PM -0300, Alvaro Herrera wrote: > /me wonders if there's anything that would suggest to make this > extensive to processes other than backends ... That's a good thought. Now ProcessInterrupts() is not used by non-backend processes. For example the WAL receiver has i

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2018-01-24 Thread Thomas Munro
On Thu, Jan 25, 2018 at 9:28 AM, Peter Geoghegan wrote: > On Wed, Jan 24, 2018 at 12:13 PM, Thomas Munro > wrote: >> On Thu, Jan 25, 2018 at 8:54 AM, Peter Geoghegan wrote: >>> I have used Thomas' chaos-monkey-fork-process.patch to verify: >>> >>> 1. The problem of fork failure causing nbtsort.c

Re: Failed to request an autovacuum work-item in silence

2018-01-24 Thread Masahiko Sawada
On Thu, Jan 25, 2018 at 12:14 AM, Fabrízio de Royes Mello wrote: > > > On Wed, Jan 24, 2018 at 12:31 PM, Fabrízio de Royes Mello > wrote: >> >> >> >> On Tue, Jan 23, 2018 at 11:44 PM, Masahiko Sawada >> wrote: >> > >> > On Tue, Jan 23, 2018 at 8:03 PM, Fabrízio de Royes Mello >> > wrote: >> > >

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2018-01-24 Thread Peter Geoghegan
On Wed, Jan 24, 2018 at 5:31 PM, Thomas Munro wrote: > Here's a version that works, and a minimal repro test module thing. > Without 0003 applied, it hangs. I can confirm that this version does in fact fix the problem with parallel CREATE INDEX hanging in the event of (simulated) worker fork() fa

Re: non-bulk inserts and tuple routing

2018-01-24 Thread Amit Langote
On 2018/01/24 17:25, Amit Langote wrote: > On 2018/01/20 7:07, Robert Haas wrote: >> On Fri, Jan 19, 2018 at 3:56 AM, Amit Langote wrote: >>> I rebased the patches, since they started conflicting with a recently >>> committed patch [1]. >> >> I think that my latest commit has managed to break this

Re: [HACKERS] Support to COMMENT ON DATABASE CURRENT_DATABASE

2018-01-24 Thread Jing Wang
>Not surprisingly, this patch no longer applies in the wake of commit >b3f840120. Rather than rebasing the pg_dump portions, I would suggest >you just drop them. It has been removed from the pg_dump codes. >I notice some other patch application failures in dbcommands.c, >objectaddress.c, and us

Further cleanup of pg_dump/pg_restore item selection code

2018-01-24 Thread Tom Lane
As I've been hacking on the pg_dump code recently, I got annoyed at the ugliness of its code for deciding whether or not to emit database-related TOC entries. That logic is implemented in a completely different place from other TOC entry selection decisions, and it has a bunch of strange behaviors

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2018-01-24 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> So are we at a consensus yet? > You had me at "make public less special", I was just trying to make sure > we all understand what that means. > +1 from me for moving forward. Applying this patch will leave us with the original pg_

RE: [HACKERS][PATCH] Applying PMDK to WAL operations for persistent memory

2018-01-24 Thread Tsunakawa, Takayuki
From: Robert Haas [mailto:robertmh...@gmail.com] > I think open_datasync will be worse on systems where fsync() is expensive > -- it forces the data out to disk immediately, even if the data doesn't > need to be flushed immediately. That's bad, because we wait immediately > when we could have defe

Further cleanup of pg_dump/pg_restore item selection code

2018-01-24 Thread David G. Johnston
On Wednesday, January 24, 2018, Tom Lane wrote: > and it has a bunch of strange > behaviors that can really only be characterized as bugs. An example is > that > > pg_dump --create -t sometable somedb > > pg_dump -t: "The -n and -N switches have no effect when -t is used, because tables

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Tom Lane
Robert Haas writes: > There is no need to collect years of data in order to tell whether or > not the time to run the tests has increased by as much on developer > machines as it has on prairiedog. You showed the time going from 3:36 > to 8:09 between 2014 and the present. That is a 2.26x increa

Re: Further cleanup of pg_dump/pg_restore item selection code

2018-01-24 Thread Tom Lane
"David G. Johnston" writes: > On Wednesday, January 24, 2018, Tom Lane wrote: >> The same behaviors occur if you do e.g. >> pg_dump -Fc -t sometable somedb | pg_restore --create >> which is another undocumented misbehavior: the docs for pg_restore do not >> suggest that --create might fail i

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Robert Haas
On Wed, Jan 24, 2018 at 11:02 PM, Tom Lane wrote: > I may be wasting my breath here, but in one more attempt to convince > you that "time make check" on your laptop is not the only number that > anyone should be interested in, ... Now that is not what I said, or at least not what I intended to sa

Re: CREATE ROUTINE MAPPING

2018-01-24 Thread David Fetter
On Thu, Jan 18, 2018 at 04:09:13PM -0500, Corey Huinker wrote: > > > > > > > > > > But other situations seem un-handle-able to me: > > > > > > SELECT remote_func1(l.x) FROM local_table l WHERE l.active = true; > > > > Do we have any way, or any plan to make a way, to push the set (SELECT > > x FROM

Re: Further cleanup of pg_dump/pg_restore item selection code

2018-01-24 Thread David G. Johnston
On Wednesday, January 24, 2018, Tom Lane wrote: > > > This does not go all the way towards making pg_restore's item selection > switches dump subsidiary objects the same as pg_dump would, because > pg_restore doesn't really have enough info to deal with indexes and > table constraints the way pg_d

Re: Further cleanup of pg_dump/pg_restore item selection code

2018-01-24 Thread David G. Johnston
On Wednesday, January 24, 2018, Tom Lane wrote: > > I think you might be missing one of the main points here, which is > that --create is specified as causing both a CREATE DATABASE and a > reconnect to that database. > I missed the implication though I read and even thought about questioning tha

Re: PATCH: Exclude unlogged tables from base backups

2018-01-24 Thread Masahiko Sawada
On Thu, Jan 25, 2018 at 3:25 AM, David Steele wrote: > Hi Masahiko, > > Thanks for the review! > > On 1/22/18 3:14 AM, Masahiko Sawada wrote: >> On Thu, Dec 14, 2017 at 11:58 PM, Robert Haas wrote: >>> >>> We would also have a problem if the missing file caused something in >>> recovery to croak

Temporary tables prevent autovacuum, leading to XID wraparound

2018-01-24 Thread Tsunakawa, Takayuki
Hello, I've found a problem that an orphaned temporary table could cause XID wraparound. Our customer encountered this problem with PG 9.5.2, but I think this will happen with the latest PG. I'm willing to fix this, but I'd like to ask you what approach we should take. PROBLEM ==

Re: Rangejoin rebased

2018-01-24 Thread Jeff Davis
On Tue, Jan 23, 2018 at 2:04 AM, Simon Riggs wrote: > Perhaps we are misunderstanding each other > > TIMESTAMP <@ RANGE1 doesn't match if RANGE1 is empty > and that is the most important case When <@ is supported, that case should be fine if range1 is on the outer. The case I was concerned about

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Jeff Davis
On Wed, Jan 24, 2018 at 1:35 PM, Pierre Ducroquet wrote: > In LLVM 5.0, it looks like DebugInfo.h is not available in llvm-c, only as a C > ++ API in llvm/IR/DebugInfo.h. The LLVM APIs don't seem to be very stable; won't there just be a continuous stream of similar issues? Pinning major postgres

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Andres Freund
Hi, On 2018-01-24 22:33:30 -0800, Jeff Davis wrote: > On Wed, Jan 24, 2018 at 1:35 PM, Pierre Ducroquet wrote: > > In LLVM 5.0, it looks like DebugInfo.h is not available in llvm-c, only as > > a C > > ++ API in llvm/IR/DebugInfo.h. > > The LLVM APIs don't seem to be very stable; won't there ju

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Jeff Davis
On Tue, Jan 23, 2018 at 11:20 PM, Andres Freund wrote: > Hi, > > I've spent the last weeks working on my LLVM compilation patchset. In > the course of that I *heavily* revised it. While still a good bit away > from committable, it's IMO definitely not a prototype anymore. Great! A couple high-le

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Andres Freund
Hi! On 2018-01-24 22:51:36 -0800, Jeff Davis wrote: > A couple high-level questions: > > 1. I notice a lot of use of the LLVM builder, for example, in > slot_compile_deform(). Why can't you do the same thing you did with > function code, where you create the ".bc" at build time from plain C > cod

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-01-24 Thread Thomas Munro
On Thu, Jan 18, 2018 at 9:17 AM, Claudio Freire wrote: > Huh. That was simpler than I thought. > > Attached rebased versions. Hi Claudio, FYI the regression test seems to have some run-to-run variation. Though it usually succeeds, recently I have seen a couple of failures like this: = C

Re: [PATCH] Logical decoding of TRUNCATE

2018-01-24 Thread Thomas Munro
On Wed, Jan 24, 2018 at 6:47 AM, Marco Nenciarini wrote: > Version 16 attached. Hi Marco, FYI this version doesn't compile: worker.c: In function ‘apply_handle_truncate’: worker.c:888:39: error: ‘cascade’ undeclared (first use in this function) relid = logicalrep_read_truncate(s, &cascade, &r

Re: ALTER TABLE ADD COLUMN fast default

2018-01-24 Thread Thomas Munro
On Wed, Jan 17, 2018 at 2:21 AM, Andrew Dunstan wrote: > Yeah, got caught by the bki/pg_attribute changes on Friday. here's an > updated version. Thanks for looking. A boring semi-automated update: this no long compiles, because 8561e4840c8 added a new call to heap_attisnull(). Pretty sure it j

Re: [HACKERS] [PATCH] Lockable views

2018-01-24 Thread Thomas Munro
On Sun, Dec 31, 2017 at 11:57 PM, Yugo Nagata wrote: > On Fri, 29 Dec 2017 23:39:39 +0900 (JST) > Tatsuo Ishii wrote: >> Your addition to the doc: >> + Automatically updatable views (see ) >> + that do not have INSTEAD OF triggers or INSTEAD >> + rules are also lockable. When a view is lock

Re: Temporary tables prevent autovacuum, leading to XID wraparound

2018-01-24 Thread Michael Paquier
On Thu, Jan 25, 2018 at 06:14:41AM +, Tsunakawa, Takayuki wrote: > I don't know why pg_temp3.fetchchunks still exists. Maybe the user > ran pg_ctl stop -mi while pg_rewind was running. Likely that was the case :( As a superuser, DROP TABLE should work on the temporary schema of another sessi

<    1   2