Re: Temporary tables prevent autovacuum, leading to XID wraparound

2018-01-24 Thread Michael Paquier
On Thu, Jan 25, 2018 at 06:14:41AM +, Tsunakawa, Takayuki wrote: > I don't know why pg_temp3.fetchchunks still exists. Maybe the user > ran pg_ctl stop -mi while pg_rewind was running. Likely that was the case :( As a superuser, DROP TABLE should work on the temporary schema of another sessi

Re: [HACKERS] [PATCH] Lockable views

2018-01-24 Thread Thomas Munro
On Sun, Dec 31, 2017 at 11:57 PM, Yugo Nagata wrote: > On Fri, 29 Dec 2017 23:39:39 +0900 (JST) > Tatsuo Ishii wrote: >> Your addition to the doc: >> + Automatically updatable views (see ) >> + that do not have INSTEAD OF triggers or INSTEAD >> + rules are also lockable. When a view is lock

Re: ALTER TABLE ADD COLUMN fast default

2018-01-24 Thread Thomas Munro
On Wed, Jan 17, 2018 at 2:21 AM, Andrew Dunstan wrote: > Yeah, got caught by the bki/pg_attribute changes on Friday. here's an > updated version. Thanks for looking. A boring semi-automated update: this no long compiles, because 8561e4840c8 added a new call to heap_attisnull(). Pretty sure it j

Re: [PATCH] Logical decoding of TRUNCATE

2018-01-24 Thread Thomas Munro
On Wed, Jan 24, 2018 at 6:47 AM, Marco Nenciarini wrote: > Version 16 attached. Hi Marco, FYI this version doesn't compile: worker.c: In function ‘apply_handle_truncate’: worker.c:888:39: error: ‘cascade’ undeclared (first use in this function) relid = logicalrep_read_truncate(s, &cascade, &r

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-01-24 Thread Thomas Munro
On Thu, Jan 18, 2018 at 9:17 AM, Claudio Freire wrote: > Huh. That was simpler than I thought. > > Attached rebased versions. Hi Claudio, FYI the regression test seems to have some run-to-run variation. Though it usually succeeds, recently I have seen a couple of failures like this: = C

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Andres Freund
Hi! On 2018-01-24 22:51:36 -0800, Jeff Davis wrote: > A couple high-level questions: > > 1. I notice a lot of use of the LLVM builder, for example, in > slot_compile_deform(). Why can't you do the same thing you did with > function code, where you create the ".bc" at build time from plain C > cod

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Jeff Davis
On Tue, Jan 23, 2018 at 11:20 PM, Andres Freund wrote: > Hi, > > I've spent the last weeks working on my LLVM compilation patchset. In > the course of that I *heavily* revised it. While still a good bit away > from committable, it's IMO definitely not a prototype anymore. Great! A couple high-le

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Andres Freund
Hi, On 2018-01-24 22:33:30 -0800, Jeff Davis wrote: > On Wed, Jan 24, 2018 at 1:35 PM, Pierre Ducroquet wrote: > > In LLVM 5.0, it looks like DebugInfo.h is not available in llvm-c, only as > > a C > > ++ API in llvm/IR/DebugInfo.h. > > The LLVM APIs don't seem to be very stable; won't there ju

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Jeff Davis
On Wed, Jan 24, 2018 at 1:35 PM, Pierre Ducroquet wrote: > In LLVM 5.0, it looks like DebugInfo.h is not available in llvm-c, only as a C > ++ API in llvm/IR/DebugInfo.h. The LLVM APIs don't seem to be very stable; won't there just be a continuous stream of similar issues? Pinning major postgres

Re: Rangejoin rebased

2018-01-24 Thread Jeff Davis
On Tue, Jan 23, 2018 at 2:04 AM, Simon Riggs wrote: > Perhaps we are misunderstanding each other > > TIMESTAMP <@ RANGE1 doesn't match if RANGE1 is empty > and that is the most important case When <@ is supported, that case should be fine if range1 is on the outer. The case I was concerned about

Temporary tables prevent autovacuum, leading to XID wraparound

2018-01-24 Thread Tsunakawa, Takayuki
Hello, I've found a problem that an orphaned temporary table could cause XID wraparound. Our customer encountered this problem with PG 9.5.2, but I think this will happen with the latest PG. I'm willing to fix this, but I'd like to ask you what approach we should take. PROBLEM ==

Re: PATCH: Exclude unlogged tables from base backups

2018-01-24 Thread Masahiko Sawada
On Thu, Jan 25, 2018 at 3:25 AM, David Steele wrote: > Hi Masahiko, > > Thanks for the review! > > On 1/22/18 3:14 AM, Masahiko Sawada wrote: >> On Thu, Dec 14, 2017 at 11:58 PM, Robert Haas wrote: >>> >>> We would also have a problem if the missing file caused something in >>> recovery to croak

Re: Further cleanup of pg_dump/pg_restore item selection code

2018-01-24 Thread David G. Johnston
On Wednesday, January 24, 2018, Tom Lane wrote: > > I think you might be missing one of the main points here, which is > that --create is specified as causing both a CREATE DATABASE and a > reconnect to that database. > I missed the implication though I read and even thought about questioning tha

Re: Further cleanup of pg_dump/pg_restore item selection code

2018-01-24 Thread David G. Johnston
On Wednesday, January 24, 2018, Tom Lane wrote: > > > This does not go all the way towards making pg_restore's item selection > switches dump subsidiary objects the same as pg_dump would, because > pg_restore doesn't really have enough info to deal with indexes and > table constraints the way pg_d

Re: CREATE ROUTINE MAPPING

2018-01-24 Thread David Fetter
On Thu, Jan 18, 2018 at 04:09:13PM -0500, Corey Huinker wrote: > > > > > > > > > > But other situations seem un-handle-able to me: > > > > > > SELECT remote_func1(l.x) FROM local_table l WHERE l.active = true; > > > > Do we have any way, or any plan to make a way, to push the set (SELECT > > x FROM

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Robert Haas
On Wed, Jan 24, 2018 at 11:02 PM, Tom Lane wrote: > I may be wasting my breath here, but in one more attempt to convince > you that "time make check" on your laptop is not the only number that > anyone should be interested in, ... Now that is not what I said, or at least not what I intended to sa

Re: Further cleanup of pg_dump/pg_restore item selection code

2018-01-24 Thread Tom Lane
"David G. Johnston" writes: > On Wednesday, January 24, 2018, Tom Lane wrote: >> The same behaviors occur if you do e.g. >> pg_dump -Fc -t sometable somedb | pg_restore --create >> which is another undocumented misbehavior: the docs for pg_restore do not >> suggest that --create might fail i

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Tom Lane
Robert Haas writes: > There is no need to collect years of data in order to tell whether or > not the time to run the tests has increased by as much on developer > machines as it has on prairiedog. You showed the time going from 3:36 > to 8:09 between 2014 and the present. That is a 2.26x increa

Further cleanup of pg_dump/pg_restore item selection code

2018-01-24 Thread David G. Johnston
On Wednesday, January 24, 2018, Tom Lane wrote: > and it has a bunch of strange > behaviors that can really only be characterized as bugs. An example is > that > > pg_dump --create -t sometable somedb > > pg_dump -t: "The -n and -N switches have no effect when -t is used, because tables

RE: [HACKERS][PATCH] Applying PMDK to WAL operations for persistent memory

2018-01-24 Thread Tsunakawa, Takayuki
From: Robert Haas [mailto:robertmh...@gmail.com] > I think open_datasync will be worse on systems where fsync() is expensive > -- it forces the data out to disk immediately, even if the data doesn't > need to be flushed immediately. That's bad, because we wait immediately > when we could have defe

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2018-01-24 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> So are we at a consensus yet? > You had me at "make public less special", I was just trying to make sure > we all understand what that means. > +1 from me for moving forward. Applying this patch will leave us with the original pg_

Further cleanup of pg_dump/pg_restore item selection code

2018-01-24 Thread Tom Lane
As I've been hacking on the pg_dump code recently, I got annoyed at the ugliness of its code for deciding whether or not to emit database-related TOC entries. That logic is implemented in a completely different place from other TOC entry selection decisions, and it has a bunch of strange behaviors

Re: [HACKERS] Support to COMMENT ON DATABASE CURRENT_DATABASE

2018-01-24 Thread Jing Wang
>Not surprisingly, this patch no longer applies in the wake of commit >b3f840120. Rather than rebasing the pg_dump portions, I would suggest >you just drop them. It has been removed from the pg_dump codes. >I notice some other patch application failures in dbcommands.c, >objectaddress.c, and us

Re: non-bulk inserts and tuple routing

2018-01-24 Thread Amit Langote
On 2018/01/24 17:25, Amit Langote wrote: > On 2018/01/20 7:07, Robert Haas wrote: >> On Fri, Jan 19, 2018 at 3:56 AM, Amit Langote wrote: >>> I rebased the patches, since they started conflicting with a recently >>> committed patch [1]. >> >> I think that my latest commit has managed to break this

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2018-01-24 Thread Peter Geoghegan
On Wed, Jan 24, 2018 at 5:31 PM, Thomas Munro wrote: > Here's a version that works, and a minimal repro test module thing. > Without 0003 applied, it hangs. I can confirm that this version does in fact fix the problem with parallel CREATE INDEX hanging in the event of (simulated) worker fork() fa

Re: Failed to request an autovacuum work-item in silence

2018-01-24 Thread Masahiko Sawada
On Thu, Jan 25, 2018 at 12:14 AM, Fabrízio de Royes Mello wrote: > > > On Wed, Jan 24, 2018 at 12:31 PM, Fabrízio de Royes Mello > wrote: >> >> >> >> On Tue, Jan 23, 2018 at 11:44 PM, Masahiko Sawada >> wrote: >> > >> > On Tue, Jan 23, 2018 at 8:03 PM, Fabrízio de Royes Mello >> > wrote: >> > >

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2018-01-24 Thread Thomas Munro
On Thu, Jan 25, 2018 at 9:28 AM, Peter Geoghegan wrote: > On Wed, Jan 24, 2018 at 12:13 PM, Thomas Munro > wrote: >> On Thu, Jan 25, 2018 at 8:54 AM, Peter Geoghegan wrote: >>> I have used Thomas' chaos-monkey-fork-process.patch to verify: >>> >>> 1. The problem of fork failure causing nbtsort.c

Re: [HACKERS] Optional message to user when terminating/cancelling backend

2018-01-24 Thread Michael Paquier
On Wed, Jan 24, 2018 at 12:45:48PM -0300, Alvaro Herrera wrote: > /me wonders if there's anything that would suggest to make this > extensive to processes other than backends ... That's a good thought. Now ProcessInterrupts() is not used by non-backend processes. For example the WAL receiver has i

Re: [HACKERS] [PATCH]make pg_rewind to not copy useless WAL files

2018-01-24 Thread Michael Paquier
On Wed, Jan 24, 2018 at 12:43:51PM -0500, Stephen Frost wrote: > * chenhj (chjis...@163.com) wrote: >> At 2018-01-23 09:56:48, "Stephen Frost" wrote: >>> I've only read through the thread to try and understand what's going on >>> and the first thing that comes to mind is that you're changing >>> p

Re: copy.c allocation constant

2018-01-24 Thread Tom Lane
Bruce Momjian writes: > On Thu, Jan 25, 2018 at 09:30:54AM +1300, Thomas Munro wrote: >> On Thu, Jan 25, 2018 at 7:19 AM, Robert Haas wrote: >>> My guess is that a fairly common pattern for larger chunks will be to >>> round the size up to a multiple of 4kB, the usual memory page size. >> >> See

Re: [HACKERS] SERIALIZABLE with parallel query

2018-01-24 Thread Thomas Munro
On Wed, Dec 13, 2017 at 5:30 PM, Thomas Munro wrote: > On Wed, Dec 13, 2017 at 2:09 PM, Haribabu Kommi > wrote: >> Thanks for explaining the problem in generating an isolation test to >> test the serialize parallel query. >> >> Committer can decide whether existing test is fine to part of the tes

Re: [HACKERS] parallel.c oblivion of worker-startup failures

2018-01-24 Thread Peter Geoghegan
On Wed, Jan 24, 2018 at 3:37 PM, Robert Haas wrote: > Well, I've been resisting that approach from the very beginning of > parallel query. Eventually, I hope that we're going to go in the > direction of changing our mind about how many workers parallel > operations use "on the fly". For example,

Re: reducing isolation tests runtime

2018-01-24 Thread Tom Lane
Alvaro Herrera writes: > On the subject of test total time, we could paralelize isolation tests. > Right now "make check" in src/test/isolation takes 1:16 on my machine. > Test "timeouts" takes full 40s of that, with nothing running in parallel > -- the machine is completely idle. BTW, one small

Re: [HACKERS] parallel.c oblivion of worker-startup failures

2018-01-24 Thread Robert Haas
On Wed, Jan 24, 2018 at 5:52 PM, Peter Geoghegan wrote: >> If we made the Gather node wait only for workers that actually reached >> the Gather -- either by using a Barrier or by some other technique -- >> then this would be a lot less fragile. And the same kind of technique >> would work for par

Re: reducing isolation tests runtime

2018-01-24 Thread Tom Lane
Robert Haas writes: > On Wed, Jan 24, 2018 at 6:10 PM, Alvaro Herrera > wrote: >> On the subject of test total time, we could paralelize isolation tests. > Oh, cool. Yes, the time the isolation tests take to run is quite > annoying. I didn't realize it would be so easy to run it in parallel.

Re: CONSTANT/NOT NULL/initializer properties for plpgsql record variables

2018-01-24 Thread Tom Lane
I wrote: > I said a couple of times in recent threads that it wouldn't be too hard > to implement $SUBJECT given the other patches I've been working on. Here's a version rebased up to HEAD, with a trivial merge conflict fixed. This now needs to be applied over the patches in https://postgr.es/m/8

Re: reducing isolation tests runtime

2018-01-24 Thread Robert Haas
On Wed, Jan 24, 2018 at 6:10 PM, Alvaro Herrera wrote: > On the subject of test total time, we could paralelize isolation tests. > Right now "make check" in src/test/isolation takes 1:16 on my machine. > Test "timeouts" takes full 40s of that, with nothing running in parallel > -- the machine is c

Re: plpgsql function startup-time improvements

2018-01-24 Thread Tom Lane
Pavel Stehule writes: > please, can you rebase all three patches necessary for patching? Done. These now need to be applied over https://www.postgresql.org/message-id/833.1516834...@sss.pgh.pa.us regards, tom lane diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgs

reducing isolation tests runtime

2018-01-24 Thread Alvaro Herrera
On the subject of test total time, we could paralelize isolation tests. Right now "make check" in src/test/isolation takes 1:16 on my machine. Test "timeouts" takes full 40s of that, with nothing running in parallel -- the machine is completely idle. Seems like we can have a lot of time back just

Re: Possible performance regression with pg_dump of a large number of relations

2018-01-24 Thread Stephen Frost
Hi there! * Luke Cowell (lcow...@gmail.com) wrote: > Hi Stephen, thank you for putting this together. Yeah, it needs more work, which I figured out after actually hacking together a patch for it and I've just not gotten back to it yet. > > If folks get a chance to take a look at the query and/or

Re: Converting plpgsql to use DTYPE_REC for named composite types

2018-01-24 Thread Tom Lane
Here's a version of this rebased up to HEAD, fixing a couple of trivial merge conflicts and incorporating the docs delta I posted separately. (I'd supposed this patch was still OK because the patch tester said so, but I now see that the tester was only testing the docs delta :-(.)

Re: [HACKERS] parallel.c oblivion of worker-startup failures

2018-01-24 Thread Peter Geoghegan
On Wed, Jan 24, 2018 at 12:05 PM, Robert Haas wrote: > In Thomas's test case, he's using 4 workers, and if even one of those > manages to start, then it'll probably do so after any fork failures > have already occurred, and the error will be noticed when that process > sends its first message to t

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2018-01-24 Thread Stephen Frost
Tom, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> Fair point, but doesn't it apply equally to non-default ACLs on any > >> other system objects? If you tweaked the permissions on say pg_ls_dir(), > >> then dump, then tweak them so

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Andres Freund
Hi, On 2018-01-24 14:06:30 -0800, Andres Freund wrote: > > In LLVM 5.0, it looks like DebugInfo.h is not available in llvm-c, only as > > a C > > ++ API in llvm/IR/DebugInfo.h. > > Hm, I compiled against 5.0 quite recently, but added the stripping of > debuginfo lateron. I'll add a fallback met

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Andres Freund
Hi, On 2018-01-24 22:35:08 +0100, Pierre Ducroquet wrote: > I tried to build on Debian sid, using GCC 7 and LLVM 5. I used the following > to compile, using your branch @3195c2821d : Thanks! > $ export LLVM_CONFIG=/usr/bin/llvm-config-5.0 > $ ./configure --with-llvm > $ make > > And I had the

Re: [HACKERS] UPDATE of partition key

2018-01-24 Thread Thomas Munro
On Thu, Jan 25, 2018 at 10:39 AM, Robert Haas wrote: > On Mon, Jan 22, 2018 at 9:46 AM, Tom Lane wrote: >> Robert Haas writes: >>> Tom, do you want to double-check that this fixes it for you? >> >> I can confirm that a valgrind run succeeded for me with the patch >> in place. > > Committed. Sor

Re: Possible performance regression with pg_dump of a large number of relations

2018-01-24 Thread Luke Cowell
Hi Stephen, thank you for putting this together. > If folks get a chance to take a look at the query and/or test, that'd be > great. I'll try to work up an actual patch to pg_dump this weekend to > run it through the regression tests and see if anything breaks. I'm not sure how I can help other

Re: documentation is now XML

2018-01-24 Thread Peter Eisentraut
On 1/24/18 09:45, Bruce Momjian wrote: > So we are not > using TeX anymore for PG11+ docs? as of PG10 -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] UPDATE of partition key

2018-01-24 Thread Robert Haas
On Mon, Jan 22, 2018 at 9:46 AM, Tom Lane wrote: > Robert Haas writes: >> Tom, do you want to double-check that this fixes it for you? > > I can confirm that a valgrind run succeeded for me with the patch > in place. Committed. Sorry for the delay. -- Robert Haas EnterpriseDB: http://www.ente

Re: JIT compiling with LLVM v9.0

2018-01-24 Thread Pierre Ducroquet
On Wednesday, January 24, 2018 8:20:38 AM CET Andres Freund wrote: > As the patchset is large (500kb) and I'm still quickly evolving it, I do > not yet want to attach it. The git tree is at > https://git.postgresql.org/git/users/andresfreund/postgres.git > in the jit branch > > https://git.post

Re: PATCH: Exclude unlogged tables from base backups

2018-01-24 Thread David Steele
On 1/24/18 4:02 PM, Adam Brightwell wrote: >>> If a new unlogged relation is created after constructed the >>> unloggedHash before sending file, we cannot exclude such relation. It >>> would not be problem if the taking backup is not long because the new >>> unlogged relation unlikely becomes so la

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2018-01-24 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Fair point, but doesn't it apply equally to non-default ACLs on any >> other system objects? If you tweaked the permissions on say pg_ls_dir(), >> then dump, then tweak them some more, you're going to get uncertain >> results if yo

Re: PATCH: Exclude unlogged tables from base backups

2018-01-24 Thread Adam Brightwell
> I agree with #1 and feel the updated docs are reasonable and > sufficient to address this case for now. > > I have retested these patches against master at d6ab720360. > > All test succeed. > > Marking "Ready for Committer". Actually, marked it "Ready for Review" to wait for Masahiko to comment/

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Robert Haas
On Wed, Jan 24, 2018 at 4:01 PM, Tom Lane wrote: > The progress-display output of pg_regress would need a complete rethink > anyhow. First thought is to emit two lines per test, one when we > launch it and one when it finishes and we check the results: > > foreign_data: launched > ... > foreign_d

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Robert Haas
On Wed, Jan 24, 2018 at 2:31 PM, Tom Lane wrote: > I find that to be a completely bogus straw-man argument. The point of > looking at the prairiedog time series is just to see a data series in > which the noise level is small enough to discern the signal. If anyone's > got years worth of data of

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Andres Freund
Hi, On 2018-01-24 15:58:16 -0500, Tom Lane wrote: > Yeah. We already have topo sort code in pg_dump, maybe we could push that > into someplace like src/common or src/fe_utils? Although pg_dump hasn't > got any need for edge weights, so maybe sharing code isn't worth it. I suspect it may be more

Re: PATCH: Exclude unlogged tables from base backups

2018-01-24 Thread Adam Brightwell
>> If a new unlogged relation is created after constructed the >> unloggedHash before sending file, we cannot exclude such relation. It >> would not be problem if the taking backup is not long because the new >> unlogged relation unlikely becomes so large. However, if takeing a >> backup takes a lo

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Tom Lane
Andres Freund writes: > On 2018-01-24 15:36:35 -0500, Tom Lane wrote: >> I'm concerned that we'd end up with a higher number of irreproducible >> test failures with no good way to investigate them. > Hm. We probably should dump the used ordering of tests somwhere upon > failure, to make it easier

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Tom Lane
Andres Freund writes: > On 2018-01-24 17:18:26 -0300, Alvaro Herrera wrote: >> Yeah, I proposed this a decade ago but never had the wits to write the >> code. > It shouldn't be too hard, right? Leaving defining the file format, > parsing it, creating the new schedule with depencencies and adaptin

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2018-01-24 Thread Stephen Frost
Tom, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> No, if you have a nondefault ACL, that will still get applied. This > >> arrangement would drop comment changes, but I can't get excited about > >> that; it's certainly far less of

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2018-01-24 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> No, if you have a nondefault ACL, that will still get applied. This >> arrangement would drop comment changes, but I can't get excited about >> that; it's certainly far less of an inconvenience in that scenario >> than dumping the

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Andres Freund
Hi, On 2018-01-24 15:36:35 -0500, Tom Lane wrote: > There'd be a lot of followup work to sanitize the tests better. For > instance, if two tests transiently create tables named "foo", it doesn't > matter as long as they're not in the same group. It would matter with > this. Right. I suspect we'

Re: copy.c allocation constant

2018-01-24 Thread Thomas Munro
On Thu, Jan 25, 2018 at 9:35 AM, Bruce Momjian wrote: > The BSD memory allocator used to allocate in powers of two, and keep the > header in a separate location. They did this so they could combine two > free, identically-sized memory blocks into a single one that was double > the size. I have n

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Andres Freund
Hi, On 2018-01-24 17:18:26 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > Besides larger groups, starting the next test(s) earlier, another way to > > gain pretty large improvements would be a test schedule feature that > > allowed to stat dependencies between tests. So instead of manuall

Re: WIP Patch: Precalculate stable functions, infrastructure v1

2018-01-24 Thread Andres Freund
On 2018-01-24 15:10:56 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2018-01-16 17:05:01 -0500, Tom Lane wrote: > >> I'm curious to know whether Andres has some other ideas, or whether he > >> feels this is all a big wart on the compiled-expression concept. > > > I don't have too many "art

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Tom Lane
Alvaro Herrera writes: > Andres Freund wrote: >> Besides larger groups, starting the next test(s) earlier, another way to >> gain pretty large improvements would be a test schedule feature that >> allowed to stat dependencies between tests. So instead of manually >> grouping the schedule, have 'nu

Re: copy.c allocation constant

2018-01-24 Thread Bruce Momjian
On Thu, Jan 25, 2018 at 09:30:54AM +1300, Thomas Munro wrote: > On Thu, Jan 25, 2018 at 7:19 AM, Robert Haas wrote: > > On Wed, Jan 24, 2018 at 12:25 PM, Tomas Vondra > > wrote: > >> At the glibc level ... I'm not so sure. AFAIK glibc uses an allocator > >> with similar ideas (freelists, ...) so

Re: copy.c allocation constant

2018-01-24 Thread Thomas Munro
On Thu, Jan 25, 2018 at 7:19 AM, Robert Haas wrote: > On Wed, Jan 24, 2018 at 12:25 PM, Tomas Vondra > wrote: >> At the glibc level ... I'm not so sure. AFAIK glibc uses an allocator >> with similar ideas (freelists, ...) so hopefully it's fine too. >> >> And then there are the systems without gl

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2018-01-24 Thread Peter Geoghegan
On Wed, Jan 24, 2018 at 12:13 PM, Thomas Munro wrote: > On Thu, Jan 25, 2018 at 8:54 AM, Peter Geoghegan wrote: >> I have used Thomas' chaos-monkey-fork-process.patch to verify: >> >> 1. The problem of fork failure causing nbtsort.c to wait forever is a >> real problem. Sure enough, the coding pa

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Alvaro Herrera
Thomas Munro wrote: > On Wed, Jan 24, 2018 at 12:10 PM, Tom Lane wrote: > > However, the trend over the last two months is very bad, and I do > > not think that we can point to any large improvement in test > > coverage that someone committed since November. > > I'm not sure if coverage.postgres

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Alvaro Herrera
Andres Freund wrote: > Besides larger groups, starting the next test(s) earlier, another way to > gain pretty large improvements would be a test schedule feature that > allowed to stat dependencies between tests. So instead of manually > grouping the schedule, have 'numerology' state that it depen

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2018-01-24 Thread Thomas Munro
On Thu, Jan 25, 2018 at 8:54 AM, Peter Geoghegan wrote: > I have used Thomas' chaos-monkey-fork-process.patch to verify: > > 1. The problem of fork failure causing nbtsort.c to wait forever is a > real problem. Sure enough, the coding pattern within > _bt_leader_heapscan() can cause us to wait for

Re: copy.c allocation constant

2018-01-24 Thread Andres Freund
Hi, On 2018-01-24 17:07:01 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > glibc's malloc does add a header. My half-informed suspicion is that > > most newer malloc backing allocators will have a header, because > > maintaining a shared lookup-by-address table is pretty expensive to > > m

Re: WIP Patch: Precalculate stable functions, infrastructure v1

2018-01-24 Thread Tom Lane
Andres Freund writes: > On 2018-01-16 17:05:01 -0500, Tom Lane wrote: >> I'm curious to know whether Andres has some other ideas, or whether he >> feels this is all a big wart on the compiled-expression concept. > I don't have too many "artistic" concerns from the compiled expression > POV. The b

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Andres Freund
Hi, On 2018-01-24 14:31:47 -0500, Tom Lane wrote: > However ... if you spend any time looking at the behavior of that, > the hashjoin tests are still problematic. I think my main problem with your arguments is that you basically seem to say that one of the more complex features in postgres can't

Re: copy.c allocation constant

2018-01-24 Thread Alvaro Herrera
Andres Freund wrote: > On 2018-01-24 14:25:37 -0500, Robert Haas wrote: > > On Wed, Jan 24, 2018 at 1:43 PM, Andres Freund wrote: > > > Indeed. Don't think RAW_BUF_SIZE is quite big enough for that on most > > > platforms though. From man mallopt: > > > Balancing these factors leads to a defa

Re: [HACKERS] parallel.c oblivion of worker-startup failures

2018-01-24 Thread Robert Haas
On Tue, Jan 23, 2018 at 9:45 PM, Amit Kapila wrote: > Hmm, I think that case will be addressed because tuple queues can > detect if the leader is not attached. It does in code path > shm_mq_receive->shm_mq_counterparty_gone. In > shm_mq_counterparty_gone, it can detect if the worker is gone by u

Re: [HACKERS] parallel.c oblivion of worker-startup failures

2018-01-24 Thread Peter Geoghegan
On Wed, Jan 24, 2018 at 1:57 AM, Thomas Munro wrote: > On Wed, Jan 24, 2018 at 5:25 PM, Thomas Munro > wrote: >> If there were some way for the postmaster to cause reason >> PROCSIG_PARALLEL_MESSAGE to be set in the leader process instead of >> just notification via kill(SIGUSR1) when it fails to

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Tom Lane
I wrote: > I find that to be a completely bogus straw-man argument. The point of > looking at the prairiedog time series is just to see a data series in > which the noise level is small enough to discern the signal. If anyone's > got years worth of data off a more modern machine, and they can ext

Re: copy.c allocation constant

2018-01-24 Thread Andres Freund
On 2018-01-24 14:25:37 -0500, Robert Haas wrote: > On Wed, Jan 24, 2018 at 1:43 PM, Andres Freund wrote: > > Indeed. Don't think RAW_BUF_SIZE is quite big enough for that on most > > platforms though. From man mallopt: > > Balancing these factors leads to a default setting of 128*1024 for the

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2018-01-24 Thread Peter Geoghegan
On Tue, Jan 23, 2018 at 9:43 PM, Amit Kapila wrote: > Right, but what if the worker dies due to something proc_exit(1) or > something like that before calling BarrierArriveAndWait. I think this > is part of the problem we have solved in > WaitForParallelWorkersToFinish such that if the worker exi

Re: pgindent run?

2018-01-24 Thread Andres Freund
On January 24, 2018 11:34:07 AM PST, Tom Lane wrote: >Andres Freund writes: >> There'd be one or two edge cases of bad formatting, but the >> end result would be far less painful than what we have today, were >> basically nobody can format their patches without a lot of manual >> cherry-picking

Re: pgindent run?

2018-01-24 Thread Tom Lane
Andres Freund writes: > FWIW, I think this problem could just as well be addressed with a few > printing heuristics instead of actually needing an actual list of > typedefs. Step right up and implement that, and we'd all be happier. Certainly the typedefs list is a pain in the rear. > There'd b

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Tom Lane
Andres Freund writes: > On 2018-01-24 13:11:22 -0500, Robert Haas wrote: >> Now, how much should we care about the performance of software with a >> planned release date of 2018 on hardware discontinued in 2001, >> hardware that is apparently about 20 times slower than a modern >> laptop? Some, p

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2018-01-24 Thread Stephen Frost
Tom, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > I'm afraid we may still get some push-back from existing users of > > --clean since, with the change you're proposing, we wouldn't be cleaning > > up anything that's been done to the public schema when it comes to > > comment

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-24 Thread Alvaro Herrera
I think this is the right fix for this problem. I wonder about exploring other callers of RelationGetIndexList to see who else could be confused ... -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 53506fd3a

Re: Foreign keys and partitioned tables

2018-01-24 Thread Alvaro Herrera
Alvaro Herrera wrote: > This patch enables foreign key constraints to and from partitioned > tables. This version is rebased on current master. 0001: fix for a get_relation_info bug in current master. Posted in <20180124174134.ma4ui2kczmqwb4um@alvherre.pgsql> 0002: Allows local partitioned

Re: copy.c allocation constant

2018-01-24 Thread Robert Haas
On Wed, Jan 24, 2018 at 1:43 PM, Andres Freund wrote: > Indeed. Don't think RAW_BUF_SIZE is quite big enough for that on most > platforms though. From man mallopt: > Balancing these factors leads to a default setting of 128*1024 for the > M_MMAP_THRESHOLD parameter. > Additionally, even when

Re: Would a BGW need shmem_access or database_connection to enumerate databases?

2018-01-24 Thread Bruce Momjian
On Wed, Jan 24, 2018 at 01:48:05PM -0500, Chapman Flack wrote: > Thanks! I had actually registered that one (with a related one) > for CF 2018-03, having missed the deadline for -01: > > https://commitfest.postgresql.org/17/1467/ OK, thanks. I added a commitfest item annotiation to say that the

Re: WIP Patch: Precalculate stable functions, infrastructure v1

2018-01-24 Thread Andres Freund
Hi, On 2018-01-16 17:05:01 -0500, Tom Lane wrote: > [ I'm sending this comment separately because I think it's an issue > Andres might take an interest in. ] Thanks for that. I indeed am interested. Sorry for the late response, was very deep into the JIT patch. > Marina Polyakova writes: > >

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2018-01-24 Thread Tom Lane
Stephen Frost writes: > I'm afraid we may still get some push-back from existing users of > --clean since, with the change you're proposing, we wouldn't be cleaning > up anything that's been done to the public schema when it comes to > comment changes or ACL changes, right? No, if you have a nond

Re: Would a BGW need shmem_access or database_connection to enumerate databases?

2018-01-24 Thread Chapman Flack
Thanks! I had actually registered that one (with a related one) for CF 2018-03, having missed the deadline for -01: https://commitfest.postgresql.org/17/1467/ -Chap On 01/24/2018 01:20 PM, Bruce Momjian wrote: > On Thu, Dec 14, 2017 at 06:12:35PM -0500, Chapman Flack wrote: >> On 12/04/2017 09:1

Re: copy.c allocation constant

2018-01-24 Thread Andres Freund
On 2018-01-24 13:19:19 -0500, Robert Haas wrote: > On Wed, Jan 24, 2018 at 12:25 PM, Tomas Vondra > wrote: > > At the glibc level ... I'm not so sure. AFAIK glibc uses an allocator > > with similar ideas (freelists, ...) so hopefully it's fine too. > > > > And then there are the systems without gl

Re: [PROPOSAL] Shared Ispell dictionaries

2018-01-24 Thread Arthur Zakirov
2018-01-24 20:57 GMT+03:00 Tomas Vondra : > > Thanks. I don't have time to review/test this before FOSDEM, but a > couple of comments regarding some of the points you mentioned. > Thank you for your thoughts. > > I thought about it. And it seems to me that we can use functions > > ts_unload() an

Re: pgindent run?

2018-01-24 Thread Andres Freund
On 2018-01-23 22:22:47 -0500, Bruce Momjian wrote: > On Tue, Nov 28, 2017 at 04:38:12PM -0500, Tom Lane wrote: > > Thomas Munro writes: > > > On Wed, Nov 29, 2017 at 9:47 AM, Tom Lane wrote: > > >> I think that'd be taking it too far, especially given that the dependency > > >> on a typedefs list

Re: pgsql: Add parallel-aware hash joins.

2018-01-24 Thread Andres Freund
Hi, On 2018-01-24 13:11:22 -0500, Robert Haas wrote: > So for me, the additional hash index tests don't cost anything > measurable and the additional hash join tests cost about a second. I > think this probably accounts for why committers other than you keep > "adding so much time to the regressi

Re: PATCH: Exclude unlogged tables from base backups

2018-01-24 Thread David Steele
Hi Masahiko, Thanks for the review! On 1/22/18 3:14 AM, Masahiko Sawada wrote: > On Thu, Dec 14, 2017 at 11:58 PM, Robert Haas wrote: >> >> We would also have a problem if the missing file caused something in >> recovery to croak on the grounds that the file was expected to be >> there, but I do

Re: [HACKERS] Patch: Add --no-comments to skip COMMENTs with pg_dump

2018-01-24 Thread Stephen Frost
Tom, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> In further testing of that, I noticed that it made the behavior of our > >> other bugaboo, the public schema, rather inconsistent. With this > >> builtin-extensions hack, the plpgs

Re: FOR EACH ROW triggers on partitioned tables

2018-01-24 Thread Robert Haas
On Tue, Jan 23, 2018 at 5:10 PM, Alvaro Herrera wrote: > The main question is this: when running the trigger function, it is > going to look as it is running in the context of the partition, not in > the context of the parent partitioned table (TG_RELNAME etc). That > seems mildly ugly: some user

Re: Would a BGW need shmem_access or database_connection to enumerate databases?

2018-01-24 Thread Bruce Momjian
On Thu, Dec 14, 2017 at 06:12:35PM -0500, Chapman Flack wrote: > On 12/04/2017 09:13 AM, Craig Ringer wrote: > > On 1 December 2017 at 23:04, Chapman Flack wrote: > >> Can I call RegisterDynamicBackgroundWorker when not in the postmaster, > >> but also not in a "regular backend", but rather anothe

  1   2   >