Re: [HACKERS] proposal: get oldest LSN - function

2016-02-26 Thread Andres Freund
On 2016-02-26 18:45:14 +0300, Kartyshov Ivan wrote: > Function pg_oldest_xlog_location gets us the oldest LSN (Log Sequence > Number) in xlog. > > It is useful additional tool for DBA (we can get replicationSlotMinLSN, so > why not in master), it can show us, if xlog replication or wal-sender is >

Re: [HACKERS] Relation cache invalidation on replica

2016-02-26 Thread Andres Freund
On 2016-02-26 18:05:55 +0300, Konstantin Knizhnik wrote: > The reason of the problem is that invalidation messages are not delivered to > replica after the end of concurrent create index. > Invalidation messages are included in xlog as part of transaction commit > record. > Concurrent index create

Re: [HACKERS] Relation cache invalidation on replica

2016-02-26 Thread Andres Freund
On 2016-02-27 01:16:34 +, Simon Riggs wrote: > If the above is true, then the proposed fix wouldn't work either. > > No point in sending a cache invalidation message on the standby if you > haven't also written WAL, since the catalog re-read would just see the old > row. > > heap_inplace_upda

Re: [HACKERS] Relation cache invalidation on replica

2016-02-26 Thread Andres Freund
On 2016-02-27 01:45:57 +, Simon Riggs wrote: > Surely then the fix is to make heap_inplace_update() assign an xid? That > way any catalog change will always generate a commit record containing the > invalidation that goes with the change. No need to fix up the breakage > later. Well, we could,

Re: [HACKERS] Performance degradation in commit 6150a1b0

2016-02-26 Thread Andres Freund
On February 26, 2016 7:55:18 PM PST, Amit Kapila wrote: >On Sat, Feb 27, 2016 at 12:41 AM, Andres Freund >wrote: >> >> Hi, >> >> On 2016-02-25 12:56:39 +0530, Amit Kapila wrote: >> > From past few weeks, we were facing some performance degradation in >

Re: [HACKERS] Sanity checking for ./configure options?

2016-02-27 Thread Andres Freund
On 2016-02-27 14:15:45 -0600, Jim Nasby wrote: > Yeah, and I don't see any reasonable way to do that... we don't require sed > or the like, do we? We actually do. Check the bottom of configure.in. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subs

Re: [HACKERS] Proposal: Generic WAL logical messages

2016-02-28 Thread Andres Freund
Hi, On 2016-02-28 22:44:12 +0100, Petr Jelinek wrote: > On 27/02/16 01:05, Andres Freund wrote: > >I'm not really convinced by RegisterStandbyMsgPrefix() et al. There's > >not much documentation about what it actually is supposed to > >acomplish. Afaics

Re: [HACKERS] WIP: Upper planner pathification

2016-02-28 Thread Andres Freund
er hand, this is a major architectural step forward; one that unblocks a number of nice features. There's also an argument to be made that integrating this now is beneficial, because it'll cause less churn for patches being developed while 9.6 is stabilizing. Greetings, Andres Freun

Re: [HACKERS] snapshot too old, configured by time

2016-02-29 Thread Andres Freund
On 2016-02-29 18:30:27 -0600, Kevin Grittner wrote: > Basically, a connection needs to remain open and interleave > commands with other connections, which the isolation tester does > just fine; but it needs to do that using a custom postgresql.conf > file, which TAP does just fine. I haven't been

Re: [HACKERS] Confusing with commit time usage in logical decoding

2016-03-01 Thread Andres Freund
Hi, On 2016-03-01 18:09:28 +0100, Petr Jelinek wrote: > On 01/03/16 17:57, Alvaro Herrera wrote: > >Artur Zakirov wrote: > >>Hello, Andres > >> > >>You have introduced a large replication progress tracking infrastructure > >>last year. And there is a problem described at the link in the quote belo

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-03-01 Thread Andres Freund
On March 1, 2016 8:41:33 PM PST, Dilip Kumar wrote: >On Tue, Mar 1, 2016 at 10:19 AM, Dilip Kumar >wrote: > >> >> OK, I will test it, sometime in this week. >> > >I have tested this patch in my laptop, and there i did not see any >regression at 1 client > >Shared buffer 10GB, 5 mins run with pg

Re: [HACKERS] Confusing with commit time usage in logical decoding

2016-03-02 Thread Andres Freund
On 2016-03-01 18:31:42 +0100, Petr Jelinek wrote: > On 01/03/16 18:18, Andres Freund wrote: > >I'd rather just initialize commit_time to parsed->xact_time. > > > >This indeed is clearly a bug. I do wonder if anybody has a good idea > >about how to add reg

Re: [HACKERS] pg_basebackup compression TODO item

2016-03-03 Thread Andres Freund
On 2016-03-03 18:31:03 +0100, Magnus Hagander wrote: > I think we want it at protocol level rather than pg_basebackup level. I think we may want both eventually, but I do agree that protocol level has a lot higher "priority" than that. Something like protocol level compression has a bit of differe

Re: [HACKERS] pg_basebackup compression TODO item

2016-03-03 Thread Andres Freund
On 2016-03-03 18:44:24 +0100, Magnus Hagander wrote: > On Thu, Mar 3, 2016 at 6:34 PM, Andres Freund wrote: > > > On 2016-03-03 18:31:03 +0100, Magnus Hagander wrote: > > > I think we want it at protocol level rather than pg_basebackup level. > > > > I think we m

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-03 Thread Andres Freund
not about safely removing the old file. Currently I'm inclined to apply this to master soon. But I think we might want to wait a while with backpatching. The recent fsync upgrade disaster kinda makes me a bit careful... Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan support on readfuncs.c)

2016-03-04 Thread Andres Freund
On 2016-02-12 15:56:45 +0100, Andres Freund wrote: > Hi, > > > On 2016-02-10 23:26:20 -0500, Robert Haas wrote: > > I think the part about whacking around the FDW API is a little more > > potentially objectionable to others, so I want to hold off doing that > > unl

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-04 Thread Andres Freund
On 2016-03-04 14:51:50 +0900, Michael Paquier wrote: > On Fri, Mar 4, 2016 at 4:06 AM, Andres Freund wrote: > > Hi, > > Thanks for the review. > > >> +/* > >> + * rename_safe -- rename of a file, making it on-disk persistent > >> + * > >> +

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-04 Thread Andres Freund
On 2016-03-05 07:29:35 +0900, Michael Paquier wrote: > OK. I could produce that by tonight my time, not before unfortunately. I'm switching to this patch, after pushing the pending logical decoding fixes. Probably not today, but tomorrow PST afternoon should work. > And FWIW, per the comments of

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-04 Thread Andres Freund
On 2016-03-05 07:43:00 +0900, Michael Paquier wrote: > On Sat, Mar 5, 2016 at 7:35 AM, Andres Freund wrote: > > On 2016-03-04 14:51:50 +0900, Michael Paquier wrote: > >> On Fri, Mar 4, 2016 at 4:06 AM, Andres Freund wrote: > >> Hm. OK. I don't see any reason why

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-05 Thread Andres Freund
call fsync_fname which always errors out in case of failure. I also don't like the new messages much. Will continue working on this tomorrow. Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Badly designed error reporting code in controldata_utils.c

2016-03-06 Thread Andres Freund
tend. We've grown several hacks around that not being present, and I think those by now have a higher aggregate complexity than a proper implementation would have. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-06 Thread Andres Freund
Hi, On 2016-03-05 19:54:05 -0800, Andres Freund wrote: > I started working on this; delayed by taking longer than planned on the > logical decoding stuff (quite a bit complicated by > e1a11d93111ff3fba7a91f3f2ac0b0aca16909a8). I'm not very happy with the > error handling as it i

Re: [HACKERS] checkpointer continuous flushing - V16

2016-03-07 Thread Andres Freund
nfused. These are the commits you tested? Those aren't the ones doing sorting and flushing? > Also, I really wonder what will happen with non-default io schedulers. I > believe all the testing so far was done with cfq, so what happens on > machines that use e.g. "deadline&quo

Re: [HACKERS] checkpointer continuous flushing - V16

2016-03-07 Thread Andres Freund
On 2016-03-07 09:41:51 -0800, Andres Freund wrote: > > Due to the difference in amount of RAM, each machine used different scales - > > the goal is to have small, ~50% RAM, >200% RAM sizes: > > > > 1) Xeon: 100, 400, 6000 > > 2) i5: 50, 200, 3000 > &g

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-07 Thread Andres Freund
On 2016-02-22 20:44:35 +0100, Fabien COELHO wrote: > > >>Random updates on 16 tables which total to 1.1GB of data, so this is in > >>buffer, no significant "read" traffic. > >> > >>(1) with 16 tablespaces (1 per table) on 1 disk : 680.0 tps > >>per second avg, stddev [ min q1 median d3 max ] <

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-07 Thread Andres Freund
On 2016-03-07 21:10:19 +0100, Fabien COELHO wrote: > Now I cannot see how having one context per table space would have a > significant negative performance impact. The 'dirty data' etc. limits are global, not per block device. By having several contexts with unflushed dirty data the total amount

[HACKERS] Allowing to run a buildfarm animal under valgrind

2016-03-07 Thread Andres Freund
Hi, I'm setting up a buildfarm animal that runs under valgrind. Unfortunately there's not really any good solution to force make check et al. to start postgres wrapped in valgrind. For now I've resorted to adding something like sub replace_postgres { my $srcdir=$use_vpath ? "../pgsql/" : "."

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-07 Thread Andres Freund
s. I proposed a couple of potential regarding that > actually, see here: > http://www.postgresql.org/message-id/cab7npqsaz9hnucmoua30jo2wj8mnrem18p2a7mcra-zrjxj...@mail.gmail.com Yea. Will be an interesting discussion... Anyway, I did run the patch through the existing checks, after enabling fsync in PostgresNode.pm. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Badly designed error reporting code in controldata_utils.c

2016-03-07 Thread Andres Freund
On 2016-03-08 13:45:25 +0900, Michael Paquier wrote: > On Mon, Mar 7, 2016 at 10:26 AM, Andres Freund wrote: > > FWIW I'm considering implementing elog/ereport properly for the > > frontend. We've grown several hacks around that not being present, and > > I t

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-07 Thread Andres Freund
On 2016-03-08 12:26:34 +0900, Michael Paquier wrote: > On Tue, Mar 8, 2016 at 12:18 PM, Andres Freund wrote: > > On 2016-03-08 12:01:18 +0900, Michael Paquier wrote: > >> I have spent a couple of hours looking at that in details, and the > >> patch is neat. > > &g

Re: [HACKERS] Allowing to run a buildfarm animal under valgrind

2016-03-07 Thread Andres Freund
Hi, On 2016-03-07 17:39:30 -0800, Andres Freund wrote: > I'm setting up a buildfarm animal that runs under valgrind. Which is now running as 'skink'. The first failed due to a missing trick in the wrapper script, but the second one looks like it had a legit issue: http://pgbui

[HACKERS] empty array case in plperl_ref_from_pg_array not handled correctly

2016-03-07 Thread Andres Freund
it die reliably, without gdb. ISTM the assumption that an array always has a dimension is a bit more widespread... E.g. split_array() looks like it'd not work nicely with a zero dimensional array... Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-07 Thread Andres Freund
Hi, On 2016-03-08 16:21:45 +0900, Michael Paquier wrote: > + durable_link_or_rename(tmppath, path, ERROR); > + durable_rename(path, xlogfpath, ERROR); > You may want to add a (void) cast in front of those calls for correctness. "correctness"? This is neatnikism, not correctness. I've actual

Re: [HACKERS] Allowing to run a buildfarm animal under valgrind

2016-03-08 Thread Andres Freund
On 2016-03-08 08:58:22 -0500, Andrew Dunstan wrote: > On 03/07/2016 08:39 PM, Andres Freund wrote: > >Does anybody have a better idea about how to do this? > > Why not just create a make target which does this? It could be run after > 'make' and before 'ma

Re: [HACKERS] Allowing to run a buildfarm animal under valgrind

2016-03-08 Thread Andres Freund
On 2016-03-08 18:24:23 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > > I do wonder if adding a PGCTLPOSTGRESWRAPPER or something to pg_ctl > > would be ok. That'd just supplant calling the postgres binary, making > > all this easier. > > This seem

Re: [HACKERS] empty array case in plperl_ref_from_pg_array not handled correctly

2016-03-08 Thread Andres Freund
On 2016-03-08 02:11:03 -0700, Alex Hunsaker wrote: > On Mon, Mar 7, 2016 at 11:32 PM, Andres Freund wrote: > > > Hi, > > > > Per the new valgrind animal we get: > > > > > > http://pgbuildfarm.org/cgi-bin/show_log.pl?nm=skink&dt=2016-03-08%

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-08 Thread Andres Freund
On 2016-03-08 23:47:48 +0100, Tomas Vondra wrote: > I've repeated the power-loss testing today. With the patches applied I'm > not longer able to reproduce the issue (despite trying about 10x), while > without them I've hit it on the first try. This is on kernel 4.4.2. Yay, thanks for testing! An

Re: [HACKERS] Idle In Transaction Session Timeout, revived

2016-03-08 Thread Andres Freund
ed in the signal handler. I'm no targuing for the current order, just that one argument ;). FWIW, I think Vik wrote this after discussing with me, and IIRC there was not a lot of reasoning going into the specific location for doing this. Greetings, Andres Freund -- Sent via pgsql-hacker

Re: [HACKERS] fun with "Ready for Committer" patches

2016-03-08 Thread Andres Freund
On 2016-03-09 08:18:09 +0900, Tatsuo Ishii wrote: > > It's hard to miss the fact that there are an absolutely breathtaking > > number of patches in this CommitFest - 80! - that are in the "needs > > review" state. We really, really, really need more review to happen - > > Many of "needs review" s

Re: [HACKERS] the include-timestamp data returned by pg_logical_slot_peek_changes() is always 2000-01-01 in 9.5.1

2016-03-09 Thread Andres Freund
On March 9, 2016 4:26:01 AM PST, Craig Ringer wrote: >On 9 March 2016 at 18:13, 李海龙 wrote: > >> >> >> HI, pgsql-hackers >> >> The include-timestamp data returned by >pg_logical_slot_peek_changes() is >> always 2000-01-01 in 9.5.1, is it not normal? It's a bug in 9.5, that has been fixed. You

Re: [HACKERS] fun with "Ready for Committer" patches

2016-03-09 Thread Andres Freund
On 2016-03-10 06:14:25 +0900, Michael Paquier wrote: > IMO, during a review one needs to think of himself as a committer. > Once the reviewer switches the patch to "Ready for committer", it > means that the last version of the patch present would have been the > version that gained the right to be

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-09 Thread Andres Freund
On 2016-03-07 21:55:52 -0800, Andres Freund wrote: > Here's my updated version. > > Note that I've split the patch into two. One for the infrastructure, and > one for the callsites. I've finally pushed these, after making a number of mostly cosmetic fixes. The only

[HACKERS] pg_rewind just doesn't fsync *anything*?

2016-03-09 Thread Andres Freund
gainst a crash? I guess the easiest fix would be to shell out to initdb -s? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: Upper planner pathification

2016-03-10 Thread Andres Freund
(which will be open source in a few days) uses several of these (make_agg, make_sort_from_sortclauses, make_limit at least). Can we please re-consider making these static? It's fine if their parameter list changes from release to release, that's easy enough to adjust to, and it's easily

Re: [HACKERS] pg_rewind just doesn't fsync *anything*?

2016-03-10 Thread Andres Freund
Hi, On 2016-03-10 08:47:16 +0100, Michael Paquier wrote: > Still, I think that we had better fsync only entries that are modified > by pg_rewind, and files that got updated, and not the whole directory Why? If any files in there are dirty, they need to be fsynced. If they're not dirty, fsync's fr

Re: [HACKERS] WIP: Upper planner pathification

2016-03-10 Thread Andres Freund
On 2016-03-10 13:48:31 -0500, Tom Lane wrote: > Andres Freund writes: > > I see that you made a lot of formerly externally visible make_* routines > > static. The Citus extension (which will be open source in a few days) > > uses several of these (make_agg, make_sort_from_so

Re: [HACKERS] pg_rewind just doesn't fsync *anything*?

2016-03-10 Thread Andres Freund
On 2016-03-10 20:31:55 +0100, Michael Paquier wrote: > On Thu, Mar 10, 2016 at 7:52 PM, Andres Freund wrote: > > Having to backpatch a single system() invocation + find_other_exec() > > call, and backporting a more general FRONTEND version of initdb's > > fsync_pgdata

Re: [HACKERS] WIP: Upper planner pathification

2016-03-10 Thread Andres Freund
On 2016-03-10 14:16:03 -0500, Tom Lane wrote: > Andres Freund writes: > > In Citus' case a full PlannedStmt is generated on the master node, to > > combine the data generated on worker nodes (where the bog standard > > postgres planner is used). It's not t

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-10 Thread Andres Freund
for the latency hit incurred by the fsync()s from smgrsync(); at least by my measurement. My current plan is to commit this with the current behaviour (as in this week[end]), and then do some actual benchmarking on this specific part. It's imo a relatively minor detail. Greetings, Andr

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-10 Thread Andres Freund
On 2016-02-21 09:49:53 +0530, Robert Haas wrote: > I think there might be a semantic distinction between these two terms. > Doesn't writeback mean writing pages to disk, and flushing mean making > sure that they are durably on disk? So for example when the Linux > kernel thinks there is too much d

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-10 Thread Andres Freund
On 2016-03-10 17:33:33 -0500, Robert Haas wrote: > On Thu, Mar 10, 2016 at 5:24 PM, Andres Freund wrote: > > On 2016-02-21 09:49:53 +0530, Robert Haas wrote: > >> I think there might be a semantic distinction between these two terms. > >> Doesn't writeback mean wri

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-10 Thread Andres Freund
On 2016-03-10 23:38:38 +0100, Fabien COELHO wrote: > I'm not sure I've seen these performance... If you have hard evidence, > please feel free to share it. Man, are you intentionally trying to be hard to work with? To quote the email you responded to: > My current plan is to commit this with the

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-10 Thread Andres Freund
On 2016-03-10 23:43:46 +0100, Fabien COELHO wrote: > > > > >Whenever more than bgwriter_flush_after bytes have > >been written by the bgwriter, attempt to force the OS to issue these > >writes to the underlying storage. Doing so will limit the amount of > >d

Re: [HACKERS] checkpointer continuous flushing - V18

2016-03-10 Thread Andres Freund
On 2016-03-11 00:23:56 +0100, Fabien COELHO wrote: > As you wish. I thought that understanding the underlying performance model > with sequential writes written in chunks is important for the admin, and as > this guc would have an impact on performance it should be hinted about, > including the lim

Re: [HACKERS] checkpointer continuous flushing

2016-03-10 Thread Andres Freund
Hi, I just pushed the two major remaining patches in this thread. Let's see what the buildfarm has to say; I'd not be surprised if there's some lingering portability problem in the flushing code. There's one remaining issue we definitely want to resolve before the next release: Right now we alwa

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-03-10 Thread Andres Freund
othing. The stdbool > issues should be looked into, for the sake of Windows and other > future-proofness. But that will likely be an entirely different patch. That seems to entirely miss the part of this discussion dealing with high bits set and such? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-03-10 Thread Andres Freund
On 2016-03-11 04:50:45 +0100, Michael Paquier wrote: > On Thu, Mar 10, 2016 at 11:40 PM, Robert Haas wrote: > > We need to decide what to do about this. I disagree with Peter: I > > think that regardless of stdbool, what we've got right now is sloppy > > coding - bad style if nothing else. Furth

Re: [HACKERS] WIP: Upper planner pathification

2016-03-10 Thread Andres Freund
Hi, On 2016-03-10 15:03:41 -0500, Tom Lane wrote: > Andres Freund writes: > > Primarily because create_plan(), and/or its children, have to know about > > what you're doing; you can hide some, but not all, things below > > CustomScan nodes. > > And which of thos

[HACKERS] Background Processes and reporting

2016-03-11 Thread Andres Freund
f progress tracking to other types of background processes. Similarly for the wait event stuff - checkpointer, wal writer, background writer are in many cases processes that very often are blocked on locks, IO and such. Thus restricting the facility to database connected processes seems like a

Re: [HACKERS] Background Processes and reporting

2016-03-11 Thread Andres Freund
d patch tried to do too many things at once, and it's authors didn't listen well to criticism. Trying to go back to that seems like a surefire way to have nothing in 9.6. > Seems that current implementation doesn’t give reasonable ways to > implement all that features and it is r

Re: [HACKERS] Background Processes and reporting

2016-03-11 Thread Andres Freund
Hi, On 2016-03-11 11:16:32 -0800, Andres Freund wrote: > It seems rather worthwhile to think about how we can expand the coverage > of progress tracking to other types of background processes. WRT the progress reporting patch, I think we should split (as afaics was discussed in the thread

Re: [HACKERS] Background Processes and reporting

2016-03-11 Thread Andres Freund
On 2016-03-12 01:05:43 +0300, Vladimir Borodin wrote: > > 12 марта 2016 г., в 0:22, Andres Freund написал(а): > > Only that it isn't. It's stored in PGPROC. > > Sorry, I missed that. So monitoring of wait events for auxiliary processes > still could be implemen

Re: [HACKERS] WIP: Upper planner pathification

2016-03-11 Thread Andres Freund
On 2016-03-10 23:38:14 -0500, Tom Lane wrote: > > Would you rather add back the exports or should I? > > I'll do it ... just send me the list. After exporting make_agg, make_limit, make_sort_from_sortclauses and making some trivial adjustments due to the pull_var_clause changes change, Citus' tes

Re: [HACKERS] Background Processes and reporting

2016-03-11 Thread Andres Freund
conflicts with Robert. But if I were in his shoes, targeted by this kind of rhetoric, I'd be very tempted to just ignore any further arguments from the origin. So I think the way forward is for everyone to cool off, and to see how we can incrementally make progress from here on.

[HACKERS] Re: [COMMITTERS] pgsql: Only try to push down foreign joins if the user mapping OIDs mat

2016-03-11 Thread Andres Freund
Hi, On 2016-01-28 19:09:01 +, Robert Haas wrote: > Only try to push down foreign joins if the user mapping OIDs match. > > Previously, the foreign join pushdown infrastructure left the question > of security entirely up to individual FDWs, but it would be easy for > a foreign data wrapper to

[HACKERS] Re: [COMMITTERS] pgsql: Only try to push down foreign joins if the user mapping OIDs mat

2016-03-12 Thread Andres Freund
Hi, On 2016-03-12 11:56:24 -0500, Robert Haas wrote: > On Fri, Mar 11, 2016 at 10:15 PM, Andres Freund wrote: > > On 2016-01-28 19:09:01 +, Robert Haas wrote: > >> Only try to push down foreign joins if the user mapping OIDs match. > >> > >> P

Re: [HACKERS] WIP: Upper planner pathification

2016-03-12 Thread Andres Freund
On 2016-03-12 12:22:01 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2016-03-10 23:38:14 -0500, Tom Lane wrote: > >> I'll do it ... just send me the list. > > > After exporting make_agg, make_limit, make_sort_from_sortclauses and > > maki

Re: [HACKERS] Background Processes and reporting

2016-03-14 Thread Andres Freund
On 2016-03-12 16:29:11 +0530, Amit Kapila wrote: > On Sat, Mar 12, 2016 at 3:10 AM, Andres Freund wrote: > > > > > > > Similarly for the wait event stuff - checkpointer, wal writer, > > > background writer are in many cases processes that very often are > >

Re: [HACKERS] Background Processes and reporting

2016-03-14 Thread Andres Freund
Hi, On 2016-03-14 16:16:43 -0400, Robert Haas wrote: > > I have already shown [0, 1] the overhead of measuring timings in linux on > > representative workload. AFAIK, these tests were the only one that showed > > any numbers. All other statements about terrible performance have been and > > remain

Re: [HACKERS] pglogical_output - a general purpose logical decoding output plugin

2016-03-14 Thread Andres Freund
On 2016-01-31 05:09:33 +0800, Craig Ringer wrote: > On 29 January 2016 at 18:16, Andres Freund wrote: > > > Hi, > > > > so, I'm reviewing the output of: > > > > Thankyou very much for the review. Afaics you've not posted an updated version of t

Re: [HACKERS] Timeline following for logical slots

2016-03-14 Thread Andres Freund
On 2016-03-14 20:10:58 -0300, Alvaro Herrera wrote: > Great, thanks. I've studied this to the point where I'm confident that > it makes sense, so I'm about to push it. I didn't change any logic, > only updated comments here and there, both in the patch and in some > preexisting code. I also chan

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-03-14 Thread Andres Freund
On 2016-03-14 17:17:02 -0700, Peter Geoghegan wrote: > On Mon, Mar 14, 2016 at 5:04 PM, Robert Haas wrote: > > There hasn't been a new version of this patch in 9 months, you're > > clearly not in a hurry to produce one, and nobody else seems to feel > > strongly that this is something that needs t

Re: [HACKERS] Timeline following for logical slots

2016-03-15 Thread Andres Freund
the client. > @@ -299,6 +312,18 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo > fcinfo, bool confirm, bool bin > CHECK_FOR_INTERRUPTS(); > } > > + /* Make sure timeline lookups use the start of the next record > *

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-15 Thread Andres Freund
; + /* > + * To guarantee renaming of the file is persistent, fsync the file with > its > + * new name, and its containing directory. > + */ > + fd = open(newfile, O_RDWR | PG_BINARY, S_IRUSR | S_IWUSR); Why is S_IRUSR | S_IWUSR specified here? Are you

Re: [HACKERS] Idle In Transaction Session Timeout, revived

2016-03-15 Thread Andres Freund
On 2016-03-15 14:21:34 -0400, Robert Haas wrote: > On Tue, Mar 8, 2016 at 6:08 PM, Andres Freund wrote: > > On 2016-03-08 16:42:37 -0500, Robert Haas wrote: > >> - I really wonder if the decision to ignore sessions that are idle in > >> transaction (aborted) shoul

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-18 Thread Andres Freund
n it. The patch is pgindented (I'd personally never decrease indentation just to fit a line into 79 chars, as pgindent does...). Thanks for looking! Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] logger process infinite loop

2016-03-18 Thread Andres Freund
is: > > commit c4901a1e03a7730e4471fd1143f1caf79695493d > Author: Andres Freund > Date: Fri Mar 18 11:43:59 2016 -0700 > > Only clear latch self-pipe/event if there is a pending notification. God, whan an awfully stupid mistake/typo. Thanks for the report! - Andres -- Sent via pgsql-ha

Re: [HACKERS] logger process infinite loop

2016-03-18 Thread Andres Freund
On 2016-03-18 22:39:25 -0700, Andres Freund wrote: > Hi, > > On 2016-03-18 21:59:01 -0700, Jeff Janes wrote: > > While testing some patches on my laptop, I noticed my knee getting > > uncomfortably warm. It turns out I has accumulating deranged logging > > processes,

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-18 Thread Andres Freund
On March 18, 2016 11:32:53 PM PDT, Amit Kapila wrote: >On Sat, Mar 19, 2016 at 12:00 AM, Andres Freund >wrote: >> >> On 2016-03-18 20:14:07 +0530, Amit Kapila wrote: >> >> > I have done some >> > tests on Windows with 0003 patch which includes runn

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
On March 18, 2016 11:52:08 PM PDT, Amit Kapila wrote: >On Sat, Mar 19, 2016 at 12:14 PM, Andres Freund >wrote: >> >> >> >> On March 18, 2016 11:32:53 PM PDT, Amit Kapila > >wrote: >> >On Sat, Mar 19, 2016 at 12:00 AM, Andres Freund >> >

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
ickly as possible. cur_events points to the event that's registered with the set, not the one that's "returned" or "modified" by poll/select, that's where the confusion is originating from. I'll add a fastpath. Greetings, Andres Freund -- Sent via pgsql-ha

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
On 2016-03-16 15:41:28 -0400, Robert Haas wrote: > On Wed, Mar 16, 2016 at 3:25 PM, Andres Freund wrote: > >> > - Given current users we don't need a large amount of events, so having > >> > to iterate through the registered events doesn't seem bothersome.

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
could you check whether 3/4/5 work? It's quite likely that I've screwed up something. Robert you'd mentioned on IM that you've a use-case for this somewhere around multiple FDWs. If somebody has started working on that, could you ask that person to check whether the API make

Re: [HACKERS] fd.c doesn't remove files on a crash-restart

2016-03-19 Thread Andres Freund
On 2016-03-16 10:53:42 -0700, Joshua D. Drake wrote: > Hello, > > fd.c[1] will remove files from pgsql_tmp on a restart but not a > crash-restart per this comment: > > /* > * NOTE: we could, but don't, call this during a post-backend-crash restart > * cycle. The argument for not doing it is that

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-19 Thread Andres Freund
e not syncing the directories, despite open_target_file() potentially creating the directory. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
On 2016-03-18 20:14:07 +0530, Amit Kapila wrote: > + * from inside a signal handler in latch_sigusr1_handler(). > > * > > * Note: we assume that the kernel calls involved in drainSelfPipe() > > * and SetLatch() will provide adequate synchronization on machines > > * with weak memory ord

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2016-03-19 Thread Andres Freund
On 2016-03-18 11:01:04 -0400, Robert Haas wrote: > On Thu, Mar 17, 2016 at 9:22 PM, Michael Paquier > wrote: > > FWIW, my instinctive thought on the matter is to report the event > > directly in WaitLatch() via a name of the event caller provided > > directly in it. The category of the event is th

Re: [HACKERS] fd.c doesn't remove files on a crash-restart

2016-03-19 Thread Andres Freund
On 2016-03-16 11:06:23 -0700, Joshua D. Drake wrote: > On 03/16/2016 11:04 AM, Andres Freund wrote: > >On 2016-03-16 11:02:09 -0700, Joshua D. Drake wrote: > >>>>3. The problem can get worse over time. If you have a very long running > >>>>instance, any time

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
Hi, On 2016-03-16 15:41:28 -0400, Robert Haas wrote: > On Wed, Mar 16, 2016 at 3:25 PM, Andres Freund wrote: > >> > - Given current users we don't need a large amount of events, so having > >> > to iterate through the registered events doesn't seem bothers

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
On 2016-01-14 12:07:23 -0500, Robert Haas wrote: > On Thu, Jan 14, 2016 at 12:06 PM, Andres Freund wrote: > > On 2016-01-14 11:31:03 -0500, Robert Haas wrote: > >> On Thu, Jan 14, 2016 at 10:56 AM, Andres Freund wrote: > >> I think your idea of a data structure the e

Re: [HACKERS] POC: Cache data in GetSnapshotData()

2016-03-19 Thread Andres Freund
On 2016-03-16 13:29:22 -0400, Robert Haas wrote: > Whoa. At 64 clients, we're hardly getting any benefit, but then by 88 > clients, we're getting a huge benefit. I wonder why there's that sharp > change there. What's the specifics of the machine tested? I wonder if it either correlates with the

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
[] > appears to be unrelated to the ostensible purpose of the patch. Well, not really. We need to know when to do drainSelfPipe(); Which gets more complicated if pfds[0] is registered optionally. I'm actually considering to drop this entirely, given the much heavier rework in the WaitEvent

Re: [HACKERS] fd.c: flush data problems on osx

2016-03-19 Thread Andres Freund
rather do an lseek(SEEK_END) to get the file size. Could you test that? > Also there are no default ifdef inside this function, is there any > check that will guarantee that pg_flush_data will not end up with > empty body on some platform? There doesn't need to be - it's purely &qu

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
s would get noticeably more complex that way. And a number of them will benefit from latches using epoll internally. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
On 2016-03-19 15:43:27 +0530, Amit Kapila wrote: > On Sat, Mar 19, 2016 at 12:40 PM, Andres Freund wrote: > > > > On March 18, 2016 11:52:08 PM PDT, Amit Kapila > wrote: > > >> >Won't the new code needs to ensure that ResetEvent(latchevent) > >

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
On 2016-03-19 16:44:49 +0530, Amit Kapila wrote: > On Fri, Mar 18, 2016 at 1:34 PM, Andres Freund wrote: > > > > > > Attached is a significantly revised version of the earlier series. Most > > importantly I have: > > * Unified the window/unix latch i

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
On 2016-03-16 15:08:07 -0400, Robert Haas wrote: > On Wed, Mar 16, 2016 at 2:52 PM, Andres Freund wrote: > > How about the following sketch of an API > > > > typedef struct LatchEvent > > { > > uint32 events; /* interesting events */ > >

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
On 2016-03-19 18:45:36 -0700, Andres Freund wrote: > On 2016-03-19 16:44:49 +0530, Amit Kapila wrote: > > On Fri, Mar 18, 2016 at 1:34 PM, Andres Freund wrote: > > > > > > > > > Attached is a significantly revised version of the earlier series. Most > >

Re: [HACKERS] fd.c doesn't remove files on a crash-restart

2016-03-19 Thread Andres Freund
On 2016-03-16 11:02:09 -0700, Joshua D. Drake wrote: > >>3. The problem can get worse over time. If you have a very long running > >>instance, any time the backend crash-restarts you have to potential to > >>increase disk space used for no purpose. > > > >But I think these outweigh the debugging be

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-19 Thread Andres Freund
h if so. You mean in WaitEventSetWait()? There's else if (rc == 0) { break; } which is the timeout case. There should never be any other case of returning 0 elements? > That's all I've got for now. Thanks for looking. Greeti

  1   2   3   4   5   6   7   8   9   10   >