Re: [HACKERS] Async execution of postgres_fdw.

2015-01-14 Thread Kyotaro HORIGUCHI
Hello, > - The issue of PREPARE is not addressed yet. ... > > In case of Prepared statements, ExecInit is called at the end of planning, > > without subsequent execution like the case of EXPLAIN. I see that the patch > > handles EXPLAIN well, but I didn't see any specific code for PREPARE. > > I

Re: [HACKERS] INSERT ... ON CONFLICT UPDATE and RLS

2015-01-14 Thread Dean Rasheed
On 12 January 2015 at 14:24, Stephen Frost wrote: > Interesting, thanks for the work! I had been suspicious that there was > an issue with the recursion handling. > So continuing to review the RLS code, I spotted the following in prepend_row_security_policies(): /* * We may end up gett

Re: [HACKERS] pg_rewind in contrib

2015-01-14 Thread Heikki Linnakangas
On 01/07/2015 06:19 PM, Alvaro Herrera wrote: Fixed most of the issues you listed. More on a few remaining ones below. Please don't name files with generic names such as util.c/h. It's annoying later; for instance when I want to open analyze.c I have to choose the one in parser or commands. (

Re: [HACKERS] pg_rewind in contrib

2015-01-14 Thread Andres Freund
On 2015-01-14 11:43:00 +0200, Heikki Linnakangas wrote: > No. The question is, should pg_rewind fsync() every file that it > modifies? Not immediately, but before the end, yes. > It would be a reasonable thing to do, to make sure that if you crash > immediately after running pg_rewind, the cluste

Re: [HACKERS] pg_rewind in contrib

2015-01-14 Thread Andres Freund
On 2015-01-09 10:48:39 +0200, Heikki Linnakangas wrote: > It would nevertheless be handy to be able to do more stuff in a replication > connection. For example, you might want to run functions like > pg_create_restore_point(), pg_xlog_replay_pause/resume etc. in a replication > connection. We shoul

Re: [HACKERS] pg_rewind in contrib

2015-01-14 Thread Andres Freund
On 2015-01-14 10:53:40 +0100, Andres Freund wrote: > On 2015-01-14 11:43:00 +0200, Heikki Linnakangas wrote: > > I'm not thrilled about sprinkling fsync() calls everywhere that we touch > > files. But I guess that would be the right thing to do. I'm planning to do > > that as an add-on patch later,

Re: [HACKERS] initdb -S and tablespaces

2015-01-14 Thread Andres Freund
Hi, On 2014-11-06 17:56:53 +0530, Abhijit Menon-Sen wrote: > + /* > + * If we need to perform crash recovery, we issue an fsync on the > + * data directory and its contents to try to ensure that any data > + * written before the crash are flushed to disk. Otherwise a power > +

Re: [HACKERS] Minor configure tweak to simplify adjusting gcc warnings

2015-01-14 Thread Andres Freund
Hi Tom, On 2015-01-13 22:19:30 -0500, Tom Lane wrote: > Would anyone object to modifying configure.in like this: > > if test "$GCC" = yes -a "$ICC" = no; then > - CFLAGS="$CFLAGS -Wall -Wmissing-prototypes -Wpointer-arith" > + CFLAGS="-Wall $CFLAGS -Wmissing-prototypes -Wpointer-arith" >#

[HACKERS] Out-of-bounds write and incorrect detection of trigger file in pg_standby

2015-01-14 Thread Michael Paquier
Hi all, In pg_standby.c, we use a 32-byte buffer in CheckForExternalTrigger to which is read the content of the trigger file defined by -f: CheckForExternalTrigger(void) { charbuf[32]; [...] if ((len = read(fd, buf, sizeof(buf))) < 0) { stuff(); } if (len <

Re: [HACKERS] Turning recovery.conf into GUCs

2015-01-14 Thread Robert Haas
On Mon, Jan 5, 2015 at 8:43 PM, Peter Eisentraut wrote: > I have previously argued for a different approach: Ready recovery.conf > as a configuration file after postgresql.conf, but keep it as a file to > start recovery. That doesn't break any old workflows, it gets you all > the advantages of ha

Re: [HACKERS] Improving RLS qual pushdown

2015-01-14 Thread Robert Haas
On Fri, Jan 9, 2015 at 7:54 AM, Dean Rasheed wrote: > A while ago [1] I proposed an enhancement to the way qual pushdown > safety is decided in RLS / security barrier views. Currently we just > test for the presence of leaky functions in the qual, but it is > possible to do better than that, by fu

Re: [HACKERS] libpq bad async behaviour

2015-01-14 Thread Robert Haas
On Fri, Jan 9, 2015 at 2:57 PM, Daurnimator wrote: > I'm worried about libpq blocking in some circumstances; particularly > around SSL renegotiations. > This came up while writing an async postgres library for lua, I > realised that this code was dangerous: > https://github.com/daurnimator/cqueues

Re: [HACKERS] Possible typo in create_policy.sgml

2015-01-14 Thread Robert Haas
On Fri, Jan 9, 2015 at 3:46 PM, Stephen Frost wrote: >A policy permits SELECT, INSERT, UPDATE or DELETE commands to access rows >in a table that has row level security enabled. Access to existing table >rows is granted if they match a policy expression specified via USING, >while

Re: [HACKERS] libpq bad async behaviour

2015-01-14 Thread Andres Freund
On 2015-01-14 08:32:19 -0500, Robert Haas wrote: > On Fri, Jan 9, 2015 at 2:57 PM, Daurnimator wrote: > > I'm worried about libpq blocking in some circumstances; particularly > > around SSL renegotiations. > > This came up while writing an async postgres library for lua, I > > realised that this c

Re: [HACKERS] Turning recovery.conf into GUCs

2015-01-14 Thread Petr Jelinek
On 14/01/15 14:24, Robert Haas wrote: On Mon, Jan 5, 2015 at 8:43 PM, Peter Eisentraut wrote: I have previously argued for a different approach: Ready recovery.conf as a configuration file after postgresql.conf, but keep it as a file to start recovery. That doesn't break any old workflows, it

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Merlin Moncure
On Tue, Jan 13, 2015 at 7:24 PM, Peter Geoghegan wrote: > On Tue, Jan 13, 2015 at 3:54 PM, Merlin Moncure wrote: >> Some more information what's happening: >> This is a ghetto logical replication engine that migrates data from >> sql sever to postgres, consolidating a sharded database into a sing

Re: [HACKERS] Patch: [BUGS] BUG #12320: json parsing with embedded double quotes

2015-01-14 Thread Robert Haas
On Thu, Jan 8, 2015 at 8:42 PM, Aaron Botsis wrote: > This version: > > * doesn't allow ENCODING in BINARY mode (existing bug) > * doesn’t allow ESCAPE in BINARY mode Hmm, it does look like ENCODING and ESCAPE don't do anything in BINARY mode, so adding error checks for that makes sense to me. I

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Merlin Moncure
On Wed, Jan 14, 2015 at 8:03 AM, Merlin Moncure wrote: > Here's a backtrace: > > #0 0x00750a97 in spin_delay () > #1 0x00750b19 in s_lock () > #2 0x00750844 in LWLockRelease () > #3 0x0073 in LockBuffer () > #4 0x004b2db4 in _bt_relandgetbuf () > #5

Re: [HACKERS] Improving RLS qual pushdown

2015-01-14 Thread Dean Rasheed
On 14 January 2015 at 13:29, Robert Haas wrote: > One thing they could still leak is the number of times they got > called, and thus possibly the number of unseen rows. Now if the > expressions get constant-folded away that won't be an issue, but a > clever user can probably avoid that. > Right

Re: [HACKERS] Minor configure tweak to simplify adjusting gcc warnings

2015-01-14 Thread Tom Lane
Andres Freund writes: > On 2015-01-13 22:19:30 -0500, Tom Lane wrote: >> The reason I got interested in this is that I attempted to pass in >> "CFLAGS=-Wno-format" to configure, to suppress format warnings on >> buildfarm member gaur (whose gcc is too old to recognize z modifiers). >> That doesn't

Re: [HACKERS] Minor configure tweak to simplify adjusting gcc warnings

2015-01-14 Thread Andres Freund
On 2015-01-14 09:34:23 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2015-01-13 22:19:30 -0500, Tom Lane wrote: > >> The reason I got interested in this is that I attempted to pass in > >> "CFLAGS=-Wno-format" to configure, to suppress format warnings on > >> buildfarm member gaur (whose g

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Tom Lane
Merlin Moncure writes: > There were seven process with that backtrace exact backtrace (except > that randomly they are sleeping in the spinloop). Something else > interesting: autovacuum has been running all night as well. Unlike > the other process however, cpu utilization does not register on

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Merlin Moncure
On Wed, Jan 14, 2015 at 8:41 AM, Tom Lane wrote: > Merlin Moncure writes: >> There were seven process with that backtrace exact backtrace (except >> that randomly they are sleeping in the spinloop). Something else >> interesting: autovacuum has been running all night as well. Unlike >> the oth

Re: [HACKERS] Minor configure tweak to simplify adjusting gcc warnings

2015-01-14 Thread Tom Lane
Andres Freund writes: > On 2015-01-14 09:34:23 -0500, Tom Lane wrote: >> Well, that would only fix my problem if we added a configure-time test >> for whether gcc recognizes "z", which frankly seems like a waste of >> cycles. I've probably got the last one left in captivity that doesn't. > Hm. I

Re: [HACKERS] Minor configure tweak to simplify adjusting gcc warnings

2015-01-14 Thread Andres Freund
On 2015-01-14 10:01:39 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2015-01-14 09:34:23 -0500, Tom Lane wrote: > >> Well, that would only fix my problem if we added a configure-time test > >> for whether gcc recognizes "z", which frankly seems like a waste of > >> cycles. I've probably g

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Tom Lane
Merlin Moncure writes: > On Wed, Jan 14, 2015 at 8:41 AM, Tom Lane wrote: >> What are the autovac processes doing (according to pg_stat_activity)? > pid,running,waiting,query > 7105,00:28:40.789221,f,autovacuum: VACUUM ANALYZE pg_catalog.pg_class Hah, I suspected as much. Is that the one that'

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Merlin Moncure
On Wed, Jan 14, 2015 at 9:05 AM, Tom Lane wrote: > Merlin Moncure writes: >> On Wed, Jan 14, 2015 at 8:41 AM, Tom Lane wrote: >>> What are the autovac processes doing (according to pg_stat_activity)? > >> pid,running,waiting,query >> 7105,00:28:40.789221,f,autovacuum: VACUUM ANALYZE pg_catalog.p

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Andres Freund
On 2015-01-14 10:05:01 -0500, Tom Lane wrote: > Merlin Moncure writes: > > On Wed, Jan 14, 2015 at 8:41 AM, Tom Lane wrote: > >> What are the autovac processes doing (according to pg_stat_activity)? > > > pid,running,waiting,query > > 7105,00:28:40.789221,f,autovacuum: VACUUM ANALYZE pg_catalog.

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Tom Lane
Merlin Moncure writes: > Yes, it is pg_class is coming from LockBufferForCleanup (). As you > can see above, it has a shorter runtime. So it was killed off once > about a half hour ago which did not free up the logjam. However, AV > spawned it again and now it does not respond to cancel. Int

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Andres Freund
On 2015-01-14 10:13:32 -0500, Tom Lane wrote: > Merlin Moncure writes: > > Yes, it is pg_class is coming from LockBufferForCleanup (). As you > > can see above, it has a shorter runtime. So it was killed off once > > about a half hour ago which did not free up the logjam. However, AV > > spaw

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Tom Lane
Andres Freund writes: > On 2015-01-14 10:05:01 -0500, Tom Lane wrote: >> Hah, I suspected as much. Is that the one that's stuck in >> LockBufferForCleanup, or the other one that's got a similar backtrace >> to all the user processes? > Do you have a theory? Right now it primarily looks like cont

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Merlin Moncure
On Wed, Jan 14, 2015 at 9:11 AM, Andres Freund wrote: > On 2015-01-14 10:05:01 -0500, Tom Lane wrote: >> Merlin Moncure writes: >> > On Wed, Jan 14, 2015 at 8:41 AM, Tom Lane wrote: >> >> What are the autovac processes doing (according to pg_stat_activity)? >> >> > pid,running,waiting,query >> >

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Andres Freund
On 2015-01-14 09:22:45 -0600, Merlin Moncure wrote: > On Wed, Jan 14, 2015 at 9:11 AM, Andres Freund wrote: > > On 2015-01-14 10:05:01 -0500, Tom Lane wrote: > >> Merlin Moncure writes: > >> > On Wed, Jan 14, 2015 at 8:41 AM, Tom Lane wrote: > >> >> What are the autovac processes doing (accordin

Re: [HACKERS] [PATCH] explain sortorder

2015-01-14 Thread Heikki Linnakangas
On 01/14/2015 05:26 PM, Timmer, Marius wrote: Hello Heikki, abbreviated version: Sorry, the problem is only the unhandy patch text format, not different opinions how to proceed. Long version: The v7 patch file already addressed your suggestions, but the file contained serveral (old) local comm

[HACKERS] Merging postgresql.conf and postgresql.auto.conf

2015-01-14 Thread Sawada Masahiko
Hi all, The postgresql.auto.conf is loaded after loading of postgresql.conf whenever configuration file is loaded or reloaded. This means that parameter in postgresql.auto.conf is quite high priority, so the parameter in postgresql.conf does not work at all even if user set it manually. If user w

Re: [HACKERS] Merging postgresql.conf and postgresql.auto.conf

2015-01-14 Thread Tom Lane
Sawada Masahiko writes: > The postgresql.auto.conf is loaded after loading of postgresql.conf > whenever configuration file is loaded or reloaded. > This means that parameter in postgresql.auto.conf is quite high > priority, so the parameter in postgresql.conf does not work at all > even if user s

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Merlin Moncure
On Wed, Jan 14, 2015 at 9:30 AM, Andres Freund wrote: > If you gdb in, and type 'fin' a couple times, to wait till the function > finishes, is there actually any progress? I'm wondering whether it's > just many catalog accesses + contention, or some other > problem. Alternatively set a breakpoint

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Andres Freund
On 2015-01-14 09:47:19 -0600, Merlin Moncure wrote: > On Wed, Jan 14, 2015 at 9:30 AM, Andres Freund wrote: > > If you gdb in, and type 'fin' a couple times, to wait till the function > > finishes, is there actually any progress? I'm wondering whether it's > > just many catalog accesses + contenti

Re: [HACKERS] Minor configure tweak to simplify adjusting gcc warnings

2015-01-14 Thread Tom Lane
Andres Freund writes: > I've already given up... Given how infrequent it is, suppressing it for > gull seems sufficient. I'm confused --- I see no format warnings in gull's current reports. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgre

Re: [HACKERS] Merging postgresql.conf and postgresql.auto.conf

2015-01-14 Thread Sawada Masahiko
On Thu, Jan 15, 2015 at 12:37 AM, Tom Lane wrote: > Sawada Masahiko writes: >> The postgresql.auto.conf is loaded after loading of postgresql.conf >> whenever configuration file is loaded or reloaded. >> This means that parameter in postgresql.auto.conf is quite high >> priority, so the parameter

Re: [HACKERS] Minor configure tweak to simplify adjusting gcc warnings

2015-01-14 Thread Andres Freund
On 2015-01-14 11:09:10 -0500, Tom Lane wrote: > Andres Freund writes: > > I've already given up... Given how infrequent it is, suppressing it for > > gull seems sufficient. > > I'm confused --- I see no format warnings in gull's current reports. Sorry it was me being confused. I somehow switched

Re: [HACKERS] Merging postgresql.conf and postgresql.auto.conf

2015-01-14 Thread Tom Lane
Sawada Masahiko writes: > On Thu, Jan 15, 2015 at 12:37 AM, Tom Lane wrote: >> Yes. The entire reason that postgresql.auto.conf is separate is that >> we despaired of reading and rewriting postgresql.conf automatically >> without making a hash of material in the comments. Calling the logic >> a

Re: [HACKERS] [RFC] Incremental backup v3: incremental PoC

2015-01-14 Thread Gabriele Bartolini
Hi Marco, thank you for sending an updated patch. I am writing down a report of this initial (and partial) review. IMPORTANT: This patch is not complete, as stated by Marco. See the "Conclusions" section for my proposed TODO list. == Patch application I have been able to successfully apply yo

Re: [HACKERS] Merging postgresql.conf and postgresql.auto.conf

2015-01-14 Thread Sawada Masahiko
On Thu, Jan 15, 2015 at 1:15 AM, Tom Lane wrote: > Sawada Masahiko writes: >> On Thu, Jan 15, 2015 at 12:37 AM, Tom Lane wrote: >>> Yes. The entire reason that postgresql.auto.conf is separate is that >>> we despaired of reading and rewriting postgresql.conf automatically >>> without making a h

Re: [HACKERS] Re: Patch to add functionality to specify ORDER BY in CREATE FUNCTION for SRFs

2015-01-14 Thread Robert Haas
On Mon, Jan 5, 2015 at 12:54 PM, Tom Lane wrote: > TBH, my first reaction to this entire patch is unfavorable: it's a > solution in search of a problem. It adds substantial complication not > only for users but for PG developers in order to solve a rather narrow > performance issue. > > What woul

Re: [HACKERS] Compression of full-page-writes

2015-01-14 Thread Robert Haas
On Fri, Jan 2, 2015 at 11:52 AM, Bruce Momjian wrote: > OK, so given your stats, the feature give a 12.5% reduction in I/O. If > that is significant, shouldn't we see a performance improvement? If we > don't see a performance improvement, is I/O reduction worthwhile? Is it > valuable in that it

Re: [HACKERS] Merging postgresql.conf and postgresql.auto.conf

2015-01-14 Thread Glyn Astill
- Original Message - > From: Sawada Masahiko > To: Tom Lane > Cc: PostgreSQL-development > Sent: Wednesday, 14 January 2015, 16:09 > Subject: Re: [HACKERS] Merging postgresql.conf and postgresql.auto.conf > > On Thu, Jan 15, 2015 at 12:37 AM, Tom Lane wrote: >> Sawada Masahiko writ

Re: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2015-01-14 Thread Robert Haas
On Fri, Jan 9, 2015 at 10:51 AM, Kouhei Kaigai wrote: > When custom-scan node replaced a join-plan, it shall have at least two > child plan-nodes. The callback handler of PlanCustomPath needs to be > able to call create_plan_recurse() to transform the underlying path-nodes > to plan-nodes, because

Re: [HACKERS] ereport bug

2015-01-14 Thread Robert Haas
On Mon, Jan 12, 2015 at 6:27 AM, Dmitry Voronin wrote: > I am attaching to this letter a test case that shows the behavior > errcontext() macro and the way to fix it. So the upshot of this is that given errfinish(A, B, C), where A, B, and C are expressions, my gcc is choosing to evaluate C, then

Re: [HACKERS] s_lock.h default definitions are rather confused

2015-01-14 Thread Robert Haas
On Mon, Jan 12, 2015 at 12:57 PM, Andres Freund wrote: > On 2015-01-10 23:03:36 +0100, Andres Freund wrote: >> On 2015-01-10 16:09:42 -0500, Tom Lane wrote: >> > I've not tried to build HEAD on my HPPA dinosaur for awhile, but I did >> > just now, and I am presented with boatloads of this: >> > >>

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Peter Geoghegan
On Wed, Jan 14, 2015 at 7:22 AM, Merlin Moncure wrote: > I'll try to pull commits that Peter suggested and see if that helps > (I'm getting ready to bring the database down). I can send the code > off-list if you guys think it'd help. Thanks for the code! I think it would be interesting to see

Re: [HACKERS] ereport bug

2015-01-14 Thread Tom Lane
Robert Haas writes: > On Mon, Jan 12, 2015 at 6:27 AM, Dmitry Voronin > wrote: >> I am attaching to this letter a test case that shows the behavior >> errcontext() macro and the way to fix it. > So the upshot of this is that given errfinish(A, B, C), where A, B, > and C are expressions, my gcc i

Re: [HACKERS] libpq bad async behaviour

2015-01-14 Thread Daurnimator
On 14 January 2015 at 08:40, Andres Freund wrote: > I think that kind of solution isn't likely to be satisfying. The amount > of porting work is just not going to be worth the cost. And it won't be > easily hideable in the API at all as the callers will expect a normal > fd. > All that consumers

Re: [HACKERS] INSERT ... ON CONFLICT UPDATE and RLS

2015-01-14 Thread Dean Rasheed
On 14 January 2015 at 08:43, Dean Rasheed wrote: > On 12 January 2015 at 14:24, Stephen Frost wrote: >> Interesting, thanks for the work! I had been suspicious that there was >> an issue with the recursion handling. >> > > So continuing to review the RLS code, I spotted the following in > prepen

Re: [HACKERS] ereport bug

2015-01-14 Thread Robert Haas
On Wed, Jan 14, 2015 at 1:40 PM, Tom Lane wrote: > Robert Haas writes: >> On Mon, Jan 12, 2015 at 6:27 AM, Dmitry Voronin >> wrote: >>> I am attaching to this letter a test case that shows the behavior >>> errcontext() macro and the way to fix it. > >> So the upshot of this is that given errfini

Re: [HACKERS] INSERT ... ON CONFLICT UPDATE and RLS

2015-01-14 Thread Stephen Frost
* Dean Rasheed (dean.a.rash...@gmail.com) wrote: > Turns out it wasn't as simple as that. prepend_row_security_policies() > really could get called multiple times for the same RTE, because the > call to query_tree_walker() at the end of fireRIRrules() would descend > into the just-added quals again

Re: [HACKERS] Removing INNER JOINs

2015-01-14 Thread Jim Nasby
On 1/13/15 5:02 AM, David Rowley wrote: I can't quite get my head around what you mean here, as the idea sounds quite similar to something that's been discussed already and ruled out. If we're joining relation a to relation b, say the plan chosen is a merge join. If we put some special node as

Re: [HACKERS] pg_basebackup fails with long tablespace paths

2015-01-14 Thread Robert Haas
On Tue, Jan 13, 2015 at 4:41 PM, Peter Eisentraut wrote: > I think the key point I'm approaching is that the information should > only ever be in one place, all the time. This is not dissimilar from > why we took the tablespace location out of the system catalogs. Users > might have all kinds of

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Merlin Moncure
On Wed, Jan 14, 2015 at 9:49 AM, Andres Freund wrote: > On 2015-01-14 09:47:19 -0600, Merlin Moncure wrote: >> On Wed, Jan 14, 2015 at 9:30 AM, Andres Freund >> wrote: >> > If you gdb in, and type 'fin' a couple times, to wait till the function >> > finishes, is there actually any progress? I'm

Re: [HACKERS] [PATCH] explain sortorder

2015-01-14 Thread Arne Scheffer
Hi, we will also remove the following is lc_collate hint in the next version, showing only mandatory info as suggested. /* for those who use COLLATE although their default is already the wanted */ if (strcmp(collname, localeptr) == 0) {

Re: [HACKERS] [PATCH] explain sortorder

2015-01-14 Thread Timmer, Marius
Hello Heikki, abbreviated version: Sorry, the problem is only the unhandy patch text format, not different opinions how to proceed. Long version: The v7 patch file already addressed your suggestions, but the file contained serveral (old) local commits, the new ones at the end of the patch text/f

Re: [HACKERS] PATCH: hashjoin - gracefully increasing NTUP_PER_BUCKET instead of batching

2015-01-14 Thread Tomas Vondra
On 11.12.2014 23:46, Tomas Vondra wrote: > On 11.12.2014 22:16, Robert Haas wrote: >> On Thu, Dec 11, 2014 at 2:51 PM, Tomas Vondra wrote: >> >>> The idea was that if we could increase the load a bit (e.g. using 2 >>> tuples per bucket instead of 1), we will still use a single batch in >>> some ca

Re: [HACKERS] Memory leak in vacuumlo

2015-01-14 Thread Robert Haas
On Tue, Jan 13, 2015 at 2:58 AM, Michael Paquier wrote: > Coverity has pointed out that vacuumlo.c is leaking memory when > grabbing the list of candidate table names and schemas. Attached is a > patch to fix the leak. It's only leaking in error cases, the first of which is quite unlikely. But I

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Peter Geoghegan
On Wed, Jan 14, 2015 at 11:49 AM, Merlin Moncure wrote: > so it looks like nobody ever exits from _bt_moveright. any last > requests before I start bisecting down? Could you write some code to print out the block number (i.e. "BlockNumber blkno") if there are more than, say, 5 retries within _b

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Peter Geoghegan
On Wed, Jan 14, 2015 at 12:24 PM, Peter Geoghegan wrote: > Could you write some code to print out the block number (i.e. > "BlockNumber blkno") if there are more than, say, 5 retries within > _bt_moveright()? Obviously I mean that the block number should be printed, no matter whether or not the P

Re: [HACKERS] __attribute__ for non-gcc compilers

2015-01-14 Thread Robert Haas
On Tue, Jan 13, 2015 at 4:18 PM, Oskari Saarenmaa wrote: > Commit db4ec2ffce35 added alignment attributes for 64-bit atomic > variables as required on 32-bit platforms using > __attribute__((aligned(8)). That works fine with GCC, and would work > with Solaris Studio Compiler [1] and IBM XL C [2],

Re: [HACKERS] Safe memory allocation functions

2015-01-14 Thread Robert Haas
On Tue, Jan 13, 2015 at 10:10 AM, Tom Lane wrote: > However, there is a larger practical problem with this whole concept, > which is that experience should teach us to be very wary of the assumption > that asking for memory the system can't give us will just lead to nice > neat malloc-returns-NULL

Re: [HACKERS] OOM on EXPLAIN with lots of nodes

2015-01-14 Thread Robert Haas
On Tue, Jan 13, 2015 at 8:16 PM, Tom Lane wrote: > I wrote: >> Heikki Linnakangas writes: >>> But do we really need to backpatch any of this? > >> Alexey's example consumes only a couple hundred MB in 9.2, vs about 7GB >> peak in 9.3 and up. That seems like a pretty nasty regression. > > I did a

Re: [HACKERS] Parallel Seq Scan

2015-01-14 Thread Robert Haas
On Tue, Jan 13, 2015 at 6:25 AM, John Gorman wrote: > One approach that I has worked well for me is to break big jobs into much > smaller bite size tasks. Each task is small enough to complete quickly. > > We add the tasks to a task queue and spawn a generic worker pool which eats > through the ta

Re: [HACKERS] Typo fix in alter_table.sgml

2015-01-14 Thread Robert Haas
On Wed, Jan 14, 2015 at 2:55 AM, Michael Paquier wrote: > I noticed that SET STATISTICS was not in a block in > alter_table.sgml: > > - SET STATISTICS acquires a SHARE UPDATE EXCLUSIVE > lock. > + SET STATISTICS acquires a > + SHARE UPDATE EXCLUSIVE lock. > > That's

Re: [HACKERS] Improving RLS qual pushdown

2015-01-14 Thread Robert Haas
On Wed, Jan 14, 2015 at 9:22 AM, Dean Rasheed wrote: > On 14 January 2015 at 13:29, Robert Haas wrote: >> One thing they could still leak is the number of times they got >> called, and thus possibly the number of unseen rows. Now if the >> expressions get constant-folded away that won't be an is

Re: [HACKERS] Async execution of postgres_fdw.

2015-01-14 Thread Robert Haas
On Tue, Jan 13, 2015 at 6:46 AM, Kyotaro HORIGUCHI wrote: >> Is it possible to use the parallel query infrastructure being built by >> Robert or to do something like parallel seq scan? That will work, not just >> for Postgres FDW but all the FDWs. > > But, I think, from the performance view, every

Re: [HACKERS] orangutan seizes up during isolation-check

2015-01-14 Thread Peter Eisentraut
On 1/1/15 11:04 PM, Noah Misch wrote: >> Clusters hosted on OS X fall into these categories: >> >> 1) Unaffected configuration. This includes everyone setting a valid messages >>locale via LANG, LC_ALL or LC_MESSAGES. >> 2) Affected configuration. Through luck and light use, the cluster would

Re: [HACKERS] PATCH: decreasing memory needlessly consumed by array_agg

2015-01-14 Thread Tomas Vondra
On 12.1.2015 01:28, Ali Akbar wrote: > > > Or else we implement what you suggest below (more comments below): > > > > Thinking about the 'release' flag a bit more - maybe we > could do > > this > > instead: > > > > if (release

Re: [HACKERS] pg_rewind in contrib

2015-01-14 Thread Robert Haas
On Wed, Jan 7, 2015 at 8:44 AM, Andrew Dunstan wrote: > I understand, but I think "pg_rewind" is likely to be misleading to many > users who will say "but I don't want just to rewind". > > I'm not wedded to the name I suggested, but I think we should look at > possible alternative names. We do hav

Re: [HACKERS] Out-of-bounds write and incorrect detection of trigger file in pg_standby

2015-01-14 Thread Robert Haas
On Wed, Jan 14, 2015 at 8:24 AM, Michael Paquier wrote: > In pg_standby.c, we use a 32-byte buffer in CheckForExternalTrigger to > which is read the content of the trigger file defined by -f: > CheckForExternalTrigger(void) > { > charbuf[32]; > [...] > if ((len = read(fd, buf,

Re: [HACKERS] __attribute__ for non-gcc compilers

2015-01-14 Thread Andres Freund
On 2015-01-14 15:48:47 -0500, Robert Haas wrote: > On Tue, Jan 13, 2015 at 4:18 PM, Oskari Saarenmaa wrote: > > Commit db4ec2ffce35 added alignment attributes for 64-bit atomic > > variables as required on 32-bit platforms using > > __attribute__((aligned(8)). That works fine with GCC, and would

Re: [HACKERS] __attribute__ for non-gcc compilers

2015-01-14 Thread Robert Haas
On Wed, Jan 14, 2015 at 5:29 PM, Andres Freund wrote: > I think it's better than the alternatives: > > a) Don't support 64bit atomics on any 32bit platform. I think that'd be >sad because there's some places that could greatly benefit from being >able to read/store/manipulate e.g. LSNs ato

Re: [HACKERS] __attribute__ for non-gcc compilers

2015-01-14 Thread Andres Freund
On 2015-01-14 17:46:39 -0500, Robert Haas wrote: > On Wed, Jan 14, 2015 at 5:29 PM, Andres Freund wrote: > > I think it's better than the alternatives: > > > > a) Don't support 64bit atomics on any 32bit platform. I think that'd be > >sad because there's some places that could greatly benefit

Re: [HACKERS] s_lock.h default definitions are rather confused

2015-01-14 Thread Andres Freund
On 2015-01-14 12:27:42 -0500, Robert Haas wrote: > On Mon, Jan 12, 2015 at 12:57 PM, Andres Freund > wrote: > > On 2015-01-10 23:03:36 +0100, Andres Freund wrote: > >> On 2015-01-10 16:09:42 -0500, Tom Lane wrote: > >> > I've not tried to build HEAD on my HPPA dinosaur for awhile, but I did > >>

Re: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2015-01-14 Thread Kouhei Kaigai
> On Fri, Jan 9, 2015 at 10:51 AM, Kouhei Kaigai wrote: > > When custom-scan node replaced a join-plan, it shall have at least two > > child plan-nodes. The callback handler of PlanCustomPath needs to be > > able to call create_plan_recurse() to transform the underlying > > path-nodes to plan-node

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Merlin Moncure
On Wed, Jan 14, 2015 at 2:32 PM, Peter Geoghegan wrote: > On Wed, Jan 14, 2015 at 12:24 PM, Peter Geoghegan wrote: >> Could you write some code to print out the block number (i.e. >> "BlockNumber blkno") if there are more than, say, 5 retries within >> _bt_moveright()? > > Obviously I mean that t

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Peter Geoghegan
On Wed, Jan 14, 2015 at 3:38 PM, Merlin Moncure wrote: > (gdb) print BufferGetBlockNumber(buf) > $15 = 9 > > ..and it stays 9, continuing several times having set breakpoint. And the index involved? I'm pretty sure that this in an internal page, no? -- Peter Geoghegan -- Sent via pgsql-hac

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Merlin Moncure
On Wed, Jan 14, 2015 at 5:39 PM, Peter Geoghegan wrote: > On Wed, Jan 14, 2015 at 3:38 PM, Merlin Moncure wrote: >> (gdb) print BufferGetBlockNumber(buf) >> $15 = 9 >> >> ..and it stays 9, continuing several times having set breakpoint. > > > And the index involved? I'm pretty sure that this in

[HACKERS] Shouldn't CREATE TABLE LIKE copy the relhasoids property?

2015-01-14 Thread Tom Lane
dst1 doesn't get an OID column: regression=# create table src1 (f1 int) with oids; CREATE TABLE regression=# create table dst1 (like src1); CREATE TABLE regression=# \d+ src1 Table "public.src1" Column | Type | Modifiers | Storage | Stats target | Description

Re: [HACKERS] s_lock.h default definitions are rather confused

2015-01-14 Thread Tom Lane
Andres Freund writes: > Right now I think a #ifdef/undef S_UNLOCK in the relevant gcc section > sufficient and acceptable. It's after all the HPPA section that doesn't > really play by the rules. Works for me. regards, tom lane -- Sent via pgsql-hackers mailing list (p

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Peter Geoghegan
On Wed, Jan 14, 2015 at 4:26 PM, Merlin Moncure wrote: > The index is the oid index on pg_class. Some more info: > > *) temp table churn is fairly high. Several dozen get spawned and > destroted at the start of a replication run, all at once, due to some > dodgy coding via dblink. During the re

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Merlin Moncure
On Wed, Jan 14, 2015 at 6:26 PM, Merlin Moncure wrote: > On Wed, Jan 14, 2015 at 5:39 PM, Peter Geoghegan wrote: >> On Wed, Jan 14, 2015 at 3:38 PM, Merlin Moncure wrote: >>> (gdb) print BufferGetBlockNumber(buf) >>> $15 = 9 >>> >>> ..and it stays 9, continuing several times having set breakpoi

Re: [HACKERS] Shouldn't CREATE TABLE LIKE copy the relhasoids property?

2015-01-14 Thread Andrew Dunstan
On 01/14/2015 07:29 PM, Tom Lane wrote: dst1 doesn't get an OID column: regression=# create table src1 (f1 int) with oids; CREATE TABLE regression=# create table dst1 (like src1); CREATE TABLE regression=# \d+ src1 Table "public.src1" Column | Type | Modifiers | S

Re: [HACKERS] Shouldn't CREATE TABLE LIKE copy the relhasoids property?

2015-01-14 Thread Tom Lane
Andrew Dunstan writes: > On 01/14/2015 07:29 PM, Tom Lane wrote: >> If you don't find that problematic, how about this case? >> >> regression=# create table src2 (f1 int, primary key(oid)) with oids; >> CREATE TABLE >> regression=# create table dst2 (like src2 including indexes); >> ERROR: colum

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Peter Geoghegan
This is great, but it's not exactly clear which bt_page_items() page is which - some are skipped, but I can't be sure which. Would you mind rewriting that query to indicate which block is under consideration by bt_page_items()? Thanks -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Merlin Moncure
On Wed, Jan 14, 2015 at 6:50 PM, Peter Geoghegan wrote: > This is great, but it's not exactly clear which bt_page_items() page > is which - some are skipped, but I can't be sure which. Would you mind > rewriting that query to indicate which block is under consideration by > bt_page_items()? yeah.

Re: [HACKERS] Turning recovery.conf into GUCs

2015-01-14 Thread Josh Berkus
On 01/15/2015 02:24 AM, Robert Haas wrote: > I think your idea of adding read-only GUCs with the same names as all > of the recovey.conf parameters is a clear win. Even if we do nothing > more than that, it makes the values visible from the SQL level, and > that's good. But I think we should go f

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Peter Geoghegan
On Wed, Jan 14, 2015 at 4:53 PM, Merlin Moncure wrote: > yeah. via: > cds2=# \copy (select s as page, (bt_page_items('pg_class_oid_index', > s)).* from generate_series(1,12) s) to '/tmp/page_items.csv' csv > header; My immediate observation here is that blocks 2 and 9 have identical metadata (fr

Re: [HACKERS] Typo fix in alter_table.sgml

2015-01-14 Thread Michael Paquier
Robert Haas wrote: > But that's not a "typo" as stated in $SUBJECT but rather a "markup fix". Definitely. Thanks. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel Seq Scan

2015-01-14 Thread Jim Nasby
On 1/13/15 9:42 PM, Amit Kapila wrote: As an example one of the the strategy could be if the table size is X MB and there are 8 workers, then divide the work as X/8 MB for each worker (which I have currently used in patch) and another could be each worker does scan 1 block at a time and then chec

Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-14 Thread Amit Langote
On 06-01-2015 PM 03:40, Amit Langote wrote: > > I agree that while we are discussing these points, we could also be > discussing how we solve problems of existing partitioning implementation > using whatever the above things end up being. Proposed approaches to > solve those problems might be usef

Re: [HACKERS] Out-of-bounds write and incorrect detection of trigger file in pg_standby

2015-01-14 Thread Michael Paquier
On Thu, Jan 15, 2015 at 7:13 AM, Robert Haas wrote: > Instead of doing this: > > if (len < sizeof(buf)) > buf[len] = '\0'; > > ...I would suggest making the size of the buffer one greater than the > size of the read(), and then always nul-terminating the buffer. It > seems to me that

Re: [HACKERS] Safe memory allocation functions

2015-01-14 Thread Alvaro Herrera
Robert Haas wrote: > On Tue, Jan 13, 2015 at 10:10 AM, Tom Lane wrote: > > However, there is a larger practical problem with this whole concept, > > which is that experience should teach us to be very wary of the assumption > > that asking for memory the system can't give us will just lead to nice

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-14 Thread Peter Geoghegan
On Wed, Jan 14, 2015 at 5:23 PM, Peter Geoghegan wrote: > My immediate observation here is that blocks 2 and 9 have identical > metadata (from their page opaque area), but partially non-matching > data items (however, the number of items on each block is consistent > and correct according to that

  1   2   >