Re: [HACKERS] pg_dump gets attributes from tables in extensions

2015-02-15 Thread Michael Paquier
On Mon, Feb 16, 2015 at 4:45 PM, Michael Paquier wrote: > but not dumping any tables that are part of ext_member. > Excuse the typo => s/but/by/. -- Michael

[HACKERS] pg_dump gets attributes from tables in extensions

2015-02-15 Thread Michael Paquier
Hi all, While looking at the patch to fix pg_dump with extensions containing tables referencing each other, I got surprised by the fact that getTableAttrs tries to dump table attributes even for tables that are part of an extension. Is that normal? Attached is a patch that I think makes things rig

Re: [HACKERS] Allow "snapshot too old" error, to prevent bloat

2015-02-15 Thread Kevin Grittner
Tom Lane wrote: > Jim Nasby writes: >> On 2/15/15 10:36 AM, Tom Lane wrote: >>> I wonder if we couldn't achieve largely the same positive effects through >>> adding a simple transaction-level timeout option. We suggested this to our customer and got out of the meeting with it looking like it *mi

Re: [HACKERS] Join push-down support for foreign tables

2015-02-15 Thread Shigeru Hanada
Kaigai-san, Oops. I rebased the patch onto your v4 custom/foreign join patch. But as you mentioned off-list, I found a flaw about inappropriate change about NestPath still remains in the patch... I might have made my dev branch into unexpected state. I'll check it soon. 2015-02-16 13:13 GMT+09

Re: [HACKERS] Join push-down support for foreign tables

2015-02-15 Thread Kouhei Kaigai
Hanada-san, Your patch mixtures enhancement of custom-/foreign-scan interface and enhancement of contrib/postgres_fdw... Probably, it is a careless mis- operation. Please make your patch as differences from my infrastructure portion. Also, I noticed this "Join pushdown support for foreign tables

Re: [HACKERS] Join push-down support for foreign tables

2015-02-15 Thread Shigeru Hanada
Hi I've revised the patch based on Kaigai-san's custom/foreign join patch posted in the thread below. http://www.postgresql.org/message-id/9a28c8860f777e439aa12e8aea7694f80108c...@bpxm15gp.gisp.nec.co.jp Basically not changed from the version in the last CF, but as Robert commented before, N-way

Re: [HACKERS] Commit fest 2015-12 enters money time

2015-02-15 Thread Michael Paquier
On Fri, Feb 13, 2015 at 10:06 AM, Michael Paquier wrote: > In order to move on to the next CF, I am going to go through all the > remaining patches and update their status accordingly. And sorry for > slacking a bit regarding that. If you wish to complain, of course feel > free to post messages o

Re: [HACKERS] Logical Replication Helpers WIP for discussion

2015-02-15 Thread Michael Paquier
On Sun, Feb 15, 2015 at 11:25 PM, Petr Jelinek wrote: > On 13/02/15 14:04, Petr Jelinek wrote: > >> On 13/02/15 08:48, Michael Paquier wrote: >> >>> >>> Looking at this patch, I don't see what we actually gain much here >>> except a decoder plugin that speaks a special protocol for a special >>>

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2015-02-15 Thread Etsuro Fujita
On 2014/09/13 0:13, Tom Lane wrote: > Albe Laurenz writes: >> Tom Lane wrote: >>> I'm not sure offhand what the new plan tree ought to look like. We could >>> just generate a ForeignScan node, but that seems like rather a misnomer. >>> Is it worth inventing a new ForeignUpdate node type? Or mayb

Re: [HACKERS] New CF app deployment

2015-02-15 Thread Peter Geoghegan
On Sun, Feb 15, 2015 at 4:59 PM, Peter Eisentraut wrote: > I think the old system where the patch submitter declared, this message > contains my patch, is the only one that will work. I tend to agree. That being said, calling out latest attachments is also useful (or highlighting that a particula

Re: [HACKERS] Really bad blowups with hash outer join and nulls

2015-02-15 Thread Andrew Gierth
> "Tomas" == Tomas Vondra writes: Tomas> Improving the estimates is always good, but it's not going to Tomas> fix the case of non-NULL values (it shouldn't be all that Tomas> difficult to create such examples with a value whose hash starts Tomas> with a bunch of zeroes). Right now, I can

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE} 2.0

2015-02-15 Thread Peter Geoghegan
On Fri, Feb 13, 2015 at 7:22 PM, Jim Nasby wrote: > In patch 1, "sepgsql is also affected by this commit. Note that this commit > necessitates an initdb, since stored ACLs are broken." > > Doesn't that warrant bumping catversion? Yes, but traditionally that is left until the last minute - when t

Re: [HACKERS] Issue installing doc tools on OSX

2015-02-15 Thread David Steele
On 2/15/15 7:50 PM, Peter Eisentraut wrote: > On 2/15/15 6:31 PM, David Steele wrote: >> I had a problem installing the doc tools following the directions for >> OSX at http://www.postgresql.org/docs/9.4/static/docguide-toolsets.html. >> I'm running OSX Yosemite. >> >> I got it to work by changing

[HACKERS] Rationalizing the API for array_ref, array_set, and friends

2015-02-15 Thread Tom Lane
The four functions array_ref, array_set, array_get_slice, array_set_slice have traditionally declared their array inputs and results as being of type "ArrayType *". This is a lie, and has been since Berkeley days, because they actually also support "fixed-length array" types such as "name" and "po

Re: [HACKERS] Really bad blowups with hash outer join and nulls

2015-02-15 Thread Tomas Vondra
Hi, On 16.2.2015 00:50, Andrew Gierth wrote: >> "Tom" == Tom Lane writes: > > I've now tried the attached patch to correct the bucketsize > estimates, and it does indeed stop the planner from considering the > offending path (in this case it just does the join the other way > round). > > One

Re: [HACKERS] restrict global access to be readonly

2015-02-15 Thread Peter Eisentraut
On 2/14/15 7:24 PM, Tom Lane wrote: > Another possibility that would be attractive for replication-related > use-cases would be "nothing that generates WAL thank you very much". This would be useful, as it essentially simulates a hot standby. -- Sent via pgsql-hackers mailing list (pgsql-hacke

Re: [HACKERS] New CF app deployment

2015-02-15 Thread Peter Eisentraut
On 2/14/15 7:30 AM, Magnus Hagander wrote: > On Mon, Feb 9, 2015 at 4:56 PM, Robert Haas Can we make it smarter, so that the kinds of things people produce > intending for them to be patches are thought by the CF app to be > patches? > > > Doing it wouldn't be too hard, as the code r

Re: [HACKERS] Issue installing doc tools on OSX

2015-02-15 Thread Peter Eisentraut
On 2/15/15 6:31 PM, David Steele wrote: > I had a problem installing the doc tools following the directions for > OSX at http://www.postgresql.org/docs/9.4/static/docguide-toolsets.html. > I'm running OSX Yosemite. > > I got it to work by changing the command from: > > sudo port install docbook-

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE} 2.0

2015-02-15 Thread Peter Geoghegan
On Sat, Feb 14, 2015 at 2:06 PM, Heikki Linnakangas wrote: >> Thanks for taking a look at it. That's somewhat cleaned up in the >> attached patchseries - V2.2. This has been rebased to repair the minor >> bit-rot pointed out by Thom. > > I don't really have the energy to review this patch in one b

Re: [HACKERS] Replication identifiers, take 4

2015-02-15 Thread Andres Freund
On 2015-02-16 01:21:55 +0100, Andres Freund wrote: > Here's my next attept attempt at producing something we can agree > upon. > > The major change that might achieve that is that I've now provided a > separate method to store the origin_id of a node. I've made it > conditional on !REPLICATION_IDE

Re: [HACKERS] Replication identifiers, take 4

2015-02-15 Thread Andres Freund
Hi, Here's my next attept attempt at producing something we can agree upon. The major change that might achieve that is that I've now provided a separate method to store the origin_id of a node. I've made it conditional on !REPLICATION_IDENTIFIER_REUSE_PADDING, to show both paths. That new method

Re: [HACKERS] Really bad blowups with hash outer join and nulls

2015-02-15 Thread Andrew Gierth
> "Tom" == Tom Lane writes: >> A quick test suggests that initializing the hash value to ~0 rather >> than 0 has a curious effect: the number of batches still explodes, >> but the performance does not suffer the same way. (I think because >> almost all the batches end up empty.) I think t

Re: [HACKERS] Manipulating complex types as non-contiguous structures in-memory

2015-02-15 Thread Robert Haas
On Sun, Feb 15, 2015 at 6:41 PM, Tom Lane wrote: > I'm going to stick this into the commitfest even though it's not really > close to being committable; I see some other people doing likewise with > their pet patches ;-). What it could particularly do with some reviewing > help on is exploring th

Re: [HACKERS] Manipulating complex types as non-contiguous structures in-memory

2015-02-15 Thread Tom Lane
Here's an updated version of the patch I sent before. Notable changes: * I switched over to calling "deserialized" objects "expanded" objects, and the default representation is now called "flat" or "flattened" instead of "reserialized". Per suggestion from Robert. * I got rid of the bit about d

[HACKERS] Issue installing doc tools on OSX

2015-02-15 Thread David Steele
I had a problem installing the doc tools following the directions for OSX at http://www.postgresql.org/docs/9.4/static/docguide-toolsets.html. I'm running OSX Yosemite. I got it to work by changing the command from: sudo port install docbook-dsssl docbook-sgml-4.2 docbook-xml-4.2 docbook-xsl lib

Re: [HACKERS] assessing parallel-safety

2015-02-15 Thread Noah Misch
It's important for parallelism to work under the extended query protocol, and that's nontrivial. exec_parse_message() sets cursorOptions. exec_bind_message() runs the planner. We don't know if a parallel plan is okay before seeing max_rows in exec_execute_message(). On Sun, Feb 15, 2015 at 01:29

Re: [HACKERS] gcc5: initdb produces gigabytes of _fsm files

2015-02-15 Thread Christoph Berg
Re: Tom Lane 2015-02-15 <21030.1424022...@sss.pgh.pa.us> > Christoph Berg writes: > > gcc5 is lurking in Debian experimental, and it's breaking initdb. > > FYI, this is now fixed in Red Hat's rawhide version: > https://bugzilla.redhat.com/show_bug.cgi?id=1190978 > > Don't know what the update pr

Re: [HACKERS] mogrify and indent features for jsonb

2015-02-15 Thread Andrew Dunstan
On 02/14/2015 10:06 PM, Andrew Dunstan wrote: Attached is a patch to provide a number of very useful facilities to jsonb that people have asked for. These are based on work by Dmitry Dolgov in his jsonbx extension, but I take responsibility for any bugs. The facilities are: new operations:

Re: [HACKERS] Selectivity estimation for inet operators

2015-02-15 Thread Emre Hasegeli
New version of the patch attached with the optimization to break the loop before looking at all of the histogram values. I can reduce join selectivity estimation runtime by reducing the values of the left hand side or both of the sides, if there is interest. > > Even if the above aspects of the c

Re: [HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Tomas Vondra
On 15.2.2015 21:38, Tom Lane wrote: > Andres Freund writes: >> On 2015-02-15 21:07:13 +0100, Tomas Vondra wrote: >>> On 15.2.2015 20:56, Heikki Linnakangas wrote: glibc's malloc() also uses mmap() for larger allocations. Precisely because those allocations can then be handed back to the

Re: [HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Tom Lane
Andres Freund writes: > On 2015-02-15 21:07:13 +0100, Tomas Vondra wrote: >> On 15.2.2015 20:56, Heikki Linnakangas wrote: >>> glibc's malloc() also uses mmap() for larger allocations. Precisely >>> because those allocations can then be handed back to the OS. I don't >>> think we'd want to use mma

Re: [HACKERS] Allow "snapshot too old" error, to prevent bloat

2015-02-15 Thread Tom Lane
Jim Nasby writes: > On 2/15/15 10:36 AM, Tom Lane wrote: >> I wonder if we couldn't achieve largely the same positive effects through >> adding a simple transaction-level timeout option. > A common use-case is long-running reports hitting relatively stable data > in a database that also has tabl

Re: [HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Tomas Vondra
On 15.2.2015 21:13, Andres Freund wrote: > On 2015-02-15 21:07:13 +0100, Tomas Vondra wrote: > >> malloc() does that only for allocations over MAP_THRESHOLD, which >> is 128kB by default. Vast majority of blocks we allocate are <= >> 8kB, so mmap() almost never happens. > > The problem is that mma

Re: [HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Andres Freund
On 2015-02-15 21:07:13 +0100, Tomas Vondra wrote: > On 15.2.2015 20:56, Heikki Linnakangas wrote: > > On 02/15/2015 08:57 PM, Tomas Vondra wrote: > >> One of the wilder ideas (I mentined beer was involved!) was a memory > >> allocator based on mmap [2], bypassing the libc malloc implementation > >>

Re: [HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Tomas Vondra
On 15.2.2015 20:56, Heikki Linnakangas wrote: > On 02/15/2015 08:57 PM, Tomas Vondra wrote: >> One of the wilder ideas (I mentined beer was involved!) was a memory >> allocator based on mmap [2], bypassing the libc malloc implementation >> altogether. mmap() has some nice features (e.g. no issues w

Re: [HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Heikki Linnakangas
On 02/15/2015 08:57 PM, Tomas Vondra wrote: One of the wilder ideas (I mentined beer was involved!) was a memory allocator based on mmap [2], bypassing the libc malloc implementation altogether. mmap() has some nice features (e.g. no issues with returning memory back to the kernel, which may be p

Re: [HACKERS] New CF app deployment

2015-02-15 Thread Peter Geoghegan
On Sun, Feb 15, 2015 at 8:36 AM, Magnus Hagander wrote: >> Hmm. This kind of looks like the right idea, but it's hard to use, >> because all you've got to work with is the subject of the message >> (which is the same for all), the time it was sent, and the author. In >> the old system, you could

Re: [HACKERS] Allow "snapshot too old" error, to prevent bloat

2015-02-15 Thread Jim Nasby
On 2/15/15 10:36 AM, Tom Lane wrote: Kevin Grittner writes: Tom Lane wrote: Kevin Grittner writes: What this patch does is add a GUC call old_snapshot_threshold. It defaults to -1, which leaves behavior matching unpatched code. Above that it allows tuples to be vacuumed away after the numb

[HACKERS] EXPERIMENTAL: mmap-based memory context / allocator

2015-02-15 Thread Tomas Vondra
Hi! While playing with the memory context internals some time ago, I've been wondering if there are better ways to allocate memory from the kernel - either tweaking libc somehow, or maybe interacting with kernel directly. I mostly forgot about that topic, but after the local conference last week

Re: [HACKERS] NOT NULL markings for BKI columns

2015-02-15 Thread Tom Lane
Andres Freund writes: > On 2015-02-15 12:31:10 -0500, Tom Lane wrote: >> Where are you thinking of sticking that exactly, and will pgindent >> do something sane with it? > Hm, I was thinking about > /* extversion should never be null, but the others can be. */ > textextver

Re: [HACKERS] NOT NULL markings for BKI columns

2015-02-15 Thread Andres Freund
On 2015-02-15 12:31:10 -0500, Tom Lane wrote: > Andres Freund writes: > > I was thinking of adding BKI_FORCENOTNULL which would be > > specified on the attributes you want it. The FORCE in there representing > > that the default choice is overwritten. > > Where are you thinking of sticking that e

Re: [HACKERS] gcc5: initdb produces gigabytes of _fsm files

2015-02-15 Thread Tom Lane
Christoph Berg writes: > gcc5 is lurking in Debian experimental, and it's breaking initdb. FYI, this is now fixed in Red Hat's rawhide version: https://bugzilla.redhat.com/show_bug.cgi?id=1190978 Don't know what the update process is like for Debian's copy, but maybe you could pester the appropr

Re: [HACKERS] NOT NULL markings for BKI columns

2015-02-15 Thread Tom Lane
Andres Freund writes: > I was thinking of adding BKI_FORCENOTNULL which would be > specified on the attributes you want it. The FORCE in there representing > that the default choice is overwritten. Where are you thinking of sticking that exactly, and will pgindent do something sane with it?

Re: [HACKERS] NOT NULL markings for BKI columns

2015-02-15 Thread Andres Freund
On 2015-02-15 12:11:52 -0500, Tom Lane wrote: > Andres Freund writes: > > So, how about providing bootstrap infrastructure for marking columns as > > NOT NULL? > > We've not desperately needed it up to now, but if you can think of a clean > representation, go for it. I'd want to preserve the pro

Re: [HACKERS] NOT NULL markings for BKI columns

2015-02-15 Thread Tom Lane
Andres Freund writes: > So, how about providing bootstrap infrastructure for marking columns as > NOT NULL? We've not desperately needed it up to now, but if you can think of a clean representation, go for it. I'd want to preserve the property that all columns accessible via C structs are automa

Re: [HACKERS] mogrify and indent features for jsonb

2015-02-15 Thread Andrew Dunstan
On 02/15/2015 11:47 AM, Sehrope Sarkuni wrote: For jsonb_indent, how about having it match up closer to the JavaScript JSON.stringify(value, replacer, space)[1]? That way a user can specify the indentation level and optionally filter the fields they'd like to output. In JS, the "replacer" param

[HACKERS] NOT NULL markings for BKI columns

2015-02-15 Thread Andres Freund
Hi, 8b38a538c0aa5a432dacd90f10805dc667a3d1a0 changed things so that all columns are checked for NOT NULL ness. Which is fine in general, but it currently does make it impossible to have a varlena column (except OID/INT2 vector...) as a key column in syscache. Even if the column is factually NOT NU

Re: [HACKERS] mogrify and indent features for jsonb

2015-02-15 Thread Sehrope Sarkuni
For jsonb_indent, how about having it match up closer to the JavaScript JSON.stringify(value, replacer, space)[1]? That way a user can specify the indentation level and optionally filter the fields they'd like to output. In JS, the "replacer" parameter can be either a JS function or an array of pr

Re: [HACKERS] Allow "snapshot too old" error, to prevent bloat

2015-02-15 Thread Tom Lane
Kevin Grittner writes: > Tom Lane wrote: >> Kevin Grittner writes: >>> What this patch does is add a GUC call old_snapshot_threshold. It >>> defaults to -1, which leaves behavior matching unpatched code. >>> Above that it allows tuples to be vacuumed away after the number of >>> transaction IDs

Re: [HACKERS] New CF app deployment

2015-02-15 Thread Magnus Hagander
On Sat, Feb 14, 2015 at 6:42 PM, Robert Haas wrote: > On Sat, Feb 14, 2015 at 7:29 AM, Magnus Hagander > wrote: > > Ok, I've pushed an attempt at doing this. > > > > For each mailthread, you can now create annotations. Each annotation is > > connected to a mail in the thread, and has a free text

Re: [HACKERS] restrict global access to be readonly

2015-02-15 Thread Andres Freund
On 2015-02-14 17:28:38 -0600, Jim Nasby wrote: > Throw an error in AssignTransactionId/GetNewTransactionId? I see 4 calls to > Get*TransactionId in logical replication, though arguably if we're fixing > that we should look at doing something special for Slony and the > likes. I don't think there a

Re: [HACKERS] restrict global access to be readonly

2015-02-15 Thread Florian Pflug
On Feb15, 2015, at 10:13 , David G Johnston wrote: > happy times wrote >> Sure, we can utilize the runtime parameter >> default_transaction_read_only, however, it does not restrict user from >> changing transaction attribute to non-readonly mode, so is not safe. > > ISTM that implementing a me

Re: [HACKERS] Really bad blowups with hash outer join and nulls

2015-02-15 Thread Tom Lane
Andrew Gierth writes: > A quick test suggests that initializing the hash value to ~0 rather than > 0 has a curious effect: the number of batches still explodes, but the > performance does not suffer the same way. (I think because almost all > the batches end up empty.) I think this is worth doing

Re: [HACKERS] Logical Replication Helpers WIP for discussion

2015-02-15 Thread Petr Jelinek
On 13/02/15 14:04, Petr Jelinek wrote: On 13/02/15 08:48, Michael Paquier wrote: Looking at this patch, I don't see what we actually gain much here except a decoder plugin that speaks a special protocol for a special background worker that has not been presented yet. What actually is the value

Re: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2015-02-15 Thread Kouhei Kaigai
The attached patch is a rebased version of join replacement with foreign-/custom-scan. Here is no feature updates at this moment but SGML documentation is added (according to Michael's comment). This infrastructure allows foreign-data-wrapper and custom-scan- provider to add alternative scan paths

Re: [HACKERS] forward vs backward slashes in msvc build code

2015-02-15 Thread Michael Paquier
On Sun, Feb 15, 2015 at 12:51 PM, Peter Eisentraut wrote: > I understand that on Windows, you can use forward slashes in path names > interchangeably with backward slashes (except possibly in the shell). I > have developed the attached patch to change the msvc build code to use > forward slashes

Re: [HACKERS] KNN-GiST with recheck

2015-02-15 Thread Alexander Korotkov
On Thu, Jan 8, 2015 at 1:12 AM, Alexander Korotkov wrote: > On Tue, Dec 16, 2014 at 4:37 PM, Heikki Linnakangas < > hlinnakan...@vmware.com> wrote: > >> Patch attached. It should be applied on top of my pairing heap patch at >> http://www.postgresql.org/message-id/548ffa2c.7060...@vmware.com. Som

[HACKERS] Really bad blowups with hash outer join and nulls

2015-02-15 Thread Andrew Gierth
This came up today on IRC, though I suspect the general problem has been seen before: create table m3 (id uuid, status integer); create table q3 (id uuid); insert into m3 select uuid_generate_v4(), floor(random() * 4)::integer from generate_series(1,100); insert into q3 select id f

Re: [HACKERS] restrict global access to be readonly

2015-02-15 Thread David G Johnston
happy times wrote > Sure, we can utilize the runtime parameter > default_transaction_read_only, however, it does not restrict user from > changing transaction attribute to non-readonly mode, so is not safe. ISTM that implementing a means to make this setting only super-user changeable would be

Re: [HACKERS] Add pg_settings.pending_restart column

2015-02-15 Thread David G Johnston
Peter Eisentraut-2 wrote > So here is a patch for that. It adds a column pending_restart to > pg_settings that is true when the configuration file contains a changed > setting that requires a restart. We already had the logic to detect > such changes, for producing the log entry. I have also set