Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Tom Lane
Peter Geoghegan writes: > On Thu, Jan 29, 2015 at 10:20 PM, Tom Lane wrote: >> I made the \u error be errcode(ERRCODE_INVALID_TEXT_REPRESENTATION) >> and errmsg("invalid input syntax for type json"), by analogy to what's >> thrown for non-ASCII Unicode escapes in non-UTF8 encoding. I'm not >

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Peter Geoghegan
On Thu, Jan 29, 2015 at 10:20 PM, Tom Lane wrote: > I made the \u error be errcode(ERRCODE_INVALID_TEXT_REPRESENTATION) > and errmsg("invalid input syntax for type json"), by analogy to what's > thrown for non-ASCII Unicode escapes in non-UTF8 encoding. I'm not > terribly happy with that, tho

Re: [HACKERS] PATCH: Reducing lock strength of trigger and foreign key DDL

2015-01-29 Thread Michael Paquier
On Fri, Jan 23, 2015 at 8:55 AM, Andreas Karlsson wrote: > On 01/22/2015 10:31 PM, Andreas Karlsson wrote: >> >> I agree with this view, and am not sure myself that it is worth lowering >> the lock level of ALTER TRIGGER RENAME. I have attached a patch without >> the changes to ALTER TRIGGER and r

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Tom Lane
Attached is a draft patch for this. It basically reverts commit 0ad1a816320a2b539a51628e2a0b1e83ff096b1d, adds a ban of \u if that would need to be converted to text (so it still works in the plain json type, so long as you don't do much processing), and adds some regression tests. I made the

Re: [HACKERS] Safe memory allocation functions

2015-01-29 Thread Michael Paquier
I wrote: > Yes, this refactoring was good for testing actually... Oops, I have been too hasty when sending previous patch, there was a bug related to huge allocations. Patch correcting this bug is attached. Attached are as well two things I have used to test the new API: - A hack refactoring the e

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Tom Lane
Peter Geoghegan writes: > I looked into it, and it turns out that MongoDB does not accept NUL in > at least some contexts (for object keys). Apparently it wasn't always > so. MongoDB previously had a security issue that was fixed by > introducing this restriction. Their JSON-centric equivalent of

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Peter Geoghegan
On Wed, Jan 28, 2015 at 9:36 AM, Tom Lane wrote: > Anyway, there is a significant amount of work involved here, and there's > no way we're getting it done for 9.4.1, or probably 9.4.anything. I think > our only realistic choice right now is to throw error for \u so that > we can preserve our

Re: [HACKERS] Exposing the stats snapshot timestamp to SQL

2015-01-29 Thread Matt Kelly
​Actually, I just did one more code review of myself, and somehow missed that I submitted the version with the wrong oid. The oid used in the first version is wrong (1) and was from before I read the docs on properly picking one. Attached is the fixed version. (hopefully with the right mime-t

Re: [HACKERS] Exposing the stats snapshot timestamp to SQL

2015-01-29 Thread Tom Lane
Matt Kelly writes: > On Thu, Jan 29, 2015 at 8:42 PM, Tom Lane wrote: >> ... or double. Have you checked that the code behaves properly with >> --disable-integer-timestamps? > I presume you meant --disable-integer-datetimes. I just ran that test case > now, all set. Right, my own sloppiness t

Re: [HACKERS] Exposing the stats snapshot timestamp to SQL

2015-01-29 Thread Matt Kelly
On Thu, Jan 29, 2015 at 8:42 PM, Tom Lane wrote: > Matt Kelly writes: > > Jim, I'm not sure I understand what you mean? This new function follows > > the same conventions as everything else in the file. TimestampTz is > just a > > typedef for int64. > > ... or double. Have you checked that th

Re: [HACKERS] Safe memory allocation functions

2015-01-29 Thread Michael Paquier
On Fri, Jan 30, 2015 at 12:29 AM, Robert Haas wrote: > On Wed, Jan 28, 2015 at 9:34 AM, Michael Paquier > wrote: >> As a result of all the comments on this thread, here are 3 patches >> implementing incrementally the different ideas from everybody: >> 1) 0001 modifies aset.c to return uncondition

Re: [HACKERS] Redesigning checkpoint_segments

2015-01-29 Thread Venkata Balaji N
Hi, I really like the idea of tuning checkpoint segments based on disk space usage. I performed series of tests for this patch and would like to share the results. My comments are in-line. To start with, I applied this patch to the master successfully - > But ... do I understand things correc

Re: [HACKERS] Possible typo in create_policy.sgml

2015-01-29 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > On Thu, Jan 29, 2015 at 9:04 PM, Stephen Frost wrote: > > A policy grants the ability to SELECT, INSERT, UPDATE, or DELETE rows > > which match the relevant policy expression. Existing table rows are > > checked against the expression specified via US

[HACKERS] Re: [COMMITTERS] pgsql: Fix column-privilege leak in error-message paths

2015-01-29 Thread Stephen Frost
Tom, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > Fix column-privilege leak in error-message paths [...] > The cause of that is the logic added to BuildIndexValueDescription, which > ignores the possibility that some of the index columns are expressions > (which will have

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2015-01-29 Thread Alvaro Herrera
Pavel Stehule wrote: > should not be used a pessimist controlled locking instead? > Patches welcome. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postg

Re: [HACKERS] Providing catalog view to pg_hba.conf file - Patch submission

2015-01-29 Thread Amit Kapila
On Fri, Jan 30, 2015 at 3:16 AM, Jim Nasby wrote: > > On 1/29/15 6:19 AM, Fabrízio de Royes Mello wrote: >> >> Perhaps a fdw can't be the best choice, maybe a complete new SQL syntax to manipulate HBA entries like we did with ALTER SYSTEM. It's just some thoughts about it. > > > Aside from Tom's c

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > On Thu, Jan 29, 2015 at 5:30 PM, Andrew Dunstan wrote: > > jsonb stores string values as postgres text values, with the unicode escapes > > resolved, just as it also resolves numbers and booleans into their native > > representation. If you want the

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Robert Haas
On Thu, Jan 29, 2015 at 5:39 PM, Tom Lane wrote: > Robert Haas writes: >> I have yet to understand what we fix by banning \u. How is >> different from any other four-digit hexadecimal number that's not a >> valid character in the current encoding? What does banning that one >> particul

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Robert Haas
On Thu, Jan 29, 2015 at 5:30 PM, Andrew Dunstan wrote: > jsonb stores string values as postgres text values, with the unicode escapes > resolved, just as it also resolves numbers and booleans into their native > representation. If you want the input perfectly preserved, use json, not > jsonb. I

Re: [HACKERS] Possible typo in create_policy.sgml

2015-01-29 Thread Robert Haas
On Thu, Jan 29, 2015 at 9:04 PM, Stephen Frost wrote: > A policy grants the ability to SELECT, INSERT, UPDATE, or DELETE rows > which match the relevant policy expression. Existing table rows are > checked against the expression specified via USING, while new rows > that would be created via INSER

Re: [HACKERS] pg_check_dir comments and implementation mismatch

2015-01-29 Thread Michael Paquier
On Fri, Jan 30, 2015 at 2:49 AM, Tom Lane wrote: > There is at least one other bug in that function now that I look at it: > in event of a readdir() failure, it neglects to execute closedir(). > Perhaps not too significant since all existing callers will just exit() > anyway after a failure, but s

Re: [HACKERS] Possible typo in create_policy.sgml

2015-01-29 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 28, 2015 at 10:45 PM, Stephen Frost wrote: > > I agree, especially after going back and re-reading this while fixing > > the issue mentioned earlier by Peter (which was an orthogonal complaint > > about the shadowing of WITH CHECK by USING

Re: [HACKERS] Exposing the stats snapshot timestamp to SQL

2015-01-29 Thread Tom Lane
Matt Kelly writes: > Jim, I'm not sure I understand what you mean? This new function follows > the same conventions as everything else in the file. TimestampTz is just a > typedef for int64. ... or double. Have you checked that the code behaves properly with --disable-integer-timestamps?

Re: [HACKERS] Error check always bypassed in tablefunc.c

2015-01-29 Thread Michael Paquier
On Fri, Jan 30, 2015 at 10:32 AM, Tom Lane wrote: > [blah] > What I did about this was to leave the behavior alone in back branches, > but insist on a type match in HEAD. I think we can reasonably tighten > the type requirements in a new major release, but doing it in minor > releases is probably

Re: [HACKERS] Error check always bypassed in tablefunc.c

2015-01-29 Thread Tom Lane
Michael Paquier writes: > So, I have been poking at this code a bit more and as the values of > the parameters are passed as-is to the SQL queries that connectby > generates internally (this is as well mentioned in the documentation > here: http://www.postgresql.org/docs/devel/static/tablefunc.htm

Re: [HACKERS] Parallel Seq Scan

2015-01-29 Thread Jim Nasby
On 1/28/15 7:27 PM, Stephen Frost wrote: * Jim Nasby (jim.na...@bluetreble.com) wrote: >On 1/28/15 9:56 AM, Stephen Frost wrote: > >Such i/o systems do exist, but a single RAID5 group over spinning rust > >with a simple filter isn't going to cut it with a modern CPU- we're just > >too darn effi

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread David Steele
On 1/29/15 8:09 PM, Jim Nasby wrote: > On 1/29/15 7:02 PM, David Steele wrote: >> On 1/29/15 7:55 PM, Jim Nasby wrote: >>> On 1/29/15 6:25 PM, David Steele wrote: Safe backups can be done without LSNs provided you are willing to trust your timestamps. >>> >>> Which AFAICT simply isn'

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread Jim Nasby
On 1/29/15 7:02 PM, David Steele wrote: On 1/29/15 7:55 PM, Jim Nasby wrote: On 1/29/15 6:25 PM, David Steele wrote: Safe backups can be done without LSNs provided you are willing to trust your timestamps. Which AFAICT simply isn't safe to do at all... except maybe with the manifest stuff you

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread David Steele
On 1/29/15 7:55 PM, Jim Nasby wrote: > On 1/29/15 6:25 PM, David Steele wrote: >> Safe backups can be done without LSNs provided you are willing to trust >> your timestamps. > > Which AFAICT simply isn't safe to do at all... except maybe with the > manifest stuff you've talked about? Yes - that's

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread Jim Nasby
On 1/29/15 6:25 PM, David Steele wrote: Safe backups can be done without LSNs provided you are willing to trust your timestamps. Which AFAICT simply isn't safe to do at all... except maybe with the manifest stuff you've talked about? >FWIW, I personally am very leery of relying on pg_upgrade

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread David Steele
On 1/29/15 7:07 PM, Jim Nasby wrote: >> Ultimately, there is no single best method. It depends a lot on your >> environment. I would prefer the official documents to contain very safe >> methods. > > How do we define safe though? Your method leaves you without a backup > server until your base ba

Re: [HACKERS] Exposing the stats snapshot timestamp to SQL

2015-01-29 Thread Matt Kelly
This is now: https://commitfest.postgresql.org/4/128/ On Thu, Jan 29, 2015 at 7:01 PM, Matt Kelly wrote: > Robert, I'll add it to the commitfest. > > Jim, I'm not sure I understand what you mean? This new function follows > the same conventions as everything else in the file. TimestampTz is ju

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread Jim Nasby
On 1/29/15 5:53 PM, David Steele wrote: On 1/29/15 12:42 PM, Josh Berkus wrote: On 01/29/2015 09:11 AM, Bruce Momjian wrote: On Thu, Jan 29, 2015 at 12:09:58PM -0500, Andrew Dunstan wrote: Then step 2 should specify that it's for the master. Right. Josh is just listing all the steps --- the

Re: [HACKERS] Exposing the stats snapshot timestamp to SQL

2015-01-29 Thread Matt Kelly
Robert, I'll add it to the commitfest. Jim, I'm not sure I understand what you mean? This new function follows the same conventions as everything else in the file. TimestampTz is just a typedef for int64. Functions like pg_stat_get_buf_alloc follow the exact same pattern on the int64 fields of

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread David Steele
On 1/29/15 10:13 AM, Bruce Momjian wrote: > Agreed. I have update the two mentions of rsync in our docs to clarify > this. Thank you. > > The patch also has pg_upgrade doc improvements suggested by comments > from Josh Berkus. It's very good to see this. Mentions of this rsync vulnerability are

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread David Steele
On 1/29/15 12:42 PM, Josh Berkus wrote: > On 01/29/2015 09:11 AM, Bruce Momjian wrote: >> On Thu, Jan 29, 2015 at 12:09:58PM -0500, Andrew Dunstan wrote: >>> Then step 2 should specify that it's for the master. >> Right. Josh is just listing all the steps --- the pg_upgrade docs >> already have th

Re: [HACKERS] Exposing the stats snapshot timestamp to SQL

2015-01-29 Thread Jim Nasby
On 1/28/15 11:18 PM, Matt Kelly wrote: In a previous thread Tom Lane said: (I'm also wondering if it'd make sense to expose the stats timestamp as a callable function, so that the case could be dealt with programmatically as well. But that's future-feature territory.) (http://www.p

Re: [HACKERS] TABLESAMPLE patch

2015-01-29 Thread Jim Nasby
On 1/29/15 10:44 AM, Bruce Momjian wrote: On Thu, Jan 29, 2015 at 11:08:55AM -0500, Robert Haas wrote: On Wed, Jan 28, 2015 at 5:19 AM, Petr Jelinek wrote: Yes, that's my view too. I would generally be for that change also and it would be worth it if the code was used in more than one place, b

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread David Steele
On 1/29/15 11:34 AM, Bruce Momjian wrote: > 3. Check that the replica is not very lagged. If it is, wait for > traffic to die down and for it to catch up. I think I'd want a something a bit more specific here. When the primary shuts down it will kick out one last WAL. The filename should be re

Re: [HACKERS] Proposal: two new role attributes and/or capabilities?

2015-01-29 Thread Tom Lane
Jim Nasby writes: > On 1/29/15 4:02 PM, Robert Haas wrote: >> I don't think this is true, and I don't think it's been true for a >> long time, if ever. The difference between a manual vacuum and >> autovacuum is that autovacuum commits suicide when it conflicts with >> somebody else's lock reques

Re: [HACKERS] Proposal: two new role attributes and/or capabilities?

2015-01-29 Thread Jim Nasby
On 1/29/15 4:02 PM, Robert Haas wrote: On Thu, Jan 29, 2015 at 4:09 PM, Jim Nasby wrote: The difference between the autovacuum-run vacuum and the cron-run vacuum is that the one running out of cron will just keep holding the lock until it's actually able to truncate the end of the relation, no?

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Andrew Dunstan
On 01/29/2015 05:39 PM, Tom Lane wrote: Robert Haas writes: I have yet to understand what we fix by banning \u. How is different from any other four-digit hexadecimal number that's not a valid character in the current encoding? What does banning that one particular value do? BTW, a

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Tom Lane
Robert Haas writes: > I have yet to understand what we fix by banning \u. How is > different from any other four-digit hexadecimal number that's not a > valid character in the current encoding? What does banning that one > particular value do? BTW, as to the point about encoding violat

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Andrew Dunstan
On 01/29/2015 04:59 PM, Robert Haas wrote: On Thu, Jan 29, 2015 at 4:33 PM, Andrew Dunstan wrote: On 01/29/2015 12:10 PM, Robert Haas wrote: On Wed, Jan 28, 2015 at 12:48 PM, Tom Lane wrote: The cause of this bug is commit 0ad1a816320a2b539a51628e2a0b1e83ff096b1d, which I'm inclined to thin

Re: [HACKERS] Misaligned BufferDescriptors causing major performance problems on AMD

2015-01-29 Thread Andres Freund
Pushed a version (hopefully) fixing Tom's complaints. On 2015-01-28 13:52:30 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2015-01-28 13:38:51 -0500, Tom Lane wrote: > >> #define BUFFERDESC_PADDED_SIZE (SIZEOF_VOID_P == 8 ? 64 : 32) > > > Hm, did you intentionally put a 32in there or

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Tom Lane
Robert Haas writes: > On Thu, Jan 29, 2015 at 4:33 PM, Andrew Dunstan wrote: >> I'm coming down more and more on the side of Tom's suggestion just to ban >> \u in jsonb. > I have yet to understand what we fix by banning \u. How is > different from any other four-digit hexadecimal n

[HACKERS] Re: [COMMITTERS] pgsql: Fix column-privilege leak in error-message paths

2015-01-29 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > Fix column-privilege leak in error-message paths > > This patch is at least one brick shy of a load: > > regression=# create table t1 (f1 int); > CREATE TABLE > regression=# create unique index on t1 (abs(f1)); > CREATE INDEX > r

Re: [HACKERS] [COMMITTERS] pgsql: Fix column-privilege leak in error-message paths

2015-01-29 Thread Tom Lane
Stephen Frost writes: > Fix column-privilege leak in error-message paths This patch is at least one brick shy of a load: regression=# create table t1 (f1 int); CREATE TABLE regression=# create unique index on t1 (abs(f1)); CREATE INDEX regression=# create user joe; CREATE ROLE regression=# gran

Re: [HACKERS] Proposal: two new role attributes and/or capabilities?

2015-01-29 Thread Robert Haas
On Thu, Jan 29, 2015 at 4:09 PM, Jim Nasby wrote: >> The difference between the autovacuum-run vacuum and the cron-run vacuum >> is that the one running out of cron will just keep holding the lock >> until it's actually able to truncate the end of the relation, no? I >> recall discussion previous

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Robert Haas
On Thu, Jan 29, 2015 at 4:33 PM, Andrew Dunstan wrote: > On 01/29/2015 12:10 PM, Robert Haas wrote: >> On Wed, Jan 28, 2015 at 12:48 PM, Tom Lane wrote: >>> The cause of this bug is commit 0ad1a816320a2b539a51628e2a0b1e83ff096b1d, >>> which I'm inclined to think we need to simply revert, not rend

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Andres Freund
On 2015-01-29 16:33:36 -0500, Andrew Dunstan wrote: > > On 01/29/2015 12:10 PM, Robert Haas wrote: > >On Wed, Jan 28, 2015 at 12:48 PM, Tom Lane wrote: > >>The cause of this bug is commit 0ad1a816320a2b539a51628e2a0b1e83ff096b1d, > >>which I'm inclined to think we need to simply revert, not rende

Re: [HACKERS] Providing catalog view to pg_hba.conf file - Patch submission

2015-01-29 Thread Jim Nasby
On 1/29/15 6:19 AM, Fabrízio de Royes Mello wrote: On Wed, Jan 28, 2015 at 5:27 PM, Tom Lane mailto:t...@sss.pgh.pa.us>> wrote: > > =?UTF-8?Q?Fabr=C3=ADzio_de_Royes_Mello?= mailto:fabriziome...@gmail.com>> writes: > > But I'm thinking about this patch and would not be interesting to have a >

Re: [HACKERS] File based Incremental backup v8

2015-01-29 Thread Andres Freund
On 2015-01-29 12:57:22 -0500, Robert Haas wrote: > The issues here are similar to those in > http://www.postgresql.org/message-id/20150120152819.gc24...@alap3.anarazel.de > - basically, I think we need to make CREATE DATABASE and ALTER > DATABASE .. SET TABLESPACE fully WAL-logged operations, or th

Re: [HACKERS] Small bug on CREATE EXTENSION pgq...

2015-01-29 Thread Jim Nasby
On 1/28/15 10:25 PM, Tom Lane wrote: Stephen Frost writes: * David Johnston (david.g.johns...@gmail.com) wrote: Fair enough but "reset" to what? I don't know the internal mechanics but if the session default is "warning" and a local change sets it to "notice" then an unconditional reset would

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Andrew Dunstan
On 01/29/2015 12:10 PM, Robert Haas wrote: On Wed, Jan 28, 2015 at 12:48 PM, Tom Lane wrote: The cause of this bug is commit 0ad1a816320a2b539a51628e2a0b1e83ff096b1d, which I'm inclined to think we need to simply revert, not render even more squirrely. Yes, that commit looks broken. If you c

Re: [HACKERS] Proposal: two new role attributes and/or capabilities?

2015-01-29 Thread Jim Nasby
On 1/28/15 7:45 PM, Stephen Frost wrote: Jim, * Jim Nasby (jim.na...@bluetreble.com) wrote: On 12/23/14 12:52 PM, Stephen Frost wrote: Autovacuum can certainly run vacuum/analyze on a few tables every 12 hours, so I'm not really following where you see autovacuum being unable to cope. I agree

Re: [HACKERS] PATCH: decreasing memory needlessly consumed by array_agg

2015-01-29 Thread Jim Nasby
On 1/28/15 4:25 PM, Tomas Vondra wrote: + * It's possible to choose whether to create a separate memory context for the + * array builde state, or whether to allocate it directly within rcontext Typo. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http:/

Re: [HACKERS] pg_check_dir comments and implementation mismatch

2015-01-29 Thread Marco Nenciarini
Il 29/01/15 18:37, Robert Haas ha scritto: > On Thu, Jan 29, 2015 at 11:00 AM, Marco Nenciarini > wrote: >> reading pgcheckdir.c code I noticed that the function comment >> was outdated. The code now can return values from 0 to 4 while the >> comment explains only values 0,1,2. > > This is not ju

Re: [HACKERS] File based Incremental backup v8

2015-01-29 Thread Robert Haas
On Thu, Jan 29, 2015 at 9:47 AM, Marco Nenciarini wrote: > The current implementation of copydir function is incompatible with LSN > based incremental backups. The problem is that new files are created, > but their blocks are still with the old LSN, so they will not be backed > up because they are

Re: [HACKERS] pg_check_dir comments and implementation mismatch

2015-01-29 Thread Tom Lane
Robert Haas writes: > On Thu, Jan 29, 2015 at 11:00 AM, Marco Nenciarini > wrote: >> reading pgcheckdir.c code I noticed that the function comment >> was outdated. The code now can return values from 0 to 4 while the >> comment explains only values 0,1,2. > This is not just a comment fix; you ar

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread Josh Berkus
On 01/29/2015 09:11 AM, Bruce Momjian wrote: > On Thu, Jan 29, 2015 at 12:09:58PM -0500, Andrew Dunstan wrote: >> Then step 2 should specify that it's for the master. > > Right. Josh is just listing all the steps --- the pg_upgrade docs > already have that spelled out in detail. What I'm also sa

Re: [HACKERS] pg_check_dir comments and implementation mismatch

2015-01-29 Thread Robert Haas
On Thu, Jan 29, 2015 at 11:00 AM, Marco Nenciarini wrote: > reading pgcheckdir.c code I noticed that the function comment > was outdated. The code now can return values from 0 to 4 while the > comment explains only values 0,1,2. This is not just a comment fix; you are clearly changing the behavio

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread Bruce Momjian
On Thu, Jan 29, 2015 at 12:09:58PM -0500, Andrew Dunstan wrote: > >>>7. shut down postgres on the replica. > >>> > >>>8. rsync both the old and new data directories from the master to the > >>>replica, using the --size-only and -H hard links options. For example, > >>>if both 9.3 and 9.4 are in /v

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Robert Haas
On Wed, Jan 28, 2015 at 12:48 PM, Tom Lane wrote: > The cause of this bug is commit 0ad1a816320a2b539a51628e2a0b1e83ff096b1d, > which I'm inclined to think we need to simply revert, not render even > more squirrely. Yes, that commit looks broken. If you convert from text to JSON you should get a

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread Andrew Dunstan
On 01/29/2015 11:34 AM, Bruce Momjian wrote: On Thu, Jan 29, 2015 at 10:21:30AM -0500, Andrew Dunstan wrote: On 01/29/2015 12:26 AM, Josh Berkus wrote: So, for my 2c, I'm on the fence about it. On the one hand, I agree, it's a bit of a complex process to get right. On the other hand, it's fa

Re: [HACKERS] Parallel Seq Scan

2015-01-29 Thread Robert Haas
On Thu, Jan 29, 2015 at 11:40 AM, Tom Lane wrote: >> In my experience with RAID, it is smart enough to take advantage of that. >> If the raid controller detects a sequential access pattern read, it >> initiates a read ahead on each disk to pre-position the data it will need >> (or at least, the be

Re: [HACKERS] Exposing the stats snapshot timestamp to SQL

2015-01-29 Thread Robert Haas
On Thu, Jan 29, 2015 at 12:18 AM, Matt Kelly wrote: > In a previous thread Tom Lane said: > >> (I'm also wondering if it'd make sense to expose the stats timestamp >> as a callable function, so that the case could be dealt with >> programmatically as well. But that's future-feature territory.) >

Re: [HACKERS] TABLESAMPLE patch

2015-01-29 Thread Bruce Momjian
On Thu, Jan 29, 2015 at 11:08:55AM -0500, Robert Haas wrote: > On Wed, Jan 28, 2015 at 5:19 AM, Petr Jelinek wrote: > > Yes, that's my view too. I would generally be for that change also and it > > would be worth it if the code was used in more than one place, but as it is > > it seems like it wil

Re: [HACKERS] Parallel Seq Scan

2015-01-29 Thread Tom Lane
Jeff Janes writes: > On Tue, Jan 27, 2015 at 11:08 PM, Heikki Linnakangas < > hlinnakan...@vmware.com> wrote: >> OTOH, spreading the I/O across multiple files is not a good thing, if you >> don't have a RAID setup like that. With a single spindle, you'll just >> induce more seeks. >> >> Perhaps t

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread Bruce Momjian
On Wed, Jan 28, 2015 at 09:26:11PM -0800, Josh Berkus wrote: > 3. Check that the replica is not very lagged. If it is, wait for > traffic to die down and for it to catch up. Is this necessary. It seems quite imprecise too. > 4. Shut down the master using -m fast or -m smart for a clean shutdown

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread Bruce Momjian
On Thu, Jan 29, 2015 at 10:21:30AM -0500, Andrew Dunstan wrote: > > On 01/29/2015 12:26 AM, Josh Berkus wrote: > >>So, for my 2c, I'm on the fence about it. On the one hand, I agree, > >>it's a bit of a complex process to get right. On the other hand, it's > >>far better if we put something out

Re: [HACKERS] Parallel Seq Scan

2015-01-29 Thread Jeff Janes
On Tue, Jan 27, 2015 at 11:08 PM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 01/28/2015 04:16 AM, Robert Haas wrote: > >> On Tue, Jan 27, 2015 at 6:00 PM, Robert Haas >> wrote: >> >>> Now, when you did what I understand to be the same test on the same >>> machine, you got times ran

Re: [HACKERS] Memory leak in gingetbitmap

2015-01-29 Thread Tom Lane
Heikki Linnakangas writes: > [ assorted GIN leaks ] > I think we need a more whole-sale approach. I'm thinking of adding a new > memory context to contain everything related to the scan keys, which can > then be destroyed in whole. > We haven't heard any complaints about this from users, but I

Re: [HACKERS] pg_dump with both --serializable-deferrable and -j

2015-01-29 Thread Tom Lane
Kevin Grittner writes: > I propose to apply the attached to master and back-patch to 9.3, > and follow that with a patch (for master only) along the lines > suggested by Andres. Since *that* change is more invasive and > changes existing behavior I will submit it to the open CF for > review.

[HACKERS] Memory leak in gingetbitmap

2015-01-29 Thread Heikki Linnakangas
While looking at the segfault that Olaf Gawenda reported (bug #12694), I realized that the GIN fast scan patch introduced a small memory leak to re-scanning a GIN index. In a nutshell, freeScanKeys() needs to pfree() the two new arrays, requiredEntries and additionalEntries. After fixing that,

Re: [HACKERS] TABLESAMPLE patch

2015-01-29 Thread Robert Haas
On Wed, Jan 28, 2015 at 5:19 AM, Petr Jelinek wrote: > Yes, that's my view too. I would generally be for that change also and it > would be worth it if the code was used in more than one place, but as it is > it seems like it will just add code/complexity for no real benefit. It would > make sense

Re: [HACKERS] Hot Standby WAL reply uses heavyweight session locks, but doesn't have enough infrastructure set up

2015-01-29 Thread Robert Haas
On Wed, Jan 28, 2015 at 2:41 AM, Michael Paquier wrote: > Andres Freund wrote: >> I think this isn't particularly pretty, but it seems to be working well >> enough, and changing it would be pretty invasive. So keeping in line >> with all that code seems to be easier. > OK, I'm convinced with this

Re: [HACKERS] pg_dump with both --serializable-deferrable and -j

2015-01-29 Thread Kevin Grittner
Kevin Grittner wrote: > Tom Lane wrote: >> A comment seems essential here, because as written anybody would >> think the test for a snapshot is a bug. > > Good point. I propose to apply the attached to master and back-patch to 9.3, and follow that with a patch (for master only) along the line

[HACKERS] pg_check_dir comments and implementation mismatch

2015-01-29 Thread Marco Nenciarini
Hi, reading pgcheckdir.c code I noticed that the function comment was outdated. The code now can return values from 0 to 4 while the comment explains only values 0,1,2. Patch attached. Regards, Marco -- Marco Nenciarini - 2ndQuadrant Italy PostgreSQL Training, Services and Support marco.nencia

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Noah Misch
On Wed, Jan 28, 2015 at 12:48:45PM -0500, Tom Lane wrote: > Noah Misch writes: > > On Tue, Jan 27, 2015 at 03:56:22PM -0500, Tom Lane wrote: > >> So at this point I propose that we reject \u when de-escaping JSON. > > > I would have agreed on 2014-12-09, and this release is the last chance to

Re: [HACKERS] Safe memory allocation functions

2015-01-29 Thread Robert Haas
On Wed, Jan 28, 2015 at 9:34 AM, Michael Paquier wrote: > As a result of all the comments on this thread, here are 3 patches > implementing incrementally the different ideas from everybody: > 1) 0001 modifies aset.c to return unconditionally NULL in case of an > OOM instead of reporting an error.

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread Andrew Dunstan
On 01/29/2015 12:26 AM, Josh Berkus wrote: So, for my 2c, I'm on the fence about it. On the one hand, I agree, it's a bit of a complex process to get right. On the other hand, it's far better if we put something out there along the lines of "if you really want to, this is how to do it" than ha

Re: [HACKERS] pg_upgrade and rsync

2015-01-29 Thread Bruce Momjian
On Tue, Jan 27, 2015 at 10:16:48PM -0500, David Steele wrote: > This is definitely an edge case. Not only does the file have to be > modified in the same second *after* rsync has done the copy, but the > file also has to not be modified in *any other subsequent second* before > the next incrementa

[HACKERS] File based Incremental backup v8

2015-01-29 Thread Marco Nenciarini
The current implementation of copydir function is incompatible with LSN based incremental backups. The problem is that new files are created, but their blocks are still with the old LSN, so they will not be backed up because they are looking old enough. copydir function is used in: CREATE DATAB

Re: [HACKERS] GetLockConflicts() and thus recovery conflicts seem pretty broken

2015-01-29 Thread Simon Riggs
On 27 January 2015 at 14:27, Andres Freund wrote: > While investigating other bugs I noticed that > ResolveRecoveryConflictWithLock() wasn't really working. Turns out > GetLockConflicts() violates it's API contract which says: > > * The result array is palloc'd and is terminated with an invalid

Re: [HACKERS] Possible typo in create_policy.sgml

2015-01-29 Thread Robert Haas
On Wed, Jan 28, 2015 at 10:45 PM, Stephen Frost wrote: > I agree, especially after going back and re-reading this while fixing > the issue mentioned earlier by Peter (which was an orthogonal complaint > about the shadowing of WITH CHECK by USING, if WITH CHECK isn't > specified). We really need a

Re: [HACKERS] Possible typo in create_policy.sgml

2015-01-29 Thread Stephen Frost
* Dean Rasheed (dean.a.rash...@gmail.com) wrote: > On 29 January 2015 at 04:00, Stephen Frost wrote: > > * Robert Haas (robertmh...@gmail.com) wrote: > >> On Wed, Jan 7, 2015 at 3:06 PM, Stephen Frost wrote: > >> > If I'm following correctly, Peter's specifically talking about: > >> > > >> >

Re: [HACKERS] Parallel Seq Scan

2015-01-29 Thread Amit Kapila
On Wed, Jan 28, 2015 at 8:59 PM, Amit Kapila wrote: > > I have tried the tests again and found that I have forgotten to increase > max_worker_processes due to which the data is so different. Basically > at higher client count it is just scanning lesser number of blocks in > fixed chunk approach.

Re: [HACKERS] jsonb, unicode escapes and escaped backslashes

2015-01-29 Thread Bruce Momjian
On Wed, Jan 28, 2015 at 03:13:47PM -0600, Jim Nasby wrote: > While I sympathize with Noah's sentiments, the only thing that makes > sense to me is that a JSON text field is treated the same way as we > treat text. Right now, that means NUL is not allowed, period. +1 -- Bruce Momjian h

Re: [HACKERS] Providing catalog view to pg_hba.conf file - Patch submission

2015-01-29 Thread Fabrízio de Royes Mello
On Wed, Jan 28, 2015 at 5:27 PM, Tom Lane wrote: > > =?UTF-8?Q?Fabr=C3=ADzio_de_Royes_Mello?= writes: > > But I'm thinking about this patch and would not be interesting to have a > > FDW to manipulate the hba file? Imagine if we are able to manipulate the > > HBA file using INSERT/UPDATE/DELETE.

Re: [HACKERS] small postgresql.conf.sample bugfix

2015-01-29 Thread Pavel Stehule
2015-01-29 12:52 GMT+01:00 Thom Brown : > On 29 January 2015 at 11:35, Pavel Stehule > wrote: > >> Hello >> >> a max_locks_per_transaction is twice times in postgresql.conf.sample >> >> Attached simple bugfix >> > > I only see one entry, which is the one you're removing. Where's the other? > Yo

Re: [HACKERS] small postgresql.conf.sample bugfix

2015-01-29 Thread Thom Brown
On 29 January 2015 at 11:35, Pavel Stehule wrote: > Hello > > a max_locks_per_transaction is twice times in postgresql.conf.sample > > Attached simple bugfix > I only see one entry, which is the one you're removing. Where's the other? Thom

[HACKERS] small postgresql.conf.sample bugfix

2015-01-29 Thread Pavel Stehule
Hello a max_locks_per_transaction is twice times in postgresql.conf.sample Attached simple bugfix Regards Pavel commit 89ee480408e63fc39fb6b7a23871c4f46438b7a2 Author: Pavel Stehule Date: Thu Jan 29 12:21:07 2015 +0100 remove twice max_locks_per_transaction variable in postgresql.conf.s

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2015-01-29 Thread Pavel Stehule
2015-01-29 10:28 GMT+01:00 Fabrízio de Royes Mello : > > > Em quinta-feira, 29 de janeiro de 2015, Pavel Stehule < > pavel.steh...@gmail.com> escreveu: > > Hi >> >> I am testing this feature on relative complex schema (38619 tables in db) >> and I got deadlock >> >> [pavel@localhost bin]$ /usr/lo

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2015-01-29 Thread Fabrízio de Royes Mello
Em quinta-feira, 29 de janeiro de 2015, Pavel Stehule < pavel.steh...@gmail.com> escreveu: > Hi > > I am testing this feature on relative complex schema (38619 tables in db) > and I got deadlock > > [pavel@localhost bin]$ /usr/local/pgsql/bin/vacuumdb test2 -fz -j 4 > vacuumdb: vacuuming database

Re: [HACKERS] Possible typo in create_policy.sgml

2015-01-29 Thread Dean Rasheed
On 29 January 2015 at 04:00, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: >> On Wed, Jan 7, 2015 at 3:06 PM, Stephen Frost wrote: >> > If I'm following correctly, Peter's specifically talking about: >> > >> > [ USING ( expression ) ] >> > [ WITH CHECK ( > > class="p