Re: [HACKERS] skipping pg_log in basebackup (was Re: pg_basebackup and pg_stat_tmp directory)

2015-06-12 Thread Abhijit Menon-Sen
At 2015-06-11 14:38:03 +0900, langote_amit...@lab.ntt.co.jp wrote: > > > On the other hand, I don't like the idea of doing (3) by adding > > command line arguments to pg_basebackup and adding a new option to > > the command. I don't think that level of "flexibility" is justified; > > it would also

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-12 Thread Michael Paquier
On Fri, Jun 12, 2015 at 3:50 PM, Fujii Masao wrote: > On Fri, Jun 12, 2015 at 3:17 PM, Michael Paquier > wrote: >> On Thu, Jun 11, 2015 at 5:48 PM, Fujii Masao wrote: >>> On Thu, Jun 11, 2015 at 2:14 PM, Michael Paquier >>> wrote: On Thu, Jun 11, 2015 at 1:51 AM, Fujii Masao wrote: >

Re: [HACKERS] Why does replication need the old history file?

2015-06-12 Thread Michael Paquier
On Fri, Jun 12, 2015 at 4:56 AM, Josh Berkus wrote: > Hackers, > > Sequence of events: > > 1. PITR backup of server on timeline 2. > > 2. Restored the backup to a new server, new-master. > > 3. Restored the backup to another new server, new-replica. > > 4. Started and promoted new-master (now on T

Re: [HACKERS] The purpose of the core team

2015-06-12 Thread Oleg Bartunov
+1 On Thu, Jun 11, 2015 at 5:12 PM, Robert Haas wrote: > On Tue, Jun 9, 2015 at 6:35 PM, Bruce Momjian wrote: > > There has been some confusion by old and new community members about the > > purpose of the core team, and this lack of understanding has caused some > > avoidable problems. Theref

Re: [HACKERS] Why does replication need the old history file?

2015-06-12 Thread Fujii Masao
On Fri, Jun 12, 2015 at 5:18 PM, Michael Paquier wrote: > On Fri, Jun 12, 2015 at 4:56 AM, Josh Berkus wrote: >> Hackers, >> >> Sequence of events: >> >> 1. PITR backup of server on timeline 2. >> >> 2. Restored the backup to a new server, new-master. >> >> 3. Restored the backup to another new s

Re: [HACKERS] The Future of Aggregation

2015-06-12 Thread David Rowley
On 11 June 2015 at 01:39, Kevin Grittner wrote: > David Rowley wrote: > > > > /* setup */ create table millionrowtable as select > > generate_series(1,100)::numeric as x; > > /* test 1 */ SELECT sum(x) / count(x) from millionrowtable; > > /* test 2 */ SELECT avg(x) from millionrowtable; > >

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-12 Thread Fujii Masao
On Fri, Jun 12, 2015 at 4:29 PM, Michael Paquier wrote: > On Fri, Jun 12, 2015 at 3:50 PM, Fujii Masao wrote: >> On Fri, Jun 12, 2015 at 3:17 PM, Michael Paquier >> wrote: >>> On Thu, Jun 11, 2015 at 5:48 PM, Fujii Masao wrote: On Thu, Jun 11, 2015 at 2:14 PM, Michael Paquier wrote:

Re: [HACKERS] Aggregate Supporting Functions

2015-06-12 Thread David Rowley
On 10 June 2015 at 02:26, Tom Lane wrote: > Kevin Grittner writes: > > David Rowley wrote: > >> [ avoid duplicate calculations for related aggregates ] > > > From the information you have proposed storing, with cost factors > > associated with the functions, it seems technically possible to > >

[HACKERS] [Proposal] Progress bar for pg_dump/pg_restore

2015-06-12 Thread Taiki Kondo
Hi, all. I am newbie in hackers. I have an idea from my point of view as one user, I would like to propose the following. Progress bar for pg_dump / pg_restore = Motivation -- "pg_dump" and "pg_restore" show nothing if users don't specify verbose (-v

Re: [HACKERS] 9.5 release notes

2015-06-12 Thread Petr Jelinek
Hi, + + +Add JSONB functions jsonb_set() and +jsonb_pretty (Dmitry Dolgov, Andrew Dunstan) + + I believe I should be 3rd author for this one as I rewrote large parts of this functionality as part of the review. -- Petr Jelinek http:/

Re: [HACKERS] Collection of memory leaks for ECPG driver

2015-06-12 Thread Michael Meskes
On Mon, Jun 08, 2015 at 10:50:25PM +0900, Michael Paquier wrote: > And the caller needs to be sure that str actually allocates enough > space.. That's not directly ECPG's business, still it feels But there is no way for us to fix this as we want to implement the API as defined in Informix. > unco

Re: [HACKERS] On columnar storage

2015-06-12 Thread Robert Haas
On Thu, Jun 11, 2015 at 7:03 PM, Alvaro Herrera wrote: > I've been trying to figure out a plan to enable native column stores > (CS or "colstore") for Postgres. Motivations: > > * avoid the 32 TB limit for tables > * avoid the 1600 column limit for tables > * increased performance To me, it feel

Re: [HACKERS] Missing XLOG_DEBUG check in AdvanceXLInsertBuffer()?

2015-06-12 Thread Robert Haas
On Wed, Jun 10, 2015 at 7:02 AM, Andres Freund wrote: > DEBUG: initialized 1 pages, upto 40/3977E000 > DEBUG: initialized 9 pages, upto 40/3979 > DEBUG: initialized 1 pages, upto 40/39792000 > DEBUG: initialized 1 pages, upto 40/39794000 > DEBUG: initialized 1 pages, upto 40/39796000 > DE

[HACKERS] git push hook to check for outdated timestamps

2015-06-12 Thread Robert Haas
Could we update our git hook to refuse a push of a new commit whose timestamp is more than, say, 24 hours in the past? Our commit history has some timestamps in it now that are over a month off, and it's really easy to do, because when you rebase a commit, it keeps the old timestamp. If you then

Re: [HACKERS] [Proposal] Progress bar for pg_dump/pg_restore

2015-06-12 Thread Merlin Moncure
On Fri, Jun 12, 2015 at 7:45 AM, Taiki Kondo wrote: > Hi, all. > > I am newbie in hackers. > I have an idea from my point of view as one user, I would like to propose the > following. > > > Progress bar for pg_dump / pg_restore > = > > Motivation > -- >

Re: [HACKERS] [Proposal] Progress bar for pg_dump/pg_restore

2015-06-12 Thread Andres Freund
Hi, On 2015-06-12 12:45:50 +, Taiki Kondo wrote: > Design & API > > When pg_dump / pg_restore is running, progress bar and estimated time to > finish is shown on screen like following. > > > => (50%) 15:50 > > The bar ("=>" in above) and percentage value ("

Re: [HACKERS] On columnar storage

2015-06-12 Thread Merlin Moncure
On Thu, Jun 11, 2015 at 6:03 PM, Alvaro Herrera wrote: > We hope to have a chance to discuss this during the upcoming developer > unconference in Ottawa. Here are some preliminary ideas to shed some > light on what we're trying to do. Quick thought. We already support out of line columnar stora

Re: [HACKERS] Reconsidering the behavior of ALTER COLUMN TYPE

2015-06-12 Thread Tom Lane
Noah Misch writes: > On Thu, Jun 11, 2015 at 03:41:49PM -0500, Merlin Moncure wrote: >> On Thu, Jun 11, 2015 at 3:12 PM, Tom Lane wrote: >>> In any case, we oughta use two different error messages for the two cases, >>> as per my comment in the above thread. That seems like a back-patchable >>>

Re: [HACKERS] reaper should restart archiver even on standby

2015-06-12 Thread Fujii Masao
On Thu, Jun 11, 2015 at 1:39 AM, Alvaro Herrera wrote: > Fujii Masao wrote: > >> Agreed. The attached patch defines the macro to check whether archiver is >> allowed to start up or not, and uses it everywhere except sigusr1_handler. >> I made sigusr1_handler use a different condition because only

Re: [HACKERS] git push hook to check for outdated timestamps

2015-06-12 Thread Andrew Dunstan
On 06/12/2015 09:31 AM, Robert Haas wrote: Could we update our git hook to refuse a push of a new commit whose timestamp is more than, say, 24 hours in the past? Our commit history has some timestamps in it now that are over a month off, and it's really easy to do, because when you rebase a com

Re: [HACKERS] git push hook to check for outdated timestamps

2015-06-12 Thread Tom Lane
Andrew Dunstan writes: > On 06/12/2015 09:31 AM, Robert Haas wrote: >> Could we update our git hook to refuse a push of a new commit whose >> timestamp is more than, say, 24 hours in the past? Our commit history >> has some timestamps in it now that are over a month off, and it's >> really easy t

Re: [HACKERS] The purpose of the core team

2015-06-12 Thread Robert Haas
On Fri, Jun 12, 2015 at 1:21 AM, Simon Riggs wrote: > Deciding "WHAT goes in the next release?" is what Committers do, by > definition. > > It seems strange to have a different mailing list for "WHEN is the next > release needed?", so those two things should be combined. Core team members have so

Re: [HACKERS] Further issues with jsonb semantics, documentation

2015-06-12 Thread Andrew Dunstan
On 06/10/2015 04:02 PM, Peter Geoghegan wrote: On Wed, Jun 10, 2015 at 11:48 AM, Andrew Dunstan wrote: Sorry for the delay on this. I've been mostly off the grid, having an all too rare visit from Tom "Mr Enum" Dunstan, and I misunderstood what you were suggesting, Thank you for working with

Re: [HACKERS] Why does replication need the old history file?

2015-06-12 Thread Josh Berkus
>>> Questions: >>> >>> A. Why does the replica need 0002.history? Shouldn't it only need >>> 0003.history? >> >> From where is the base backup taken in case of the node started at 5? It is the same backup used to restore the master, restored to a point in time 5 minutes earlier just to m

Re: [HACKERS] On columnar storage

2015-06-12 Thread Alvaro Herrera
Amit Kapila wrote: > On Fri, Jun 12, 2015 at 4:33 AM, Alvaro Herrera > wrote: > > There are several parts to this: > > > > 1. the CSM API > > 2. Cataloguing column stores > > 3. Query processing: rewriter, optimizer, executor > > > > I think another important point is about the format of column

Re: [HACKERS] Entities created in one query not available in another in extended protocol

2015-06-12 Thread Simon Riggs
On 11 June 2015 at 22:12, Shay Rojansky wrote: > Thanks everyone for your time (or rather sorry for having wasted it). > > Just in case it's interesting to you... The reason we implemented things > this way is in order to avoid a deadlock situation - if we send two queries > as P1/D1/B1/E1/P2/D2/

Re: [HACKERS] On columnar storage

2015-06-12 Thread Tom Lane
Alvaro Herrera writes: > Amit Kapila wrote: >> Will the column store obey snapshot model similar to current heap tuples, >> if so will it derive the transaction information from heap tuple? > Yes, visibility will be tied to the heap tuple -- a value is accessed > only when its corresponding heap

Re: [HACKERS] Entities created in one query not available in another in extended protocol

2015-06-12 Thread Tom Lane
Simon Riggs writes: > On 11 June 2015 at 22:12, Shay Rojansky wrote: >> Just in case it's interesting to you... The reason we implemented things >> this way is in order to avoid a deadlock situation - if we send two queries >> as P1/D1/B1/E1/P2/D2/B2/E2, and the first query has a large resultset,

Re: [HACKERS] On columnar storage

2015-06-12 Thread Tomas Vondra
Hi, On 06/12/15 15:56, Merlin Moncure wrote: On Thu, Jun 11, 2015 at 6:03 PM, Alvaro Herrera wrote: We hope to have a chance to discuss this during the upcoming developer unconference in Ottawa. Here are some preliminary ideas to shed some light on what we're trying to do. Quick thought. We

Re: [HACKERS] The Future of Aggregation

2015-06-12 Thread Kevin Grittner
David Rowley wrote: >> I am a little curious what sort of machine you're running on, >> because my i7 is much slower. I ran a few other tests with your >> table for perspective. > > Assert enabled build? Mystery solved. Too often I forget to reconfigure with optimization and without cassert f

Re: [HACKERS] Further issues with jsonb semantics, documentation

2015-06-12 Thread Andrew Dunstan
On 06/12/2015 12:29 PM, I wrote: I agree that the json case looks a bit nasty. Maybe a better approach would be to provide a function that, given a JsonLexContext, returns the number of array elements of the current array. In get_array_start we could call that if the relevant path element

Re: [HACKERS] Further issues with jsonb semantics, documentation

2015-06-12 Thread Peter Geoghegan
On Fri, Jun 12, 2015 at 12:26 PM, Andrew Dunstan wrote: > Here's some code for the count piece of that. Thanks. I'll look into integrating this with what I have. BTW, on reflection I'm not so sure about my decision to not touch the logic within jsonb_delete_idx() (commit b81c7b409). I probably s

Re: [HACKERS] On columnar storage

2015-06-12 Thread Alvaro Herrera
Merlin Moncure wrote: > On Thu, Jun 11, 2015 at 6:03 PM, Alvaro Herrera > wrote: > > We hope to have a chance to discuss this during the upcoming developer > > unconference in Ottawa. Here are some preliminary ideas to shed some > > light on what we're trying to do. > > Quick thought. We alread

Re: [HACKERS] Hash index creation warning

2015-06-12 Thread Thom Brown
On 18 October 2014 at 15:36, Bruce Momjian wrote: > On Fri, Oct 17, 2014 at 02:36:55PM -0400, Bruce Momjian wrote: >> On Fri, Oct 17, 2014 at 12:56:52PM -0400, Tom Lane wrote: >> > David G Johnston writes: >> > > The question is whether we explain the implications of not being >> > > WAL-logged

Re: [HACKERS] On columnar storage

2015-06-12 Thread Michael Nolan
On Thu, Jun 11, 2015 at 7:03 PM, Alvaro Herrera wrote: > We hope to have a chance to discuss this during the upcoming developer > unconference in Ottawa. Here are some preliminary ideas to shed some > light on what we're trying to do. > > > I've been trying to figure out a plan to enable native

Re: [HACKERS] Further issues with jsonb semantics, documentation

2015-06-12 Thread Peter Geoghegan
On Thu, Jun 4, 2015 at 5:43 PM, Peter Geoghegan wrote: > > BTW, there is a bug here -- strtol() needs additional defenses [1] > (before casting to int): > > postgres=# select jsonb_set('[1, 2, 3, 4, > 5,6,7,8,9,10,11,12,13,14,15,16,17,18]', > '{"9223372036854775806"}'::text[], '"Input unsanitized"

Re: [HACKERS] Entities created in one query not available in another in extended protocol

2015-06-12 Thread Sehrope Sarkuni
The JDBC driver tries to handle this by estimating how much data has been buffered. It mainly comes up when executing batch INSERTS as a large number of statements may be sent to the backend prior to reading back any results. There's a nice write up of the potential deadlock and the driver's logic

Re: [HACKERS] [GENERAL] 9.4.1 -> 9.4.2 problem: could not access status of transaction 1

2015-06-12 Thread Steve Kehlet
Just wanted to report that I rolled back my VM to where it was with 9.4.2 installed and it wouldn't start. I installed 9.4.4 and now it starts up just fine: > 2015-06-12 16:05:58 PDT [6453]: [1-1] LOG: database system was shut down at 2015-05-27 13:12:55 PDT > 2015-06-12 16:05:58 PDT [6453]: [2-1

Re: [HACKERS] Further issues with jsonb semantics, documentation

2015-06-12 Thread Andrew Dunstan
On 06/12/2015 06:16 PM, Peter Geoghegan wrote: On Thu, Jun 4, 2015 at 5:43 PM, Peter Geoghegan wrote: BTW, there is a bug here -- strtol() needs additional defenses [1] (before casting to int): postgres=# select jsonb_set('[1, 2, 3, 4, 5,6,7,8,9,10,11,12,13,14,15,16,17,18]', '{"92233720368547

Re: [HACKERS] Further issues with jsonb semantics, documentation

2015-06-12 Thread Peter Geoghegan
On Fri, Jun 12, 2015 at 4:31 PM, Andrew Dunstan wrote: > OK, pushed, although you'd have to be trying really hard to break this. > Still, it's reasonable to defend against. I was trying really hard. :-) Thanks -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgre

[HACKERS] Time to fully remove heap_formtuple() and friends?

2015-06-12 Thread Peter Geoghegan
Commit 902d1cb3, made in 2008, established that the functions heap_formtuple(), heap_modifytuple(), and heap_deformtuple() were deprecated. The commit also actually removed those routines, replacing them with simple wrappers around their real replacements, which are spelled slightly differently and

Re: [HACKERS] Is it possible to have a "fast-write" Index?

2015-06-12 Thread deavid
So I just ran a test case for hash, btree, gin_btree and brin indexes. Also without indexes, and without primary keys. * Testing "deliverynotes" table. - Definition and use case: It is a table contaning real delivery note headers of several years It consists of 300k rows, 128 columns, 63 indexes,

Re: [HACKERS] 9.5 release notes

2015-06-12 Thread Bruce Momjian
On Thu, Jun 11, 2015 at 01:27:23PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Thu, Jun 11, 2015 at 05:16:07PM +1200, David Rowley wrote: > >> Would you also be able to mention something about�f15821e and�d222585 ? > > > I am going to defer to Tom on that. I have added optimizer change

Re: [HACKERS] could not truncate directory "pg_subtrans": apparent wraparound

2015-06-12 Thread Thomas Munro
Hi Since the multixact equivalent of this problem[1] fell through the cracks on the multixact mega-thread, here is an updated patch that addresses this problem for both pg_subtrans and pg_multixact/offsets using the same approach: always step back one multixact/xid (rather than doing so only if ol

Re: [HACKERS] Entities created in one query not available in another in extended protocol

2015-06-12 Thread Simon Riggs
On 12 June 2015 at 20:06, Tom Lane wrote: > Simon Riggs writes: > > On 11 June 2015 at 22:12, Shay Rojansky wrote: > >> Just in case it's interesting to you... The reason we implemented things > >> this way is in order to avoid a deadlock situation - if we send two > queries > >> as P1/D1/B1/E1

Re: [HACKERS] 9.5 release notes

2015-06-12 Thread Bruce Momjian
On Thu, Jun 11, 2015 at 01:31:01PM -0400, Tom Lane wrote: > Robert Haas writes: > > On Thu, Jun 11, 2015 at 11:32 AM, Bruce Momjian wrote: > >> Improve hash creation and lookup performance (Tomas Vondra, > >> Teodor Sigaev, Tom Lane, Robert Haas) > > > I suggest haveing two separate items. One

Re: [HACKERS] 9.5 release notes

2015-06-12 Thread Bruce Momjian
On Thu, Jun 11, 2015 at 09:02:35PM +0200, Magnus Hagander wrote: > On Thu, Jun 11, 2015 at 8:56 PM, Josh Berkus wrote: > > On 06/10/2015 09:50 PM, Amit Kapila wrote: > > Also shall we mention about below in Migrations to 9.5 section > > > > "pg_basebackup will not not work in tar

Re: [HACKERS] 9.5 release notes

2015-06-12 Thread Bruce Momjian
On Thu, Jun 11, 2015 at 05:17:54PM -0400, Robert Haas wrote: > On Thu, Jun 11, 2015 at 4:23 PM, Peter Geoghegan wrote: > > Secondly, Robert didn't credit himself as an author in his commit > > message for the abbreviated keys infrastructure + text opclass support > > *at all*. However, I think tha

Re: [HACKERS] 9.5 release notes

2015-06-12 Thread Bruce Momjian
On Fri, Jun 12, 2015 at 12:49:11PM +0900, Fujii Masao wrote: > On Thu, Jun 11, 2015 at 1:15 PM, Bruce Momjian wrote: > > I have committed the first draft of the 9.5 release notes. You can view > > the output here: > > > > http://momjian.us/pgsql_docs/release-9-5.html > > > > and it will e

Re: [HACKERS] 9.5 release notes

2015-06-12 Thread Bruce Momjian
On Fri, Jun 12, 2015 at 02:47:22PM +0200, Petr Jelinek wrote: > Hi, > > + > + > +Add JSONB functions jsonb_set() and > +jsonb_pretty (Dmitry Dolgov, Andrew Dunstan) > + > + > > I believe I should be 3rd author for this one as I rewrote large > parts of th

Re: [HACKERS] 9.5 release notes

2015-06-12 Thread Amit Kapila
On Sat, Jun 13, 2015 at 7:47 AM, Bruce Momjian wrote: > > On Thu, Jun 11, 2015 at 09:02:35PM +0200, Magnus Hagander wrote: > > On Thu, Jun 11, 2015 at 8:56 PM, Josh Berkus wrote: > > > > On 06/10/2015 09:50 PM, Amit Kapila wrote: > > > Also shall we mention about below in Migrations to 9.

Re: [HACKERS] Collection of memory leaks for ECPG driver

2015-06-12 Thread Michael Paquier
On Fri, Jun 12, 2015 at 10:01 PM, Michael Meskes wrote: > On Mon, Jun 08, 2015 at 10:50:25PM +0900, Michael Paquier wrote: > Right, that's because they were developed before free received the safeguard, > or the programmer simply didn't know at that point in time. Unless I'm > mistaken we have o

Re: [HACKERS] Time to fully remove heap_formtuple() and friends?

2015-06-12 Thread Tom Lane
Peter Geoghegan writes: > Attached patch actually removes heap_formtuple() and friends. Does > this seem worthwhile? Seems reasonable, but at this point I would say this is 9.6 material; third-party-module authors have enough to do with the API breaks we've already created for 9.5. Please enter

Re: [HACKERS] Collection of memory leaks for ECPG driver

2015-06-12 Thread Tom Lane
Michael Paquier writes: > On Fri, Jun 12, 2015 at 10:01 PM, Michael Meskes > wrote: >> More seriously, though, does anyone know of any platform where free(NULL) is >> *not* a noop? > I recall reading that some past versions of SunOS crashed, but it is > rather old... Yeah, SunOS 4.x had issue

Re: [HACKERS] Time to fully remove heap_formtuple() and friends?

2015-06-12 Thread Peter Geoghegan
On Fri, Jun 12, 2015 at 8:47 PM, Tom Lane wrote: > Peter Geoghegan writes: >> Attached patch actually removes heap_formtuple() and friends. Does >> this seem worthwhile? > > Seems reasonable, but at this point I would say this is 9.6 material; > third-party-module authors have enough to do with t

Re: [HACKERS] Incompatible trig error handling

2015-06-12 Thread Noah Misch
On Wed, Apr 29, 2015 at 05:11:48PM -0700, Tom Lane wrote: > John Gorman writes: > > Two of the trigonometry functions have differing error condition behavior > > between Linux and OSX. The Linux behavior follows the standard set by the > > other trig functions. > > We have never considered it par