Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Daniel Farina
On Fri, Jun 7, 2013 at 12:14 PM, Josh Berkus wrote: > Right now, what we're telling users is "You can have continuous backup > with Postgres, but you'd better hire and expensive consultant to set it > up for you, or use this external tool of dubious provenance which > there's no packages for, or y

Re: [HACKERS] Last log line for log_temp_files is disassociated with query

2013-06-07 Thread Tom Lane
Josh Berkus writes: > ... that is, for some reason, the last log line reported by > log_temp_files becomes disassociated with the query which was using temp > space in the first place. Often, that last file is the biggest one, as > well. > Is this a fixable bug? Without a concrete test case, it

Re: [HACKERS] Partitioning performance: cache stringToNode() of pg_constraint.ccbin

2013-06-07 Thread Noah Misch
On Fri, Jun 07, 2013 at 11:58:14AM +0530, Amit Kapila wrote: > So the memory increase number's would like: > > Example for 64-bit m/c > In database, there are 100 tables, each having 2 constraints and 30 live > connections > > Size increase = no. of tables * size of (Node*) * number of constraint

Re: [HACKERS] Last log line for log_temp_files is disassociated with query

2013-06-07 Thread Josh Berkus
On 06/07/2013 04:52 PM, Josh Berkus wrote: > In many cases, this is what I see (9.2.4): > > session_line_num|message|query > 423|temporary file: path ...procid.0 file size: 525122|SELECT ... > 424|temporary file: path ...procid.1 file size: 622044|SELECT ... > 425|duration: 17078.635 ms execute :

[HACKERS] Last log line for log_temp_files is disassociated with query

2013-06-07 Thread Josh Berkus
Folks, I've been trying to write some tools which (for example) give how much total temp file space was used by specific individual slow queries. And this turns out to be inordinately hard to do because of how log_temp_files works. In many cases, this is what I see (9.2.4): session_line_num|mes

Re: [HACKERS] About large objects asynchronous and non-blocking support

2013-06-07 Thread Tatsuo Ishii
> 2013/6/6 Tatsuo Ishii > >> > Hi. >> > >> > At the moment libpq doesn't seem to support asynchronous and >> > non-blocking support for large objects, in the style of >> > PQsendQuery/PQgetResult. This makes large objects hardly suited for >> > single-threaded programs based on some variant of se

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Noah Misch
On Fri, Jun 07, 2013 at 02:49:37PM -0400, Tom Lane wrote: > Noah Misch writes: > > On Fri, Jun 07, 2013 at 12:26:59PM -0400, Tom Lane wrote: > >> Essentially the argument for allowing this without a permissions check > >> is "I'm not really doing anything to the schema, just preconfiguring the > >

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Joshua D. Drake
On 06/07/2013 01:41 PM, David Johnston wrote: "Please check server log for specifics" is not a good message for something sent to a client that in many normal situation would have no access to said logs. I don't agree. The user doesn't need access to the logs. If they get that error they

Re: [HACKERS] About large objects asynchronous and non-blocking support

2013-06-07 Thread Dmitriy Igrishin
2013/6/8 Giovanni Mascellani > Hi. > > Il 05/06/2013 22:52, Dmitriy Igrishin ha scritto: > >> At the moment libpq doesn't seem to support asynchronous and > >> non-blocking support for large objects, in the style of > >> PQsendQuery/PQgetResult. This makes large objects hardly suited for > >> sin

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread David Johnston
Joshua D. Drake wrote > On 06/07/2013 12:31 PM, Tom Lane wrote: >> "Joshua D. Drake" < > jd@ > > writes: >>> On 06/07/2013 11:57 AM, Tom Lane wrote: I think it's intentional that we don't tell the *client* that level of detail. >> >>> Why? That seems rather silly. >> >> The general poli

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Hannu Krosing
On 06/07/2013 09:16 PM, Andres Freund wrote: > On 2013-06-07 20:10:55 +0100, Simon Riggs wrote: >> On 7 June 2013 19:56, Heikki Linnakangas wrote: >>> On 07.06.2013 21:33, Simon Riggs wrote: Now that I consider Greg's line of thought, the idea we focused on here was about avoiding freezi

Re: [HACKERS] About large objects asynchronous and non-blocking support

2013-06-07 Thread Giovanni Mascellani
Hi. Il 05/06/2013 22:52, Dmitriy Igrishin ha scritto: >> At the moment libpq doesn't seem to support asynchronous and >> non-blocking support for large objects, in the style of >> PQsendQuery/PQgetResult. This makes large objects hardly suited for >> single-threaded programs based on some variant

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Hannu Krosing
On 06/07/2013 08:56 PM, Heikki Linnakangas wrote: > On 07.06.2013 21:33, Simon Riggs wrote: >> Now that I consider Greg's line of thought, the idea we focused on >> here was about avoiding freezing. But Greg makes me think that we may >> also wish to look at allowing queries to run longer than one

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Rodrigo Gonzalez
On Fri, 07 Jun 2013 13:07:21 -0700 "Joshua D. Drake" wrote: > > On 06/07/2013 12:31 PM, Tom Lane wrote: > > "Joshua D. Drake" writes: > >> On 06/07/2013 11:57 AM, Tom Lane wrote: > >>> I think it's intentional that we don't tell the *client* that > >>> level of detail. > > > >> Why? That seems

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Joshua D. Drake
On 06/07/2013 12:31 PM, Tom Lane wrote: "Joshua D. Drake" writes: On 06/07/2013 11:57 AM, Tom Lane wrote: I think it's intentional that we don't tell the *client* that level of detail. Why? That seems rather silly. The general policy on authentication failure reports is that we don't tel

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Simon Riggs
On 7 June 2013 20:23, Tom Lane wrote: > As for other databases, I suspect that ones that have parallel execution > are probably doing it with a thread model not a process model. Separate processes are more common because it covers the general case where query execution is spread across multiple

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 3:23 PM, Tom Lane wrote: > Robert Haas writes: >> On Fri, Jun 7, 2013 at 2:27 PM, Tom Lane wrote: >>> I don't think that bgworkers as currently implemented make this any more >>> practical than it was before. The communication overhead with a >>> separate process would sw

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Tom Lane
David Johnston writes: > I presume that "password" in this context refers to the method by which > identity is checked; some alternatives being "trust" and "ident"? Right. > Using the same logic of why you would not expose the fact that the user is > expired versus the user has provided invalid

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-07 Thread Kevin Grittner
Robert Haas wrote: > Kevin Grittner wrote: >> One such surprise was that the conversion ran faster, even on a >> "largish" database of around 200GB, with 3 checkpoint_segments >> than with larger settings. > > ! > > I can't account for that finding, because my experience is that > small checkpoi

Avoiding bloat in the presence of a long-running transaction (Re: [HACKERS] Freezing without write I/O)

2013-06-07 Thread Heikki Linnakangas
On 07.06.2013 22:15, Robert Haas wrote: On Fri, Jun 7, 2013 at 3:10 PM, Simon Riggs wrote: The long running query problem hasn't ever been looked at, it seems, until here and now. For what it's worth (and that may not be much), I think most people will die a horrible death due to bloat after

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Tom Lane
"Joshua D. Drake" writes: > On 06/07/2013 11:57 AM, Tom Lane wrote: >> I think it's intentional that we don't tell the *client* that level of >> detail. > Why? That seems rather silly. The general policy on authentication failure reports is that we don't tell the client anything it doesn't know

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Simon Riggs
On 7 June 2013 20:16, Andres Freund wrote: > On 2013-06-07 20:10:55 +0100, Simon Riggs wrote: >> On 7 June 2013 19:56, Heikki Linnakangas wrote: >> > On 07.06.2013 21:33, Simon Riggs wrote: >> >> >> >> Now that I consider Greg's line of thought, the idea we focused on >> >> here was about avoidin

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread David Johnston
Tom Lane-2 wrote > "Joshua D. Drake" < > jd@ > > writes: >> I had a customer pulling their hair out today because they couldn't >> login to their system. The error was consistently: > >> 2013-06-07 08:42:44 MST postgres 10.1.11.67 27440 FATAL: password >> authentication failed for user "user >

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Tom Lane
Robert Haas writes: > On Fri, Jun 7, 2013 at 2:27 PM, Tom Lane wrote: >> I don't think that bgworkers as currently implemented make this any more >> practical than it was before. The communication overhead with a >> separate process would swamp any benefit in most cases. > I'm baffled by your s

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 3:14 PM, Kevin Grittner wrote: > Some findings were unsurprising, like that a direct connection > between the servers using a cross-wired network patch cable was > faster than plugging both machines into the same switch. But we > tested all of our assumptions, and re-tested

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Joshua D. Drake
On 06/07/2013 11:57 AM, Tom Lane wrote: "Joshua D. Drake" writes: I had a customer pulling their hair out today because they couldn't login to their system. The error was consistently: 2013-06-07 08:42:44 MST postgres 10.1.11.67 27440 FATAL: password authentication failed for user "user

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Andres Freund
On 2013-06-07 20:10:55 +0100, Simon Riggs wrote: > On 7 June 2013 19:56, Heikki Linnakangas wrote: > > On 07.06.2013 21:33, Simon Riggs wrote: > >> > >> Now that I consider Greg's line of thought, the idea we focused on > >> here was about avoiding freezing. But Greg makes me think that we may > >

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 3:10 PM, Simon Riggs wrote: > The long running query problem hasn't ever been looked at, it seems, > until here and now. For what it's worth (and that may not be much), I think most people will die a horrible death due to bloat after holding a transaction open for a tiny fr

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-07 Thread Kevin Grittner
Robert Haas wrote: > (As to why smaller checkpoint_segments can help, here's my guess: > if checkpoint_segments is relatively small, then when we recycle > a segment we're likely to find its data already in cache.  That's > a lot better than reading it back in from disk just to overwrite > the da

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Josh Berkus
>> I would oppose that as the solution, either an unconditional one, or >> configurable with is it as the default. Those segments are not >> unneeded. I need them. That is why I set up archiving in the first >> place. If you need to shut down the database rather than violate my >> established

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Simon Riggs
On 7 June 2013 19:56, Heikki Linnakangas wrote: > On 07.06.2013 21:33, Simon Riggs wrote: >> >> Now that I consider Greg's line of thought, the idea we focused on >> here was about avoiding freezing. But Greg makes me think that we may >> also wish to look at allowing queries to run longer than on

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 2:27 PM, Tom Lane wrote: > "Fred&Dani&Pandora&Aquiles" writes: >> I asked a while ago in this group about the possibility to implement a >> parallel planner in a multithread way, and the replies were that the >> proposed approach couldn't be implemented, because the postgr

Re: [HACKERS] Bad error message on valuntil

2013-06-07 Thread Tom Lane
"Joshua D. Drake" writes: > I had a customer pulling their hair out today because they couldn't > login to their system. The error was consistently: > 2013-06-07 08:42:44 MST postgres 10.1.11.67 27440 FATAL: password > authentication failed for user "user > However the problem had nothing to d

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Heikki Linnakangas
On 07.06.2013 21:33, Simon Riggs wrote: Now that I consider Greg's line of thought, the idea we focused on here was about avoiding freezing. But Greg makes me think that we may also wish to look at allowing queries to run longer than one epoch as well, if the epoch wrap time is likely to come dow

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Tom Lane
Noah Misch writes: > On Fri, Jun 07, 2013 at 12:26:59PM -0400, Tom Lane wrote: >> Essentially the argument for allowing this without a permissions check >> is "I'm not really doing anything to the schema, just preconfiguring the >> rights that will be attached to a new object if I later (successfu

[HACKERS] Bad error message on valuntil

2013-06-07 Thread Joshua D. Drake
Hello, I had a customer pulling their hair out today because they couldn't login to their system. The error was consistently: 2013-06-07 08:42:44 MST postgres 10.1.11.67 27440 FATAL: password authentication failed for user "user However the problem had nothing to do with password authentica

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-07 Thread Robert Haas
On Thu, Jun 6, 2013 at 10:43 PM, Greg Smith wrote: > The general complaint the last time I suggested a change in this area, to > make checkpoint_segments larger for the average user, was that some people > had seen workloads where that was counterproductive. Pretty sure Kevin > Grittner said he'd

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Simon Riggs
On 7 June 2013 19:08, Heikki Linnakangas wrote: > On 07.06.2013 20:54, Robert Haas wrote: >> >> On Thu, Jun 6, 2013 at 6:28 PM, Greg Stark wrote: >>> >>> On Thu, Jun 6, 2013 at 1:39 PM, Heikki Linnakangas >>> wrote: That will keep OldestXmin from advancing. Which will keep vacuum from

Re: [HACKERS] Parallell Optimizer

2013-06-07 Thread Tom Lane
"Fred&Dani&Pandora&Aquiles" writes: > I asked a while ago in this group about the possibility to implement a > parallel planner in a multithread way, and the replies were that the > proposed approach couldn't be implemented, because the postgres is not > thread-safe. With the new feature Backgrou

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Heikki Linnakangas
On 07.06.2013 20:54, Robert Haas wrote: On Thu, Jun 6, 2013 at 6:28 PM, Greg Stark wrote: On Thu, Jun 6, 2013 at 1:39 PM, Heikki Linnakangas wrote: That will keep OldestXmin from advancing. Which will keep vacuum from advancing relfrozenxid/datfrozenxid. Which will first trigger the warnings

Re: [HACKERS] SPGist "triple parity" concept doesn't work

2013-06-07 Thread Tom Lane
Alvaro Herrera writes: > Is this the chance to add a metapage? SPGist indexes do have a metapage already --- you're confusing them with Gist. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: h

Re: [HACKERS] Freezing without write I/O

2013-06-07 Thread Robert Haas
On Thu, Jun 6, 2013 at 6:28 PM, Greg Stark wrote: > On Thu, Jun 6, 2013 at 1:39 PM, Heikki Linnakangas > wrote: >> That will keep OldestXmin from advancing. Which will keep vacuum from >> advancing relfrozenxid/datfrozenxid. Which will first trigger the warnings >> about wrap-around, then stops n

Re: [HACKERS] Cost limited statements RFC

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 12:55 PM, Greg Smith wrote: > On 6/7/13 12:42 PM, Robert Haas wrote: >> GUCs in terms of units that are meaningful to the user. One could >> have something like io_rate_limit (measured in MB/s), >> io_read_multiplier = 1.0, io_dirty_multiplier = 1.0, and I think that >> wou

Re: [HACKERS] background processes vs. hot standby

2013-06-07 Thread Robert Haas
On Fri, May 24, 2013 at 12:35 PM, Robert Haas wrote: > But I don't know what to do about the problem of needing to know how > many backends there are. I agree with Andres that it's not very > friendly to enforce a restriction that all the same worker processes > must be present on the standby. O

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Jesper Krogh
On 07/06/13 16:39, Tom Lane wrote: Amit Langote writes: Okay, aside from that case, what else would move those to the main structure? They (the entries in the unsorted pending list) are in the local memory (work_mem?) of the backend, right? No. If they were, it wouldn't be crash-safe. Thats

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Noah Misch
On Fri, Jun 07, 2013 at 12:26:59PM -0400, Tom Lane wrote: > I wrote: > > Noah Misch writes: > >> The particular restriction at hand, namely that a role have CREATE rights > >> on a > >> schema before assigning role-specific default privileges, seems like > >> needless > >> paternalism. It would

[HACKERS] Parallell Optimizer

2013-06-07 Thread
I asked a while ago in this group about the possibility to implement a parallel planner in a multithread way, and the replies were that the proposed approach couldn't be implemented, because the postgres is not thread-safe. With the new feature Background Worker Processes, such implementation woul

Re: [HACKERS] SPGist "triple parity" concept doesn't work

2013-06-07 Thread Alvaro Herrera
Tom Lane wrote: > I don't immediately see any good way to fix this. I think the "triple > parity" rule as it stands is hopelessly broken, but I don't know what > to replace it with, even granting that we don't need to maintain on-disk > compatibility. (We'd have to tell people to reindex SPGist

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 12:44 PM, Stephen Frost wrote: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Essentially the argument for allowing this without a permissions check >> is "I'm not really doing anything to the schema, just preconfiguring the >> rights that will be attached to a new object if I

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Hm. Throwing a NOTICE saying "btw, this won't be of any value until the > user has CREATE rights in that schema" might be a reasonable compromise. Yeah, a NOTICE makes more sense than a WARNING, so +1 from me. Thanks, Stephen si

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Tom Lane
Heikki Linnakangas writes: > On 07.06.2013 19:33, Tom Lane wrote: >> Not only is that a horrible layering/modularity violation, but surely >> LockBuffer can have no idea how much WAL space will be needed. > It can be just a conservative guess, like, 32KB. That should be enough > for almost all W

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Essentially the argument for allowing this without a permissions check >> is "I'm not really doing anything to the schema, just preconfiguring the >> rights that will be attached to a new object if I later (successfully) >> create o

Re: [HACKERS] Cost limited statements RFC

2013-06-07 Thread Greg Smith
On 6/7/13 12:42 PM, Robert Haas wrote: GUCs in terms of units that are meaningful to the user. One could have something like io_rate_limit (measured in MB/s), io_read_multiplier = 1.0, io_dirty_multiplier = 1.0, and I think that would be reasonably clear. There's one other way to frame this:

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Essentially the argument for allowing this without a permissions check > is "I'm not really doing anything to the schema, just preconfiguring the > rights that will be attached to a new object if I later (successfully) > create one in this schema". Makes se

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Heikki Linnakangas
On 07.06.2013 19:33, Tom Lane wrote: Heikki Linnakangas writes: On 06.06.2013 17:00, Heikki Linnakangas wrote: A more workable idea is to sprinkle checks in higher-level code, before you hold any critical locks, to check that there is enough preallocated WAL. Like, at the beginning of heap_ins

Re: [HACKERS] Cost limited statements RFC

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 11:35 AM, Greg Smith wrote: > I wasn't talking about disruption of the data that's in the buffer cache. > The only time the scenario I was describing plays out is when the data is > already in shared_buffers. The concern is damage done to the CPU's data > cache by this acti

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Tom Lane
Heikki Linnakangas writes: > On 06.06.2013 17:00, Heikki Linnakangas wrote: >> A more workable idea is to sprinkle checks in higher-level code, before >> you hold any critical locks, to check that there is enough preallocated >> WAL. Like, at the beginning of heap_insert, heap_update, etc., and al

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Andres Freund
On 2013-06-07 12:16:48 -0400, Stephen Frost wrote: > * Andres Freund (and...@2ndquadrant.com) wrote: > > Currently on a little endian system the pglz header contains the length > > in the first four bytes as: > > [][][][xxdd] > > Where dd are valid length bits for pglz a

Re: [HACKERS] ALTER DEFAULT PRIVILEGES FOR ROLE is broken

2013-06-07 Thread Tom Lane
I wrote: > Noah Misch writes: >> The particular restriction at hand, namely that a role have CREATE rights on >> a >> schema before assigning role-specific default privileges, seems like needless >> paternalism. It would be akin to forbidding ALTER ROLE ... PASSWORD on a >> NOLOGIN role. I'd su

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Heikki Linnakangas
On 06.06.2013 17:00, Heikki Linnakangas wrote: A more workable idea is to sprinkle checks in higher-level code, before you hold any critical locks, to check that there is enough preallocated WAL. Like, at the beginning of heap_insert, heap_update, etc., and all similar indexam entry points. Act

Re: [HACKERS] create a git symbolic-ref for REL9_3_STABLE

2013-06-07 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > I think we should just go ahead and really create the branch. The CF > is supposed to start in a week. +1 Thanks, Stephen signature.asc Description: Digital signature

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > Currently on a little endian system the pglz header contains the length > in the first four bytes as: > [][][][xxdd] > Where dd are valid length bits for pglz and xx are the two bits which > are always zero since we only

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Hannu Krosing
On 06/07/2013 05:38 PM, Andres Freund wrote: > On 2013-06-07 17:27:28 +0200, Hannu Krosing wrote: >> On 06/07/2013 04:54 PM, Andres Freund wrote: >>> I mean, we don't necessarily need to make it configurable if we just add >>> one canonical new "better" compression format. I am not sure that's >>>

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Tom Lane
Andres Freund writes: > On 2013-06-07 11:46:45 -0400, Tom Lane wrote: >> IME, once we've changed it once, the odds that we'll want to change it >> again go up drastically. I think if we're going to make a change here >> we should leave room for future revisions. > The above bit was just about ho

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Alvaro Herrera
Andres Freund escribió: > 2) Combined patch that adds indirect tuple and snappy compression. Those > coul be separated, but this is an experiment so far... Can we have a separate header for toast definitions? (i.e. split them out of postgres.h) -- Álvaro Herrerahttp://www.2ndQua

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Andres Freund
On 2013-06-07 11:46:45 -0400, Tom Lane wrote: > Andres Freund writes: > > I mean, we don't necessarily need to make it configurable if we just add > > one canonical new "better" compression format. I am not sure that's > > sufficient since I can see usecases for 'very fast but not too well > > com

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-07 Thread Kevin Grittner
Tom Lane wrote: > Kevin Grittner writes: >> Tom Lane wrote: >>> Actually, I think this is a bug and the right thing is to make the code >>> match the documentation not vice versa. > >> I assume that this should be a 9.3 code fix, and a doc fix prior to >> that, since it would require changing ca

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Tom Lane
Andres Freund writes: > I mean, we don't necessarily need to make it configurable if we just add > one canonical new "better" compression format. I am not sure that's > sufficient since I can see usecases for 'very fast but not too well > compressed' and 'very well compressed but slow', but I am p

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Andres Freund
On 2013-06-07 17:27:28 +0200, Hannu Krosing wrote: > On 06/07/2013 04:54 PM, Andres Freund wrote: > > > > I mean, we don't necessarily need to make it configurable if we just add > > one canonical new "better" compression format. I am not sure that's > > sufficient since I can see usecases for 'ver

Re: [HACKERS] Cost limited statements RFC

2013-06-07 Thread Greg Smith
On 6/7/13 10:14 AM, Robert Haas wrote: If the page hit limit goes away, the user with a single core server who used to having autovacuum only pillage shared_buffers at 78MB/s might complain that if it became unbounded. Except that it shouldn't become unbounded, because of the ring-buffer stuff.

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Hannu Krosing
On 06/07/2013 04:54 PM, Andres Freund wrote: > > I mean, we don't necessarily need to make it configurable if we just add > one canonical new "better" compression format. I am not sure that's > sufficient since I can see usecases for 'very fast but not too well > compressed' and 'very well compress

Re: [HACKERS] Proposal for CSN based snapshots

2013-06-07 Thread Kevin Grittner
Ants Aasma wrote: > Serializable transactions > - > > I won't pretend to be familiar with SSI code, but as far as I can > tell serializable transactions don't need any modifications to > work with the CSN based snapshot scheme. There actually already > is a commit sequence

Re: [HACKERS] pg_ugprade use of --check and --link

2013-06-07 Thread Bruce Momjian
On Thu, Jun 6, 2013 at 10:24:30AM -0400, Alvaro Herrera wrote: > Bruce Momjian wrote: > > In a private bug report, I have realized that if you are eventually > > going to be using link mode with pg_upgrade, and you run --check mode, > > you should use --link with --check to check that both cluster

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-07 Thread Tom Lane
Kevin Grittner writes: > Tom Lane wrote: >> Actually, I think this is a bug and the right thing is to make the code >> match the documentation not vice versa. > I assume that this should be a 9.3 code fix, and a doc fix prior to > that, since it would require changing catalogs and might break >

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Andres Freund
On 2013-06-07 10:44:24 -0400, Robert Haas wrote: > On Fri, Jun 7, 2013 at 10:30 AM, Andres Freund wrote: > > Turns out the benefits are imo big enough to make it worth pursuing > > further. > > Yeah, those were nifty numbers. > > > The problem is that to discern from pglz on little endian the by

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Amit Langote
On Fri, Jun 7, 2013 at 11:36 PM, Andres Freund wrote: > On 2013-06-07 23:28:56 +0900, Amit Langote wrote: >> On Fri, Jun 7, 2013 at 11:15 PM, Robert Haas wrote: >> > On Wed, Jun 5, 2013 at 10:06 PM, Amit Langote >> > wrote: >> >> At what point do the entries in the pending list are moved to the

Re: [HACKERS] MVCC catalog access

2013-06-07 Thread Robert Haas
On Thu, Jun 6, 2013 at 2:48 PM, Jim Nasby wrote: > On 6/5/13 3:49 PM, Robert Haas wrote: >> >> Now, I did find a couple that I thought should probably stick with >> SnapshotNow, specifically pgrowlocks and pgstattuple. > > FWIW, I've often wished for a way to make all stat access transactional, >

Re: [HACKERS] create a git symbolic-ref for REL9_3_STABLE

2013-06-07 Thread Robert Haas
On Mon, Jun 3, 2013 at 9:30 PM, Peter Eisentraut wrote: > I suppose we'll be branching off 9.3 in a few weeks. That event always > creates a service gap in the build farm and similar services, and a race > in the NLS service to get everything adjusted to the new branch. > > It seems to me we coul

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Robert Haas
On Fri, Jun 7, 2013 at 10:30 AM, Andres Freund wrote: > Turns out the benefits are imo big enough to make it worth pursuing > further. Yeah, those were nifty numbers. > The problem is that to discern from pglz on little endian the byte with > the two high bits unset is actually the fourth byte i

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Tom Lane
Amit Langote writes: > Okay, aside from that case, what else would move those to the main > structure? They (the entries in the unsorted pending list) are in the > local memory (work_mem?) of the backend, right? No. If they were, it wouldn't be crash-safe. regards, tom l

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-07 Thread Kevin Grittner
Tom Lane wrote: > Actually, I think this is a bug and the right thing is to make the code > match the documentation not vice versa.  ev_attr isn't being used for > much at the moment, but if it were being used as an AttrNumber, -1 would > not mean "whole row".  It would be a reference to the syst

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Andres Freund
On 2013-06-07 23:28:56 +0900, Amit Langote wrote: > On Fri, Jun 7, 2013 at 11:15 PM, Robert Haas wrote: > > On Wed, Jun 5, 2013 at 10:06 PM, Amit Langote > > wrote: > >> At what point do the entries in the pending list are moved to the main > >> GIN data structure? > >> From documentation, I rea

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Andres Freund
On 2013-06-07 10:04:15 -0400, Robert Haas wrote: > On Wed, Jun 5, 2013 at 11:01 AM, Andres Freund wrote: > > On 2013-05-31 23:42:51 -0400, Robert Haas wrote: > >> > This should allow for fairly easy development of a new compression > >> > scheme for out-of-line toast tuples. It will *not* work for

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Amit Langote
On Fri, Jun 7, 2013 at 11:15 PM, Robert Haas wrote: > On Wed, Jun 5, 2013 at 10:06 PM, Amit Langote wrote: >> At what point do the entries in the pending list are moved to the main >> GIN data structure? >> From documentation, I read that overflowing work_mem and vacuum are >> two such causes; wh

Re: [HACKERS] Regarding GIN Fast Update Technique

2013-06-07 Thread Robert Haas
On Wed, Jun 5, 2013 at 10:06 PM, Amit Langote wrote: > At what point do the entries in the pending list are moved to the main > GIN data structure? > From documentation, I read that overflowing work_mem and vacuum are > two such causes; what about when the concerned backend is to exit and > autova

Re: [HACKERS] Cost limited statements RFC

2013-06-07 Thread Robert Haas
On Thu, Jun 6, 2013 at 7:36 PM, Greg Smith wrote: > I have also subjected some busy sites to a field test here since the > original discussion, to try and nail down if this is really necessary. So > far I haven't gotten any objections, and I've seen one serious improvement, > after setting vacuum_

Re: [HACKERS] extensible external toast tuple support & snappy prototype

2013-06-07 Thread Robert Haas
On Wed, Jun 5, 2013 at 11:01 AM, Andres Freund wrote: > On 2013-05-31 23:42:51 -0400, Robert Haas wrote: >> > This should allow for fairly easy development of a new compression >> > scheme for out-of-line toast tuples. It will *not* work for compressed >> > inline tuples (i.e. VARATT_4B_C). I am n

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-07 Thread Tom Lane
Kevin Grittner writes: > Hari Babu wrote: >> system catalog pg_rewrite column ev_attr document description as shown below >> >> ev_attr  - The column this rule is for (currently, always zero to indicate >> the whole table) >> >> But In the code the column value is always set as -1. can we chang

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-07 Thread Kevin Grittner
Hari Babu wrote: > system catalog pg_rewrite column ev_attr document description as shown below > > ev_attr  - The column this rule is for (currently, always zero to indicate > the whole table) > > But In the code the column value is always set as -1. can we change the > column description as bel

Re: [HACKERS] Proposal for CSN based snapshots

2013-06-07 Thread Ants Aasma
On Fri, Jun 7, 2013 at 3:47 PM, Greg Stark wrote: > On Thu, Jun 6, 2013 at 11:42 PM, Ants Aasma wrote: >> To refresh your memory the basic idea is to change visibility >> determination to be based on a commit sequence number (CSN for short) >> - a 8 byte number incremented on every commit represe

Re: [HACKERS] Proposal for CSN based snapshots

2013-06-07 Thread Ants Aasma
On Fri, Jun 7, 2013 at 2:59 PM, Markus Wanner wrote: >> To refresh your memory the basic idea is to change visibility >> determination to be based on a commit sequence number (CSN for short) >> - a 8 byte number incremented on every commit representing the total >> ordering of commits. To take a s

Re: [HACKERS] Proposal for CSN based snapshots

2013-06-07 Thread Greg Stark
On Thu, Jun 6, 2013 at 11:42 PM, Ants Aasma wrote: > To refresh your memory the basic idea is to change visibility > determination to be based on a commit sequence number (CSN for short) > - a 8 byte number incremented on every commit representing the total > ordering of commits I think we would

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Bernd Helmle
--On 6. Juni 2013 16:25:29 -0700 Josh Berkus wrote: Archiving - In some ways, this is the simplest case. Really, we just need a way to know when the available WAL space has become 90% full, and abort archiving at that stage. Once we stop attempting to archive, we can clean up the u

Re: [HACKERS] Proposal for CSN based snapshots

2013-06-07 Thread Markus Wanner
Ants, On 06/07/2013 12:42 AM, Ants Aasma wrote: > Given the recent ideas being thrown about changing how freezing and > clog is handled and MVCC catalog access I thought I would write out > the ideas that I have had about speeding up snapshots in case there is > an interesting tie in with the curr

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-07 Thread Heikki Linnakangas
On 07.06.2013 00:38, Andres Freund wrote: On 2013-06-06 23:28:19 +0200, Christian Ullrich wrote: * Heikki Linnakangas wrote: The current situation is that if you run out of disk space while writing WAL, you get a PANIC, and the server shuts down. That's awful. We can So we need to somehow s

[HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-07 Thread Hari Babu
Hi, system catalog pg_rewrite column ev_attr document description as shown below ev_attr - The column this rule is for (currently, always zero to indicate the whole table) But In the code the column value is always set as -1. can we change the column description as below is fine? ev_attr

Re: [HACKERS] RFC: ExecNodeExtender

2013-06-07 Thread Kohei KaiGai
2013/6/6 Robert Haas : > On Tue, Jun 4, 2013 at 2:50 PM, Kohei KaiGai wrote: >> Also, I don't think ExecNodeExtender is not a good naming, because it >> is a bit long and >> abbreviation (ENE?) is hard to imagine the feature. Please give this >> feature a cool and >> well understandable name. > >