Re: [HACKERS] Bad error message on valuntil

2013-06-08 Thread Craig Ringer
On 06/08/2013 04:07 AM, Joshua D. Drake wrote: > > FATAL: Authentication failed: Check server log for specifics > > And then we make sure we log proper info? "FATAL: Authentication using method 'password' failed, possible reasons are no/wrong password sent, account expired; see server log for detai

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Craig Ringer
On 06/09/2013 03:02 AM, Jeff Janes wrote: > It would be nice to have the ability to specify multiple log destinations > with different log_min_messages for each one. I'm sure syslog already must > implement some kind of method for doing that, but I've been happy enough > with the text logs that I

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Craig Ringer
On 06/09/2013 08:32 AM, MauMau wrote: > > - Failure of a disk containing data directory or tablespace > If checkpoint can't write buffers to disk because of disk failure, > checkpoint cannot complete, thus WAL files accumulate in pg_xlog/. > This means that one disk failure will lead to postgres sh

Re: [HACKERS] small patch to crypt.c

2013-06-08 Thread Stephen Frost
* Joshua D. Drake (j...@commandprompt.com) wrote: > Well I was more referring to the default is: > > check if null, if true return ok > check if valuntil < today, if true return error > else return ok > > To me we don't need the null check. However, when I tested it, > without the null check you

Re: [HACKERS] small patch to crypt.c

2013-06-08 Thread Joshua D. Drake
On 06/08/2013 08:47 PM, Stephen Frost wrote: JD, * Joshua D. Drake (j...@commandprompt.com) wrote: In my quest to understand how all the logging etc works with authentication I came across the area of crypt.c that checks for valid_until but it seems like it has an extraneous check. If I am wr

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Craig Ringer
On 06/08/2013 10:57 AM, Daniel Farina wrote: > >> At which point most sensible users say "no thanks, I'll use something else". > [snip] > > I have a clear bias in experience here, but I can't relate to someone > who sets up archives but is totally okay losing a segment unceremoniously, > because it

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Craig Ringer
On 06/06/2013 10:00 PM, Heikki Linnakangas wrote: > > I've seen a case, where it was even worse than a PANIC and shutdown. > pg_xlog was on a separate partition that had nothing else on it. The > partition filled up, and the system shut down with a PANIC. Because > there was no space left, it could

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-08 Thread Craig Ringer
On 06/06/2013 03:21 PM, Joshua D. Drake wrote: > > Not to be unkind but the problems of the uniformed certainly are not > the problems of the informed. Or perhaps they are certainly the > problems of the informed :P. I'm not convinced that's a particularly good argument not to improve something. Su

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-08 Thread Craig Ringer
On 06/07/2013 01:00 AM, Josh Berkus wrote: > Daniel, > > So your suggestion is that if archiving is falling behind, we should > introduce delays on COMMIT in order to slow down the rate of WAL writing? Delaying commit wouldn't be enough; consider a huge COPY, which can produce a lot of WAL at a hig

[HACKERS] Re: [COMMITTERS] pgsql: Don't downcase non-ascii identifier chars in multi-byte encoding

2013-06-08 Thread Noah Misch
On Sat, Jun 08, 2013 at 11:50:53PM -0400, Andrew Dunstan wrote: > On 06/08/2013 10:52 PM, Noah Misch wrote: >> Let's return to the drawing board on this one. I would be inclined to keep >> the current bad behavior until we implement the i18n-aware case folding >> required by SQL. If I'm alone in

Re: [HACKERS] Batch API for After Triggers

2013-06-08 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: > While fiddling with FK tuning, Noah suggested batching trigger > executions together to avoid execution overhead. I like the general idea, but I'd prefer a way to avoid having to queue up tons of trigger events to be executed in the first place. Aggre

[HACKERS] Re: [COMMITTERS] pgsql: Don't downcase non-ascii identifier chars in multi-byte encoding

2013-06-08 Thread Andrew Dunstan
On 06/08/2013 10:52 PM, Noah Misch wrote: On Sat, Jun 08, 2013 at 08:09:15PM -0400, Robert Haas wrote: On Sat, Jun 8, 2013 at 10:25 AM, Andrew Dunstan wrote: Don't downcase non-ascii identifier chars in multi-byte encodings. Long-standing code has called tolower() on identifier character byt

Re: [HACKERS] small patch to crypt.c

2013-06-08 Thread Stephen Frost
JD, * Joshua D. Drake (j...@commandprompt.com) wrote: > In my quest to understand how all the logging etc works with > authentication I came across the area of crypt.c that checks for > valid_until but it seems like it has an extraneous check. > > If I am wrong I apologize for the noise but would

Re: [HACKERS] Proposed patch: remove hard-coded limit MAX_ALLOCATED_DESCS

2013-06-08 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Recently we had a gripe about how you can't read very many files > concurrently with contrib/file_fdw: > http://www.postgresql.org/message-id/of419b5767.8a3c9adb-on85257b79.005491e9-85257b79.0054f...@isn.rtss.qc.ca Ouch, that's pretty bad. > Barring object

[HACKERS] Re: [COMMITTERS] pgsql: Don't downcase non-ascii identifier chars in multi-byte encoding

2013-06-08 Thread Noah Misch
On Sat, Jun 08, 2013 at 08:09:15PM -0400, Robert Haas wrote: > On Sat, Jun 8, 2013 at 10:25 AM, Andrew Dunstan wrote: > > Don't downcase non-ascii identifier chars in multi-byte encodings. > > > > Long-standing code has called tolower() on identifier character bytes > > with the high bit set. This

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Noah Misch
On Sat, Jun 08, 2013 at 08:20:42PM -0400, Robert Haas wrote: > On Sat, Jun 8, 2013 at 5:41 PM, Noah Misch wrote: > > Likewise; I don't see why we couldn't perform an optimistic check ASAP and > > schedule a final after-statement check when an early check fails. That > > changes performance charac

Re: [HACKERS] Cost limited statements RFC

2013-06-08 Thread Robert Haas
On Sat, Jun 8, 2013 at 4:43 PM, Jeff Janes wrote: > I don't know what two independent setting would look like. Say you keep two > independent counters, where each can trigger a sleep, and the triggering of > that sleep clears only its own counter. Now you still have a limit on the > linear combi

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread MauMau
From: "Josh Berkus" There's actually three potential failure cases here: - One Volume: WAL is on the same volume as PGDATA, and that volume is completely out of space. - XLog Partition: WAL is on its own partition/volume, and fills it up. - Archiving: archiving is failing or too slow, causing

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Robert Haas
On Sat, Jun 8, 2013 at 5:41 PM, Noah Misch wrote: > This does appear to specify FK timing semantics like PostgreSQL gives today. > Namely, it does not permit a FK-induced error when later actions of the query > that prompted the check could possibly remedy the violation. Yeah. Standard or no sta

[HACKERS] Re: [COMMITTERS] pgsql: Don't downcase non-ascii identifier chars in multi-byte encoding

2013-06-08 Thread Robert Haas
On Sat, Jun 8, 2013 at 10:25 AM, Andrew Dunstan wrote: > Don't downcase non-ascii identifier chars in multi-byte encodings. > > Long-standing code has called tolower() on identifier character bytes > with the high bit set. This is clearly an error and produces junk output > when the encoding is mu

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread MauMau
From: "Joshua D. Drake" On 06/08/2013 11:27 AM, Andres Freund wrote: You know, the PANIC isn't there just because we like to piss of users. There's actual technical reasons that don't just go away by judging the PANIC as stupid. Yes I know we aren't trying to piss off users. What I am saying

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread MauMau
From: "Joshua D. Drake" On 06/08/2013 07:36 AM, MauMau wrote: 3. You cannot know the reason of archive_command failure (e.g. archive area full) if you don't use PostgreSQL's server logging. This is because archive_command failure is not logged in syslog/eventlog. Wait, what? Is this true (som

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Jeff Janes
On Sat, Jun 8, 2013 at 11:27 AM, Andres Freund wrote: > > You know, the PANIC isn't there just because we like to piss of > users. There's actual technical reasons that don't just go away by > judging the PANIC as stupid. > At the points where the XLogInsert()s happens we're in critical sections >

Re: [HACKERS] Cost limited statements RFC

2013-06-08 Thread Greg Smith
On 6/8/13 5:20 PM, Kevin Grittner wrote: I'll believe that all of that is true, but I think there's another reason. With multiple layers of write cache (PostgreSQL shared_buffers, OS cache, controller or SSD cache) I think there's a tendency for an "avalanche" effect. Dirty pages stick to cache

Re: [HACKERS] Cost limited statements RFC

2013-06-08 Thread Greg Smith
On 6/8/13 5:17 PM, Jeff Janes wrote: But my gut feeling is that if autovacuum is trying to read faster than the hardware will support, it will just automatically get throttled, by inherent IO waits, at a level which can be comfortably supported. And this will cause minimal interference with oth

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Noah Misch
On Sat, Jun 08, 2013 at 09:39:14PM +0100, Simon Riggs wrote: > On 8 June 2013 15:30, Noah Misch wrote: > > On Tue, Jun 04, 2013 at 02:45:17PM +0100, Simon Riggs wrote: > >> 2. Don't store FK events in the after trigger queue at all, but apply > >> them as we go. That solves problems2 and 3. That

Re: [HACKERS] Proposed patch: remove hard-coded limit MAX_ALLOCATED_DESCS

2013-06-08 Thread Tom Lane
=?iso-8859-1?q?C=E9dric_Villemain?= writes: > I'm not sure of expected value of "max_safe_fds". Your patch now initialize > with 5 slots instead of 10, if max_safe_fds is large maybe it is better to > double the size each time we need instead of jumping directly to the largest > size ? I don't

Re: [HACKERS] Batch API for After Triggers

2013-06-08 Thread Kevin Grittner
Simon Riggs wrote: > Comments please. How much of this problem space do you think could be addressed by providing OLD and NEW *relations* to AFTER EACH STATEMENT triggers? -- Kevin Grittner EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers

Re: [HACKERS] Cost limited statements RFC

2013-06-08 Thread Kevin Grittner
Greg Smith wrote: > I suspect the reason we don't see as many complaints is that a > lot more systems can handle 7.8MB/s of random reads then there > are ones that can do 3.9MB/s of random writes.  If we removed > that read limit, a lot more complaints would start rolling in > about the read side

Re: [HACKERS] Cost limited statements RFC

2013-06-08 Thread Jeff Janes
On Sat, Jun 8, 2013 at 1:57 PM, Greg Smith wrote: > On 6/8/13 4:43 PM, Jeff Janes wrote: > > Also, in all the anecdotes I've been hearing about autovacuum causing >> problems from too much IO, in which people can identify the specific >> problem, it has always been the write pressure, not the re

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Simon Riggs
On 7 June 2013 10:02, Heikki Linnakangas wrote: > On 07.06.2013 00:38, Andres Freund wrote: >> >> On 2013-06-06 23:28:19 +0200, Christian Ullrich wrote: >>> >>> * Heikki Linnakangas wrote: >>> The current situation is that if you run out of disk space while writing WAL, you get a PANIC,

[HACKERS] Batch API for After Triggers

2013-06-08 Thread Simon Riggs
While fiddling with FK tuning, Noah suggested batching trigger executions together to avoid execution overhead. It turns out there is no easy way to write triggers that can take advantage of the knowledge that they are being executed as a set of trigger executions. Some API is required to allow a

Re: [HACKERS] Cost limited statements RFC

2013-06-08 Thread Greg Smith
On 6/8/13 4:43 PM, Jeff Janes wrote: Also, in all the anecdotes I've been hearing about autovacuum causing problems from too much IO, in which people can identify the specific problem, it has always been the write pressure, not the read, that caused the problem. Should the default be to have th

[HACKERS] ALTER TABLE ... ALTER CONSTRAINT

2013-06-08 Thread Simon Riggs
While fiddling with FK tuning, it was useful to be able to enable and disable the DEFERRED mode of constraints. That is not currently possible in SQL, so I wrote this patch. Without this you have to drop and then re-add a constraint, which is impractical for large tables. e.g. CREATE TABLE fktabl

Re: [HACKERS] Cost limited statements RFC

2013-06-08 Thread Jeff Janes
On Thu, Jun 6, 2013 at 1:02 PM, Robert Haas wrote: > > If we can see our way clear to ripping out the autovacuum costing > stuff and replacing them with a read rate limit and a dirty rate > limit, I'd be in favor of that. The current system limits the linear > combination of those with user-spec

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Simon Riggs
On 8 June 2013 15:30, Noah Misch wrote: > On Tue, Jun 04, 2013 at 02:45:17PM +0100, Simon Riggs wrote: >> > On Sun, Jun 02, 2013 at 10:45:21AM +0100, Simon Riggs wrote: >> >> For clarity the 4 problems are >> >> 1. SQL execution overhead >> >> 2. Memory usage >> >> 3. Memory scrolling >> >> 4. Loc

[HACKERS] small patch to crypt.c

2013-06-08 Thread Joshua D. Drake
Hello, In my quest to understand how all the logging etc works with authentication I came across the area of crypt.c that checks for valid_until but it seems like it has an extraneous check. If I am wrong I apologize for the noise but wouldn't mind an explanation. index f01d904..8d809b2 100

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Joshua D. Drake
On 06/08/2013 11:27 AM, Andres Freund wrote: On 2013-06-08 11:15:40 -0700, Joshua D. Drake wrote: To me, a more pragmatic approach makes sense. Obviously having some kind of code that checks the space makes sense but I don't know that it needs to be around any operation other than we are creat

Re: [HACKERS] Cost limited statements RFC

2013-06-08 Thread Jeff Janes
On Thu, Jun 6, 2013 at 2:27 PM, Andres Freund wrote: > On 2013-06-06 12:34:01 -0700, Jeff Janes wrote: > > On Fri, May 24, 2013 at 11:51 AM, Greg Smith > wrote: > > > > > On 5/24/13 9:21 AM, Robert Haas wrote: > > > > > > But I wonder if we wouldn't be better off coming up with a little more > >

Re: [HACKERS] Redesigning checkpoint_segments

2013-06-08 Thread Greg Smith
On 6/7/13 2:43 PM, Robert Haas wrote: name. What I would like to see is a single number here in memory units that replaces both checkpoint_segments and wal_keep_segments. This isn't really making sense to me. I don't think we should assume that someone who wants to keep WAL around for replica

Re: [HACKERS] Proposed patch: remove hard-coded limit MAX_ALLOCATED_DESCS

2013-06-08 Thread Cédric Villemain
> > I just wonder about this statement that you removed: > > * Since we don't want to encourage heavy use of those functions, > > * it seems OK to put a pretty small maximum limit on the number of > > * simultaneously allocated descs. > > > Is it now encouraged to use those functions, or at

Re: [HACKERS] [PATCH] pgbench --throttle (submission 7 - with lag measurement)

2013-06-08 Thread Greg Smith
On 6/1/13 5:00 AM, Fabien COELHO wrote: Question 1: should it report the maximum lang encountered? I haven't found the lag measurement to be very useful yet, outside of debugging the feature itself. Accordingly I don't see a reason to add even more statistics about the number outside of test

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Jeff Janes
On Sat, Jun 8, 2013 at 11:15 AM, Joshua D. Drake wrote: > > On 06/06/2013 07:52 AM, Heikki Linnakangas wrote: > >> I think it can be made fairly robust otherwise, and the performance >> impact should be pretty easy to measure with e.g pgbench. >> > > Once upon a time in a land far, far away, we ex

Re: [HACKERS] Proposed patch: remove hard-coded limit MAX_ALLOCATED_DESCS

2013-06-08 Thread Tom Lane
=?iso-8859-15?q?C=E9dric_Villemain?= writes: > I just wonder about this statement that you removed: > * Since we don't want to encourage heavy use of those functions, > * it seems OK to put a pretty small maximum limit on the number of > * simultaneously allocated descs. > Is it now encour

Re: [HACKERS] Proposed patch: remove hard-coded limit MAX_ALLOCATED_DESCS

2013-06-08 Thread Cédric Villemain
Le samedi 8 juin 2013 19:06:58, Tom Lane a écrit : > Recently we had a gripe about how you can't read very many files > concurrently with contrib/file_fdw: > http://www.postgresql.org/message-id/OF419B5767.8A3C9ADB-ON85257B79.005491E > 9-85257b79.0054f...@isn.rtss.qc.ca > > The reason for this is

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Jeff Janes
On Fri, Jun 7, 2013 at 12:14 PM, Josh Berkus wrote: > > >> The archive command can be made a shell script (or that matter a > >> compiled program) which can do anything it wants upon failure, including > >> emailing people. > > You're talking about using external tools -- frequently hackish, > wo

Re: [HACKERS] Bad error message on valuntil

2013-06-08 Thread Joshua D. Drake
On 06/07/2013 12:31 PM, Tom Lane wrote: "Joshua D. Drake" writes: On 06/07/2013 11:57 AM, Tom Lane wrote: I think it's intentional that we don't tell the *client* that level of detail. Why? That seems rather silly. The general policy on authentication failure reports is that we don't tel

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-08 Thread Kevin Grittner
Tom Lane wrote: > Kevin Grittner writes: >>> If we were a little earlier in the release cycle I would argue that >>> if we're going to do anything with this column we should drop it. > >> Which is exactly what I think we should do as soon as we branch. > > If we're going to do that, there d

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Andres Freund
On 2013-06-07 12:02:57 +0300, Heikki Linnakangas wrote: > On 07.06.2013 00:38, Andres Freund wrote: > >On 2013-06-06 23:28:19 +0200, Christian Ullrich wrote: > >>* Heikki Linnakangas wrote: > >> > >>>The current situation is that if you run out of disk space while writing > >>>WAL, you get a PANIC,

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Andres Freund
On 2013-06-08 11:15:40 -0700, Joshua D. Drake wrote: > To me, a more pragmatic approach makes sense. Obviously having some kind of > code that checks the space makes sense but I don't know that it needs to be > around any operation other than we are creating a segment. What do we care > why the seg

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Joshua D. Drake
On 06/06/2013 07:52 AM, Heikki Linnakangas wrote: I think it can be made fairly robust otherwise, and the performance impact should be pretty easy to measure with e.g pgbench. Once upon a time in a land far, far away, we expected users to manage their own systems. We had things like soft and

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Joshua D. Drake
On 06/08/2013 07:36 AM, MauMau wrote: 1. If the machine or postgres crashes while archive_command is copying a WAL file, later archive recovery fails. This is because cp leaves a file of less than 16MB in archive area, and postgres refuses to start when it finds such a small archive WAL file. T

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread Joshua D. Drake
On 06/07/2013 12:14 PM, Josh Berkus wrote: Right now, what we're telling users is "You can have continuous backup with Postgres, but you'd better hire and expensive consultant to set it up for you, or use this external tool of dubious provenance which there's no packages for, or you might accid

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-08 Thread Tom Lane
Kevin Grittner writes: >> If we were a little earlier in the release cycle I would argue that >> if we're going to do anything with this column we should drop it. > Which is exactly what I think we should do as soon as we branch. If we're going to do that, there doesn't seem to me to be a lot of

[HACKERS] Proposed patch: remove hard-coded limit MAX_ALLOCATED_DESCS

2013-06-08 Thread Tom Lane
Recently we had a gripe about how you can't read very many files concurrently with contrib/file_fdw: http://www.postgresql.org/message-id/of419b5767.8a3c9adb-on85257b79.005491e9-85257b79.0054f...@isn.rtss.qc.ca The reason for this is that file_fdw goes through the COPY code, which uses AllocateFil

Re: [HACKERS] system catalog pg_rewrite column ev_attr document description problem

2013-06-08 Thread Kevin Grittner
Kevin Grittner wrote: > I'll leave this alone for a day.  If nobody objects, I will change > the ruleutils.c code to work with either value (to support > pg_upgrade) and change the code to set this to zero, for 9.3 and > forward only.  I will change the 9.3 docs to mention that newly > created ro

Re: [HACKERS] UTF-8 encoding problem w/ libpq

2013-06-08 Thread Andrew Dunstan
On 06/03/2013 02:41 PM, Andrew Dunstan wrote: On 06/03/2013 02:28 PM, Tom Lane wrote: . I wonder though if we couldn't just fix this code to not do anything to high-bit-set bytes in multibyte encodings. That's exactly what I suggested back in November. This thread seems to have gone cold

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2013-06-08 Thread MauMau
From: "Daniel Farina" On Fri, Jun 7, 2013 at 12:14 PM, Josh Berkus wrote: Right now, what we're telling users is "You can have continuous backup with Postgres, but you'd better hire and expensive consultant to set it up for you, or use this external tool of dubious provenance which there's no

Re: [HACKERS] Optimising Foreign Key checks

2013-06-08 Thread Noah Misch
On Tue, Jun 04, 2013 at 02:45:17PM +0100, Simon Riggs wrote: > > On Sun, Jun 02, 2013 at 10:45:21AM +0100, Simon Riggs wrote: > >> For clarity the 4 problems are > >> 1. SQL execution overhead > >> 2. Memory usage > >> 3. Memory scrolling > >> 4. Locking overhead, specifically FPWs and WAL records

Re: [HACKERS] Possible bug in cascaded standby

2013-06-08 Thread Pavan Deolasee
> I'll retry and report back if I see the problem on the offending platform. > > Just to close out this thread, I can't reproduce this on the Mac OS either. While I'd done a "make clean" earlier, "make distclean" did the trick. Sorry for the noise. Thanks, Pavan -- Pavan Deolasee http://www.link