Re: [HACKERS] Streaming replication for psycopg2

2015-10-04 Thread Craig Ringer
On 30 June 2015 at 22:42, Shulgin, Oleksandr wrote: > On Thu, Jun 4, 2015 at 5:49 PM, Shulgin, Oleksandr > wrote: >> >> On Tue, Jun 2, 2015 at 2:23 PM, Shulgin, Oleksandr >> wrote: >> > >> > I've submitted a patch to psycopg2 to support streaming replication >> > protocol (COPY_BOTH): https://gi

Re: [HACKERS] Streaming Replication clusters and load balancing

2015-09-21 Thread Josh Berkus
On 09/17/2015 07:27 PM, James Sewell wrote: > Hello all, > > I have recently been working with PostgreSQL and HAProxy to provide > seamless load balancing to a group of database servers. This on it's own > isn't a hard thing: I have an implementation finished and am now > thinking about the best w

Re: [HACKERS] Streaming Replication clusters and load balancing

2015-09-21 Thread Dmitry Vasilyev
Hi! By default, HAproxy configuration can not be changed without breaking a connection with the client :) -- Dmitry Vasilyev Postgres Professional: http://www.postgrespro.com The Russian Postgres Company On Fri, 2015-09-18 at 12:27 +1000, James Sewell wrote: > Hello all, > > I have recently

Re: [HACKERS] Streaming replication for psycopg2

2015-06-30 Thread Shulgin, Oleksandr
On Thu, Jun 4, 2015 at 5:49 PM, Shulgin, Oleksandr < oleksandr.shul...@zalando.de> wrote: > On Tue, Jun 2, 2015 at 2:23 PM, Shulgin, Oleksandr < > oleksandr.shul...@zalando.de> wrote: > > > > Hello, > > > > I've submitted a patch to psycopg2 to support streaming replication > protocol (COPY_BOTH):

Re: [HACKERS] Streaming replication for psycopg2

2015-06-04 Thread Shulgin, Oleksandr
On Tue, Jun 2, 2015 at 2:23 PM, Shulgin, Oleksandr < oleksandr.shul...@zalando.de> wrote: > > Hello, > > I've submitted a patch to psycopg2 to support streaming replication protocol (COPY_BOTH): https://github.com/psycopg/psycopg2/pull/322 > > It would be great if more people had a chance to take a

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-13 Thread Heikki Linnakangas
On 05/13/2015 04:29 PM, Robert Haas wrote: On Wed, May 13, 2015 at 8:53 AM, Heikki Linnakangas wrote: Our manual says that archive_command should refuse to overwrite an existing file. But to work-around the double-archival problem, where the same file is archived twice, it would be even better

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-13 Thread Robert Haas
On Wed, May 13, 2015 at 8:53 AM, Heikki Linnakangas wrote: > Our manual says that archive_command should refuse to overwrite an existing > file. But to work-around the double-archival problem, where the same file is > archived twice, it would be even better if it would simply return success if > t

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-13 Thread Heikki Linnakangas
On 05/13/2015 03:36 PM, Robert Haas wrote: On Mon, May 11, 2015 at 12:00 PM, Heikki Linnakangas wrote: And here is a new version of the patch. I kept the approach of using pgstat, but it now only polls pgstat every 10 seconds, and doesn't block to wait for updated stats. It's not entirely a n

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-13 Thread Robert Haas
On Mon, May 11, 2015 at 12:00 PM, Heikki Linnakangas wrote: > And here is a new version of the patch. I kept the approach of using pgstat, > but it now only polls pgstat every 10 seconds, and doesn't block to wait for > updated stats. It's not entirely a new problem, but this error message has go

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-11 Thread Heikki Linnakangas
On 05/08/2015 04:21 PM, Heikki Linnakangas wrote: On 04/22/2015 10:07 AM, Michael Paquier wrote: On Wed, Apr 22, 2015 at 3:38 PM, Heikki Linnakangas wrote: I feel that the best approach is to archive the last, partial segment, but with the .partial suffix. I don't see any plausible real-wold s

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-05-08 Thread Heikki Linnakangas
On 04/22/2015 10:07 AM, Michael Paquier wrote: On Wed, Apr 22, 2015 at 3:38 PM, Heikki Linnakangas wrote: I feel that the best approach is to archive the last, partial segment, but with the .partial suffix. I don't see any plausible real-wold setup where the current behavior would be better. I

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-23 Thread Heikki Linnakangas
On 04/22/2015 11:58 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 3:34 PM, Heikki Linnakangas wrote: On 04/22/2015 10:21 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas wrote: For example, imagine that perform point-in-time recovery to WAL position 0/1237E568, on

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Robert Haas
On Wed, Apr 22, 2015 at 3:34 PM, Heikki Linnakangas wrote: > On 04/22/2015 10:21 PM, Robert Haas wrote: >> On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas >> wrote: >>> For example, imagine that perform point-in-time recovery to WAL position >>> 0/1237E568, on timeline 1. That falls within se

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Heikki Linnakangas
On 04/22/2015 10:21 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas wrote: For example, imagine that perform point-in-time recovery to WAL position 0/1237E568, on timeline 1. That falls within segment 00010012. Then we end recovery, and switch to timel

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Robert Haas
On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas wrote: > On 04/22/2015 09:30 PM, Robert Haas wrote: >> On Wed, Apr 22, 2015 at 2:17 AM, Heikki Linnakangas >> wrote: >>> >>> Note that it's a bit complicated to set up that scenario today. Archiving >>> is >>> never enabled in recovery mode, so

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Heikki Linnakangas
On 04/22/2015 09:30 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 2:17 AM, Heikki Linnakangas wrote: Note that it's a bit complicated to set up that scenario today. Archiving is never enabled in recovery mode, so you'll need to use a custom cron job or something to maintain the archive that C

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Robert Haas
On Tue, Apr 21, 2015 at 8:30 PM, Michael Paquier wrote: > This .partial segment renaming is something that we > should let the archive_command manage with its internal logic. This strikes me as equivalent to saying "we don't know how to make this work right, but maybe our users will know". That

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Robert Haas
On Wed, Apr 22, 2015 at 2:17 AM, Heikki Linnakangas wrote: > Note that it's a bit complicated to set up that scenario today. Archiving is > never enabled in recovery mode, so you'll need to use a custom cron job or > something to maintain the archive that C uses. The files will not > automatically

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-22 Thread Michael Paquier
On Wed, Apr 22, 2015 at 3:38 PM, Heikki Linnakangas wrote: > On 04/22/2015 03:30 AM, Michael Paquier wrote: >> >> This is going to change a behavior that people are used to for a >> couple of releases. I would not mind having this patch do >> "archive_mode = on during recovery" => archive only seg

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Heikki Linnakangas
On 04/22/2015 03:30 AM, Michael Paquier wrote: This is going to change a behavior that people are used to for a couple of releases. I would not mind having this patch do "archive_mode = on during recovery" => archive only segments generated by this node + the last partial segment on the old timel

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Heikki Linnakangas
On 04/22/2015 12:42 AM, Robert Haas wrote: On Tue, Apr 21, 2015 at 6:55 AM, Heikki Linnakangas wrote: On 04/21/2015 12:04 PM, Michael Paquier wrote: On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas wrote: Note that even though we don't archive the partial last segment on the previous time

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Michael Paquier
On Wed, Apr 22, 2015 at 6:42 AM, Robert Haas wrote: > > On Tue, Apr 21, 2015 at 6:55 AM, Heikki Linnakangas wrote: > > On 04/21/2015 12:04 PM, Michael Paquier wrote: > >> On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas > >> wrote: > >>> Note that even though we don't archive the partial last

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Robert Haas
On Tue, Apr 21, 2015 at 6:55 AM, Heikki Linnakangas wrote: > On 04/21/2015 12:04 PM, Michael Paquier wrote: >> On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas >> wrote: >>> Note that even though we don't archive the partial last segment on the >>> previous timeline, the same WAL is copied to

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Heikki Linnakangas
On 04/21/2015 12:04 PM, Michael Paquier wrote: On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas wrote: Note that even though we don't archive the partial last segment on the previous timeline, the same WAL is copied to the first segment on the new timeline. So the WAL isn't lost. But if t

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Michael Paquier
On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas wrote: > On 04/21/2015 09:53 AM, Michael Paquier wrote: > >> On Thu, Apr 16, 2015 at 8:57 PM, Heikki Linnakangas wrote: >> >>> Oh, hang on, that's not necessarily true. On promotion, the standby >>> >> archives >> >>> the last, partial WAL segme

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-21 Thread Heikki Linnakangas
On 04/21/2015 09:53 AM, Michael Paquier wrote: On Thu, Apr 16, 2015 at 8:57 PM, Heikki Linnakangas wrote: Oh, hang on, that's not necessarily true. On promotion, the standby archives the last, partial WAL segment from the old timeline. That's just wrong (http://www.postgresql.org/message-id/52

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-20 Thread Michael Paquier
On Thu, Apr 16, 2015 at 8:57 PM, Heikki Linnakangas wrote: > Oh, hang on, that's not necessarily true. On promotion, the standby archives > the last, partial WAL segment from the old timeline. That's just wrong > (http://www.postgresql.org/message-id/52fcd37c.3070...@vmware.com), and in > fact I so

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-16 Thread Heikki Linnakangas
On 03/01/2015 12:36 AM, Venkata Balaji N wrote: Patch did get applied successfully to the latest master. Can you please rebase. Here you go. On 01/31/2015 03:07 PM, Andres Freund wrote: On 2014-12-19 22:56:40 +0200, Heikki Linnakangas wrote: This add two new archive_modes, 'shared' and 'alwa

Re: [HACKERS] Streaming replication

2015-03-31 Thread Greg Stark
The key word you're misunderstanding is "filled". It means it doesn't wait for the 16MB file to be completely filled with records. I.e. what would happen in the file shipping form of replication.

Re: [HACKERS] Streaming replication

2015-03-30 Thread Tatsuo Ishii
> On 31/03/15 12:45, Tatsuo Ishii wrote: >> In the doc: >> >> 25.2.5. Streaming Replication >> : >> The standby connects to the primary, which streams WAL records to the >> standby as they're generated, without waiting for the WAL file to be >> filled. >> >> This seems to claim that walsender sen

Re: [HACKERS] Streaming replication

2015-03-30 Thread Mark Kirkwood
On 31/03/15 12:45, Tatsuo Ishii wrote: > In the doc: > > 25.2.5. Streaming Replication > : > The standby connects to the primary, which streams WAL records to the > standby as they're generated, without waiting for the WAL file to be > filled. > > This seems to claim that walsender sends WAL file

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-02-28 Thread Venkata Balaji N
> > > Here's a first cut at this. It includes the changes from your > standby_wal_archiving_v1.patch, so you get that behaviour if you set > archive_mode='always', and the new behaviour I wanted with > archive_mode='shared'. I wrote it on top of the other patch I posted > recently to not archive bo

[HACKERS] Re: [HACKERS] Streaming replication and WAL archive interactions

2015-02-11 Thread Миша Тюрин
>  This should be a very common setup in the field, so how are  people doing it >in practice? One of possible workaround with archive and streaming was to use pg_receivexlog from standby to copy/save WALs to archive. but with pg_receivexlog was also issue with fsync. [ master ] -- streaming 

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-01-31 Thread Andres Freund
Hi, On 2014-12-19 22:56:40 +0200, Heikki Linnakangas wrote: > This add two new archive_modes, 'shared' and 'always', to indicate whether > the WAL archive is shared between the primary and standby, or not. In > shared mode, the standby tracks which files have been archived by the > primary. The st

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-19 Thread Heikki Linnakangas
On 12/18/2014 12:32 PM, Fujii Masao wrote: On Wed, Dec 17, 2014 at 4:11 AM, Heikki Linnakangas wrote: On 12/16/2014 10:24 AM, Borodin Vladimir wrote: 12 дек. 2014 г., в 16:46, Heikki Linnakangas написал(а): There have been a few threads on the behavior of WAL archiving, after a standby ser

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-18 Thread Fujii Masao
On Wed, Dec 17, 2014 at 4:11 AM, Heikki Linnakangas wrote: > On 12/16/2014 10:24 AM, Borodin Vladimir wrote: >> >> 12 дек. 2014 г., в 16:46, Heikki Linnakangas >> написал(а): >> >>> There have been a few threads on the behavior of WAL archiving, >>> after a standby server is promoted [1] [2]. In

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-16 Thread Heikki Linnakangas
On 12/16/2014 10:24 AM, Borodin Vladimir wrote: 12 дек. 2014 г., в 16:46, Heikki Linnakangas написал(а): There have been a few threads on the behavior of WAL archiving, after a standby server is promoted [1] [2]. In short, it doesn't work as you might expect. The standby will start archiving a

Re: [HACKERS] Streaming replication and WAL archive interactions

2014-12-16 Thread Borodin Vladimir
12 дек. 2014 г., в 16:46, Heikki Linnakangas написал(а): > There have been a few threads on the behavior of WAL archiving, after a > standby server is promoted [1] [2]. In short, it doesn't work as you might > expect. The standby will start archiving after it's promoted, but it will not > ar

Re: [HACKERS] Streaming replication bug in 9.3.2, "WAL contains references to invalid pages"

2014-01-03 Thread Omar Kilani
We had the same issues running 9.2.4: [2013-10-15 00:23:01 GMT/0/15396] WARNING: page 8789807 of relation base/16429/2349631976 is uninitialized [2013-10-15 00:23:01 GMT/0/15396] CONTEXT: xlog redo vacuum: rel 1663/16429/2349631976; blk 8858544, lastBlockVacuumed 0 [2013-10-15 00:23:01 GMT/0/153

Re: [HACKERS] Streaming replication bug in 9.3.2, "WAL contains references to invalid pages"

2014-01-03 Thread Sergey Konoplev
On Thu, Jan 2, 2014 at 11:59 AM, Christophe Pettus wrote: > In both cases, the indicated relation was a primary key index. In one case, > rebuilding the primary key index caused the problem to go away permanently > (to date). In the second case, the problem returned even after a full dump / >

Re: [HACKERS] Streaming replication bug in 9.3.2, "WAL contains references to invalid pages"

2014-01-02 Thread MauMau
From: "Christophe Pettus" We've had two clients experience a crash on the secondary of a streaming replication pair, running PostgreSQL 9.3.2. In both cases, the messages were close to this example: 2013-12-30 18:08:00.464 PST,,,23869,,52ab4839.5d3d,16,,2013-12-13 09:47:37 PST,1/0,0,WARNING

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-29 Thread David Powers
It's another possibility, but I think it's still somewhat remote given how long we've been using this method with this code. It's sadly hard to test because taking the full backup without the hard linking is fairly expensive (the databases comprise multiple terabytes). As a possibly unsatisfying

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-28 Thread Robert Haas
On Tue, May 28, 2013 at 10:53 AM, Benedikt Grundmann wrote: > Today we have seen > > 2013-05-28 04:11:12.300 EDT,,,30600,,51a41946.7788,1,,2013-05-27 22:41:10 > EDT,,0,ERROR,XX000,"xlog flush request 1E95/AFB2DB10 is not satisfied --- > flushed only to 1E7E/21CB79A0","writing block 9 of relati

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-28 Thread Benedikt Grundmann
Today we have seen 2013-05-28 04:11:12.300 EDT,,,30600,,51a41946.7788,1,,2013-05-27 22:41:10 EDT,,0,ERROR,XX000,"xlog flush request 1E95/AFB2DB10 is not satisfied --- flushed only to 1E7E/21CB79A0","writing block 9 of relation base/16416/293974676""" 2013-05-28 04:11:13.316 EDT,,,30600,,51

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-23 Thread David Powers
Thanks for the response. I have some evidence against an issue in the backup procedure (though I'm not ruling it out). We moved back to taking the backup off of the primary and all errors for all three clusters went away. All of the hardware is the same, OS and postgres versions are largely the

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-23 Thread Robert Haas
On Tue, May 21, 2013 at 11:59 AM, Benedikt Grundmann wrote: > We are seeing these errors on a regular basis on the testing box now. We > have even changed the backup script to > shutdown the hot standby, take lvm snapshot, restart the hot standby, rsync > the lvm snapshot. It still happens. > >

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-21 Thread Benedikt Grundmann
We are seeing these errors on a regular basis on the testing box now. We have even changed the backup script to shutdown the hot standby, take lvm snapshot, restart the hot standby, rsync the lvm snapshot. It still happens. We have never seen this before we introduced the hot standby. So we wil

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-16 Thread David Powers
I'll try to get the primary upgraded over the weekend when we can afford a restart. In the meantime I have a single test showing that a shutdown, snapshot, restart produces a backup that passes the vacuum analyze test. I'm going to run a full vacuum today. -David On Wed, May 15, 2013 at 3:53 P

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-15 Thread Heikki Linnakangas
On 15.05.2013 22:50, Benedikt Grundmann wrote: On Wed, May 15, 2013 at 2:50 PM, Heikki Linnakangas The subject says 9.2.3. Are you sure you're running 9.2.4 on all the servers? There was a fix to a bug related to starting a standby server from a filesystem snapshot. I don't think it was quite the

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-15 Thread Benedikt Grundmann
On Wed, May 15, 2013 at 2:50 PM, Heikki Linnakangas wrote: > On 15.05.2013 15:42, David Powers wrote: > >> First, thanks for the replies. This sort of thing is frustrating and hard >> to diagnose at a distance, and any help is appreciated. >> >> Here is some more background: >> >> We have 3 9.2.

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-15 Thread Heikki Linnakangas
On 15.05.2013 15:42, David Powers wrote: First, thanks for the replies. This sort of thing is frustrating and hard to diagnose at a distance, and any help is appreciated. Here is some more background: We have 3 9.2.4 databases using the following setup: The subject says 9.2.3. Are you sure y

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-15 Thread David Powers
First, thanks for the replies. This sort of thing is frustrating and hard to diagnose at a distance, and any help is appreciated. Here is some more background: We have 3 9.2.4 databases using the following setup: - A primary box - A standby box running as a hot streaming replica from the primar

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-14 Thread Heikki Linnakangas
On 14.05.2013 23:47, Benedikt Grundmann wrote: The only thing that is *new* is that we took the snapshot from the streaming replica. So again my best guess as of now is that if the database crashes while it is in streaming standby a invalid disk state can result during during the following start

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-14 Thread Amit Kapila
On Tuesday, May 14, 2013 7:19 PM Benedikt Grundmann wrote: >It's on the production database and the streaming replica.  But not on the snapshot. > production > -rw--- 1 postgres postgres 312778752 May 13 21:28 /database/postgres/base/16416/291498116.3 > streaming replica > -rw--- 1 postgr

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-14 Thread Benedikt Grundmann
I think my previous message wasn't clear enough. I do *NOT* think that LVM snapshot is the culprit. However I cannot discount it as one of the possibilities. But I have no evidence in either /var/log/messages or in dmesg that the LVM snapshot went into a bad state AND we have been using this met

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-14 Thread Benedikt Grundmann
That's one possible explanation. It's worth noting that we haven't seen this before moving to streaming rep first and we have been using that method for a long time. On Tue, May 14, 2013 at 11:34 AM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 14.05.2013 16:48, Benedikt Grundmann

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-14 Thread Heikki Linnakangas
On 14.05.2013 16:48, Benedikt Grundmann wrote: It's on the production database and the streaming replica. But not on the snapshot. So, the LVM snapshot didn't work correctly? - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-14 Thread Benedikt Grundmann
It's on the production database and the streaming replica. But not on the snapshot. production -rw--- 1 postgres postgres 312778752 May 13 21:28 /database/postgres/base/16416/291498116.3 streaming replica -rw--- 1 postgres postgres 312778752 May 13 23:50 /database/postgres/base/16416/291

Re: [HACKERS] streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

2013-05-14 Thread Heikki Linnakangas
On 14.05.2013 14:57, Benedikt Grundmann wrote: Today we have seen this on our testing database instance: ERROR: could not open file "base/16416/291498116.3" (target block 431006): No such file or directory That database get's created by rsyncing the LVM snapshot of the standby, which is a read

Re: [HACKERS] Streaming replication as a separate permissions

2011-01-05 Thread Peter Eisentraut
On tis, 2011-01-04 at 22:24 -0500, Robert Haas wrote: > > Just to be clear: are we saying that "CREATE ROLE foo SUPERUSER" > > should grant both superuser and replication, as well as the default > > "postgres" user also having replication as well? > > I think that's what we're saying. So now supe

Re: [HACKERS] Streaming replication as a separate permissions

2011-01-05 Thread Peter Eisentraut
On mån, 2011-01-03 at 11:20 -0500, Tom Lane wrote: > You might want to reflect on rolcatupdate a bit before asserting that > there are no cases where privileges are ever denied to superusers. Arguably, the reason that that is hardly documented and slightly deprecated is that the underlying design

Re: [HACKERS] Streaming replication as a separate permissions

2011-01-05 Thread Robert Haas
On Wed, Jan 5, 2011 at 8:24 AM, Magnus Hagander wrote: > Ok, done and applied. Thanks. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://

Re: [HACKERS] Streaming replication as a separate permissions

2011-01-05 Thread Magnus Hagander
On Wed, Jan 5, 2011 at 04:24, Robert Haas wrote: > On Mon, Jan 3, 2011 at 5:50 PM, Magnus Hagander wrote: >> On Mon, Jan 3, 2011 at 17:23, Robert Haas wrote: >>> On Mon, Jan 3, 2011 at 11:20 AM, Tom Lane wrote: Robert Haas writes: > On the other hand, the REPLICATION privilege is deny

Re: [HACKERS] Streaming replication as a separate permissions

2011-01-04 Thread Robert Haas
On Mon, Jan 3, 2011 at 5:50 PM, Magnus Hagander wrote: > On Mon, Jan 3, 2011 at 17:23, Robert Haas wrote: >> On Mon, Jan 3, 2011 at 11:20 AM, Tom Lane wrote: >>> Robert Haas writes: On the other hand, the REPLICATION privilege is denying you the right to perform an operation *even tho

Re: [HACKERS] Streaming replication as a separate permissions

2011-01-03 Thread Magnus Hagander
On Mon, Jan 3, 2011 at 17:23, Robert Haas wrote: > On Mon, Jan 3, 2011 at 11:20 AM, Tom Lane wrote: >> Robert Haas writes: >>> On the other hand, the REPLICATION privilege is denying you the right to >>> perform an operation *even though you already are authenticated as a >>> superuser*.  I don'

Re: [HACKERS] Streaming replication as a separate permissions

2011-01-03 Thread Robert Haas
On Mon, Jan 3, 2011 at 11:20 AM, Tom Lane wrote: > Robert Haas writes: >> On the other hand, the REPLICATION privilege is denying you the right to >> perform an operation *even though you already are authenticated as a >> superuser*.  I don't think there's anywhere else in the system where >> we

Re: [HACKERS] Streaming replication as a separate permissions

2011-01-03 Thread Tom Lane
Robert Haas writes: > On the other hand, the REPLICATION privilege is denying you the right to > perform an operation *even though you already are authenticated as a > superuser*. I don't think there's anywhere else in the system where > we allow a privilege to non-super-users but deny that same

Re: [HACKERS] Streaming replication as a separate permissions

2011-01-03 Thread Robert Haas
On Mon, Jan 3, 2011 at 6:00 AM, Magnus Hagander wrote: > On Fri, Dec 31, 2010 at 15:38, Magnus Hagander wrote: >> On Thu, Dec 30, 2010 at 15:54, Peter Eisentraut wrote: >>> On ons, 2010-12-29 at 11:09 +0100, Magnus Hagander wrote: I've applied this version (with some minor typo-fixes). >>>

Re: [HACKERS] Streaming replication as a separate permissions

2011-01-03 Thread Magnus Hagander
On Fri, Dec 31, 2010 at 15:38, Magnus Hagander wrote: > On Thu, Dec 30, 2010 at 15:54, Peter Eisentraut wrote: >> On ons, 2010-12-29 at 11:09 +0100, Magnus Hagander wrote: >>> I've applied this version (with some minor typo-fixes). >> >> This page is now somewhat invalidated: >> >> http://develop

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-31 Thread Magnus Hagander
On Thu, Dec 30, 2010 at 15:54, Peter Eisentraut wrote: > On ons, 2010-12-29 at 11:09 +0100, Magnus Hagander wrote: >> I've applied this version (with some minor typo-fixes). > > This page is now somewhat invalidated: > > http://developer.postgresql.org/pgdocs/postgres/role-attributes.html Hmm. So

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-30 Thread Tom Lane
Magnus Hagander writes: > But yes, I see in commit 12c942383296bd626131241c012c2ab81b081738 the > comment "convert some keywords.c symbols to KEYWORD_P to prevent > conflict". > Based on that, I should probably change it back, right? I just tried a > patch for it and it compiles and checks just f

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-30 Thread Robert Haas
On Thu, Dec 30, 2010 at 9:54 AM, Peter Eisentraut wrote: > On tor, 2010-12-23 at 17:29 -0500, Tom Lane wrote: >> Josh Berkus writes: >> > On 12/23/10 2:21 PM, Tom Lane wrote: >> >> Well, that's one laudable goal here, but "secure by default" is another >> >> one that ought to be taken into consid

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-30 Thread Peter Eisentraut
On ons, 2010-12-29 at 11:09 +0100, Magnus Hagander wrote: > I've applied this version (with some minor typo-fixes). This page is now somewhat invalidated: http://developer.postgresql.org/pgdocs/postgres/role-attributes.html First, it doesn't mention the replication privilege, and second it conti

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-30 Thread Peter Eisentraut
On tor, 2010-12-23 at 17:29 -0500, Tom Lane wrote: > Josh Berkus writes: > > On 12/23/10 2:21 PM, Tom Lane wrote: > >> Well, that's one laudable goal here, but "secure by default" is another > >> one that ought to be taken into consideration. > > > I don't see how *not* granting the superuser rep

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-30 Thread Alvaro Herrera
Excerpts from Magnus Hagander's message of jue dic 30 08:57:09 -0300 2010: > On Wed, Dec 29, 2010 at 20:12, Alvaro Herrera > wrote: > > Some lexer keywords have a _P prefix because otherwise they'd collide > > with some symbol in Windows header files or something like that.  It's > > old stuff, b

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-30 Thread Magnus Hagander
On Wed, Dec 29, 2010 at 20:12, Alvaro Herrera wrote: > Excerpts from Magnus Hagander's message of mié dic 29 11:40:34 -0300 2010: >> On Wed, Dec 29, 2010 at 15:05, Gurjeet Singh wrote: > >> > Any specific reason NOREPLICATION_P and REPLICATION_P use the _P suffix? >> >> Um, I just copied it off a

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-29 Thread Alvaro Herrera
Excerpts from Magnus Hagander's message of mié dic 29 11:40:34 -0300 2010: > On Wed, Dec 29, 2010 at 15:05, Gurjeet Singh wrote: > > Any specific reason NOREPLICATION_P and REPLICATION_P use the _P suffix? > > Um, I just copied it off a similar entry elsewhere. I saw no comment > about what _P a

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-29 Thread Magnus Hagander
On Wed, Dec 29, 2010 at 15:05, Gurjeet Singh wrote: > On Wed, Dec 29, 2010 at 5:09 AM, Magnus Hagander > wrote: >> >> > Ok, here's an updated patch that does both these and includes >> > documentation and regression test changes. With that, I think we're >> > good to go. >> >> I've applied this v

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-29 Thread Gurjeet Singh
On Wed, Dec 29, 2010 at 5:09 AM, Magnus Hagander wrote: > > Ok, here's an updated patch that does both these and includes > > documentation and regression test changes. With that, I think we're > > good to go. > > I've applied this version (with some minor typo-fixes). > > Do you think we could ha

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-29 Thread Magnus Hagander
On Tue, Dec 28, 2010 at 13:05, Magnus Hagander wrote: > On Mon, Dec 27, 2010 at 22:53, Magnus Hagander wrote: >> On Mon, Dec 27, 2010 at 22:42, Tom Lane wrote: >>> Magnus Hagander writes: Updated patch, still pending docs, but otherwise updated: allow start/stop backup, make sure only

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Magnus Hagander
On Mon, Dec 27, 2010 at 22:42, Tom Lane wrote: > Magnus Hagander writes: >> Updated patch, still pending docs, but otherwise updated: allow >> start/stop backup, make sure only superuser can turn on/off the flag, >> include in system views, show properly in psql. > > I'd suggest avoiding creating

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Tom Lane
Magnus Hagander writes: > Updated patch, still pending docs, but otherwise updated: allow > start/stop backup, make sure only superuser can turn on/off the flag, > include in system views, show properly in psql. I'd suggest avoiding creating the static cache variable AuthenticatedUserIsReplicatio

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Magnus Hagander
On Mon, Dec 27, 2010 at 16:40, Magnus Hagander wrote: > On Mon, Dec 27, 2010 at 16:33, Tom Lane wrote: >> Magnus Hagander writes: >>> On Mon, Dec 27, 2010 at 10:53, Magnus Hagander wrote: We could quite easily make a replication role *never* be able to connect to a non-walsender backe

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Magnus Hagander
On Mon, Dec 27, 2010 at 16:45, Simon Riggs wrote: > On Mon, 2010-12-27 at 14:54 +0100, Magnus Hagander wrote: > >> You will certainly be able to log into the standby with a superuser >> account, nobody is preventing that. This is about protecting the >> *master*. For example, from modifications ma

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Simon Riggs
On Mon, 2010-12-27 at 14:54 +0100, Magnus Hagander wrote: > You will certainly be able to log into the standby with a superuser > account, nobody is preventing that. This is about protecting the > *master*. For example, from modifications made by a user who hacked > the standby. The users for ma

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Magnus Hagander
On Mon, Dec 27, 2010 at 16:33, Tom Lane wrote: > Magnus Hagander writes: >> On Mon, Dec 27, 2010 at 10:53, Magnus Hagander wrote: >>> We could quite easily make a replication role *never* be able to >>> connect to a non-walsender backend. That would mean that if you set >>> your role to WITH REP

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Tom Lane
Magnus Hagander writes: > On Mon, Dec 27, 2010 at 10:53, Magnus Hagander wrote: >> We could quite easily make a replication role *never* be able to >> connect to a non-walsender backend. That would mean that if you set >> your role to WITH REPLICATION, it can *only* be used for replication >> and

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Magnus Hagander
On Mon, Dec 27, 2010 at 14:51, Simon Riggs wrote: > On Mon, 2010-12-27 at 14:41 +0100, Magnus Hagander wrote: >> >> > >> >> > Where does pg_start_backup()/stop fit? >> >> >> >> Good question :) >> >> >> >> Given that the integrated-base-backup would call it for you, that one >> >> would definitely

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Simon Riggs
On Mon, 2010-12-27 at 14:41 +0100, Magnus Hagander wrote: > >> > > >> > Where does pg_start_backup()/stop fit? > >> > >> Good question :) > >> > >> Given that the integrated-base-backup would call it for you, that one > >> would definitely get it automatically. > >> > >> Given that the latest disci

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Magnus Hagander
On Mon, Dec 27, 2010 at 14:25, Simon Riggs wrote: > On Mon, 2010-12-27 at 12:00 +0100, Magnus Hagander wrote: >> On Mon, Dec 27, 2010 at 11:34, Simon Riggs wrote: >> > On Mon, 2010-12-27 at 10:36 +0100, Magnus Hagander wrote: >> >> > Is backup part of this new privilege, or not? >> >> >> >> The "

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Simon Riggs
On Mon, 2010-12-27 at 12:00 +0100, Magnus Hagander wrote: > On Mon, Dec 27, 2010 at 11:34, Simon Riggs wrote: > > On Mon, 2010-12-27 at 10:36 +0100, Magnus Hagander wrote: > >> > Is backup part of this new privilege, or not? > >> > >> The "integrated base backup", once we have that, that's based o

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Florian Pflug
On Dec27, 2010, at 12:15 , Magnus Hagander wrote: > Actually, having implemented that and tested it, I realize that's a > pretty bad idea. For one thing, it broke my own pg_streamrecv program, > since it requires the ability to connect to the master and select a > pg_current_xlog_location(). I'm s

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Magnus Hagander
On Mon, Dec 27, 2010 at 10:53, Magnus Hagander wrote: > On Fri, Dec 24, 2010 at 05:46, Stephen Frost wrote: >> * Robert Haas (robertmh...@gmail.com) wrote: >>> I think I agree with Florian about the confusing-ness of the proposed >>> semantics.  Aren't you saying you want NOLOGIN mean "not allowe

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Magnus Hagander
On Mon, Dec 27, 2010 at 11:34, Simon Riggs wrote: > On Mon, 2010-12-27 at 10:36 +0100, Magnus Hagander wrote: >> > Is backup part of this new privilege, or not? >> >> The "integrated base backup", once we have that, that's based on the >> walsender protocol? yes. >> pg_dump style backups? No. > >

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Simon Riggs
On Mon, 2010-12-27 at 10:36 +0100, Magnus Hagander wrote: > > Is backup part of this new privilege, or not? > > The "integrated base backup", once we have that, that's based on the > walsender protocol? yes. > pg_dump style backups? No. Where does pg_start_backup()/stop fit? -- Simon Riggs

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Magnus Hagander
On Fri, Dec 24, 2010 at 05:46, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: >> I think I agree with Florian about the confusing-ness of the proposed >> semantics.  Aren't you saying you want NOLOGIN mean "not allowed to >> log in for the purposes of issuing SQL commands, but

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Dave Page
On Mon, Dec 27, 2010 at 9:36 AM, Magnus Hagander wrote: > Seeing logged SQL isn't - but being able to filter the logfiles on > that requires a *lot* more than just defining a security privilege. If > we mean "arbitrary log file reading", the easiest way to fix that > would be to stop checking for

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Magnus Hagander
On Mon, Dec 27, 2010 at 09:32, Simon Riggs wrote: > On Thu, 2010-12-23 at 10:53 +0100, Magnus Hagander wrote: > >> Here's a patch that changes walsender to require a special privilege >> for replication instead of relying on superuser permissions. We >> discussed this back before 9.0 was finalized

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-27 Thread Simon Riggs
On Thu, 2010-12-23 at 10:53 +0100, Magnus Hagander wrote: > Here's a patch that changes walsender to require a special privilege > for replication instead of relying on superuser permissions. We > discussed this back before 9.0 was finalized, but IIRC we ran out of > time. The motivation being tha

Re: [HACKERS] Streaming replication as a separate permissions

2010-12-24 Thread Florian Pflug
On Dec24, 2010, at 05:00 , Tom Lane wrote: > Florian Pflug writes: >> The problem here is that you suggest NOLOGIN should mean "Not allowed >> to issue SQL commands", which really isn't what the name "NOLOGIN" >> conveys. > > No, it means "not allowed to connect". Exactly. Which proves my point,

  1   2   3   4   5   6   7   >