Re: [GENERAL] Data Replication

2008-12-12 Thread Joshua D. Drake
On Fri, 2008-12-12 at 18:45 +0100, Markus Wanner wrote: > Hello Joshua, > Well, yeah, maybe Postgres-R is going to loose that sale as well. But > hey, it's not long ago since you've open sourced it. What makes you > think that you've already "lost that sale"? I for example didn't find > time to lo

Re: [GENERAL] Data Replication

2008-12-12 Thread Markus Wanner
Hello Joshua, Joshua D. Drake wrote: > I think the point is that right now Postgres-R (just like Replicator) > keeps its own tree that incorporates the PostgreSQL code. ..as does every other patch for Postgres before possibly it lands on mainline. But that's neither good nor bad per se, IMO. > W

Re: [GENERAL] Data Replication

2008-12-12 Thread Joshua D. Drake
On Fri, 2008-12-12 at 16:31 +0100, Markus Wanner wrote: > Hi, > > Let me simply point out and clarify, that I have absolutely no intent to > fork from Postgres. Quite the opposite, I'm interested in working > together with other Postgres hackers. I think the point is that right now Postgres-R (j

Re: [GENERAL] Data Replication

2008-12-12 Thread Rutherdale, Will
...@commandprompt.com Cc: Rutherdale, Will; pgsql-general@postgresql.org Subject: Re: [GENERAL] Data Replication Hi, Joshua D. Drake wrote: > On Wed, 2008-12-10 at 17:18 -0500, Rutherdale, Will wrote: >> I attempted some searches in various areas and came up with a >> bewildering array

Re: [GENERAL] Data Replication

2008-12-12 Thread Markus Wanner
Hi, Joshua D. Drake wrote: > On Wed, 2008-12-10 at 17:18 -0500, Rutherdale, Will wrote: >> I attempted some searches in various areas and came up with a >> bewildering array of results but no clear answer. Let's extend the list even further: h) If you are up for Java, you might like Sequoia.

Re: [GENERAL] Data Replication

2008-12-12 Thread Markus Wanner
Hi, Joshua D. Drake wrote: >> c) Postgres-R for multi-master data replication, appears to be a code >> fork of PostgreSQL > > Not stable as far as I know. Correct, it's not meant to be stable at this stage of development. I'm a bit disturbed by the tag "code fork", which has a rather negative

Re: [GENERAL] Data Replication

2008-12-11 Thread Joshua D. Drake
On Thu, 2008-12-11 at 19:33 +, Simon Riggs wrote: > On Thu, 2008-12-11 at 11:29 -0800, Joshua D. Drake wrote: > > > > As I said before, if you think something is missing, submit a software > > > or a doc patch and submit it to peer review. Until then, I think its > > > misleading to claim that

Re: [GENERAL] Data Replication

2008-12-11 Thread Simon Riggs
On Thu, 2008-12-11 at 11:29 -0800, Joshua D. Drake wrote: > > As I said before, if you think something is missing, submit a software > > or a doc patch and submit it to peer review. Until then, I think its > > misleading to claim that only your magic spice makes replication work > > correctly and

Re: [GENERAL] Data Replication

2008-12-11 Thread Joshua D. Drake
On Thu, 2008-12-11 at 19:24 +, Simon Riggs wrote: > > > True, we rely on the existence of rsync, scp etc.. and go to great pains > > > to provide as much choice as possible. > > > > > > If you think other things are required you are welcome to contribute > > > them so they can be verified fau

Re: [GENERAL] Data Replication

2008-12-11 Thread Simon Riggs
On Thu, 2008-12-11 at 09:52 -0800, Joshua D. Drake wrote: > On Thu, 2008-12-11 at 17:37 +, Simon Riggs wrote: > > On Thu, 2008-12-11 at 09:14 -0800, Joshua D. Drake wrote: > > > > > I think this statement is misleading. The only thing core contains is > > > the ability to use a bunch of util

Re: [GENERAL] Data Replication

2008-12-11 Thread Joshua D. Drake
On Thu, 2008-12-11 at 17:37 +, Simon Riggs wrote: > On Thu, 2008-12-11 at 09:14 -0800, Joshua D. Drake wrote: > > > I think this statement is misleading. The only thing core contains is > > the ability to use a bunch of utilities (with the exception of > > pg_standby) that aren't in core to p

Re: [GENERAL] Data Replication

2008-12-11 Thread Simon Riggs
On Thu, 2008-12-11 at 09:14 -0800, Joshua D. Drake wrote: > On Thu, 2008-12-11 at 17:09 +, Simon Riggs wrote: > > On Wed, 2008-12-10 at 18:34 -0500, Rutherdale, Will wrote: > > > Thanks very much, Steve. > > > Yes, everything you need for log shipping has been contributed to the > > main proj

Re: [GENERAL] Data Replication

2008-12-11 Thread Joshua D. Drake
On Thu, 2008-12-11 at 17:09 +, Simon Riggs wrote: > On Wed, 2008-12-10 at 18:34 -0500, Rutherdale, Will wrote: > > Thanks very much, Steve. > Yes, everything you need for log shipping has been contributed to the > main project. If you read things elsewhere, please refer closely to the > docs w

Re: [GENERAL] Data Replication

2008-12-11 Thread Simon Riggs
On Wed, 2008-12-10 at 18:34 -0500, Rutherdale, Will wrote: > Thanks very much, Steve. > > The main (but not only) type of data replication activity I'm interested > in right now would be the warm standby. Thus it appears from the > documents you showed me that log shipping is one solution curren

Re: [GENERAL] Data Replication

2008-12-11 Thread Andrew Sullivan
On Wed, Dec 10, 2008 at 08:41:30PM -0700, Scott Marlowe wrote: > one of the real time replication. Failover in slony is pretty easy to > do and happens in seconds. But you do have to resubscribe the master > as a slave and copy everything over again after a failover to make the > old master the

Re: [GENERAL] Data Replication

2008-12-11 Thread Tim Uckun
> No probably not. I mean they are all pretty easy (especially log > shipping) but it is definitely true they are slow, depending on the size > of the database. > As an alternative is there a clustering or multi master replication scheme that would be useful in a WAN? Preferably with a "prefered

Re: [GENERAL] Data Replication

2008-12-10 Thread Joshua D. Drake
On Wed, 2008-12-10 at 21:39 -0700, Scott Marlowe wrote: > On Wed, Dec 10, 2008 at 8:43 PM, Joshua D. Drake <[EMAIL PROTECTED]> wrote: > > On Wed, 2008-12-10 at 20:41 -0700, Scott Marlowe wrote: > >> On Wed, Dec 10, 2008 at 7:40 PM, Tim Uckun <[EMAIL PROTECTED]> wrote: > > > >> Log shipping doesn't

Re: [GENERAL] Data Replication

2008-12-10 Thread Scott Marlowe
On Wed, Dec 10, 2008 at 8:43 PM, Joshua D. Drake <[EMAIL PROTECTED]> wrote: > On Wed, 2008-12-10 at 20:41 -0700, Scott Marlowe wrote: >> On Wed, Dec 10, 2008 at 7:40 PM, Tim Uckun <[EMAIL PROTECTED]> wrote: > >> Log shipping doesn't really lends itself to switching back and forth >> between masters

Re: [GENERAL] Data Replication

2008-12-10 Thread Joshua D. Drake
On Wed, 2008-12-10 at 20:41 -0700, Scott Marlowe wrote: > On Wed, Dec 10, 2008 at 7:40 PM, Tim Uckun <[EMAIL PROTECTED]> wrote: > Log shipping doesn't really lends itself to switching back and forth > between masters and slaves. Really? It seems to me that you can make a base backup just as fast

Re: [GENERAL] Data Replication

2008-12-10 Thread Scott Marlowe
On Wed, Dec 10, 2008 at 7:40 PM, Tim Uckun <[EMAIL PROTECTED]> wrote: >> >> You have to run a new base backup and have the slave ship logs to the >> master. > > Mmmm. Does this backup have to be a full backup? What if your database > is very large? Yes. Your backup is very large. > I am hoping t

Re: [GENERAL] Data Replication

2008-12-10 Thread Tim Uckun
> > You have to run a new base backup and have the slave ship logs to the > master. Mmmm. Does this backup have to be a full backup? What if your database is very large? I am hoping to get a setup which is similar to SQL server mirroring. It uses a witness server to keep track of who got what "lo

Re: [GENERAL] Data Replication

2008-12-10 Thread Joshua D. Drake
On Thu, 2008-12-11 at 15:21 +1300, Tim Uckun wrote: > What happens when I bring the primary back on line. I now want this to > be primary again and catch up on all the transactions that were sent > to the secondary. I want the secondary to resume it's backup status. > You have to run a new base

Re: [GENERAL] Data Replication

2008-12-10 Thread Tim Uckun
On Thu, Dec 11, 2008 at 1:05 PM, David Wall <[EMAIL PROTECTED]> wrote: > We've done warm standby as you indicate, and we've not needed anything > special. Thanks for sharing your configuation. I have one additional question thought... How do you handle the reverting? For example. Say I have a p

Re: [GENERAL] Data Replication

2008-12-10 Thread David Wall
We've done warm standby as you indicate, and we've not needed anything special. On the primary's postgresql.conf we use: archive_command = '~/postgresql/bin/copyWAL "%p" "%f"' Our copyWAL script is just a wrapper for 'scp' since we want to copy the data encrypted over the network: #!/bin/ba

Re: [GENERAL] Data Replication

2008-12-10 Thread Joshua D. Drake
On Wed, 2008-12-10 at 18:45 -0500, Rutherdale, Will wrote: > Thanks, Joshua. > > As I mentioned to Steve, warm standby / log shipping seems to be the > main feature I'm looking for. > > The PITR solution you mention: is that an improvement over regular log > shipping? Or do I misunderstand wher

Re: [GENERAL] Data Replication

2008-12-10 Thread Rutherdale, Will
From: Joshua D. Drake [mailto:[EMAIL PROTECTED] Sent: 10 December 2008 17:52 To: Rutherdale, Will Cc: pgsql-general@postgresql.org Subject: Re: [GENERAL] Data Replication On Wed, 2008-12-10 at 17:18 -0500, Rutherdale, Will wrote: > Hi. > > I am trying to determine what kind of data

Re: [GENERAL] Data Replication

2008-12-10 Thread Joshua D. Drake
On Wed, 2008-12-10 at 18:34 -0500, Rutherdale, Will wrote: > Thanks very much, Steve. > > The main (but not only) type of data replication activity I'm interested > in right now would be the warm standby. Thus it appears from the > documents you showed me that log shipping is one solution current

Re: [GENERAL] Data Replication

2008-12-10 Thread Rutherdale, Will
BTW Hope you don't mind my quoting style. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steve Atkins Sent: 10 December 2008 17:50 To: PostgreSQL General Subject: Re: [GENERAL] Data Replication On Dec 10, 2008, at 2:18 PM, Rutherdale, Will wr

Re: [GENERAL] Data Replication

2008-12-10 Thread Joshua D. Drake
On Wed, 2008-12-10 at 17:18 -0500, Rutherdale, Will wrote: > Hi. > > I am trying to determine what kind of data replication is currently > available in PostgreSQL. This is for purposes of examining capabilities > of PostgreSQL as compared to other RDBMSs. > > I attempted some searches in various

Re: [GENERAL] Data Replication

2008-12-10 Thread Steve Atkins
On Dec 10, 2008, at 2:18 PM, Rutherdale, Will wrote: Hi. I am trying to determine what kind of data replication is currently available in PostgreSQL. This is for purposes of examining capabilities of PostgreSQL as compared to other RDBMSs. I attempted some searches in various areas and ca

[GENERAL] Data Replication

2008-12-10 Thread Rutherdale, Will
Hi. I am trying to determine what kind of data replication is currently available in PostgreSQL. This is for purposes of examining capabilities of PostgreSQL as compared to other RDBMSs. I attempted some searches in various areas and came up with a bewildering array of results but no clear answe

Re: [GENERAL] Data replication through disk replication

2007-05-20 Thread Hannes Dorbath
Ben wrote: > On May 20, 2007, at 3:01 AM, Thomas Lopatic wrote: > The problem comes when the primary is cannot replicate to the secondary > but can, for whatever reason, still talk to clients. If a client is told > something is committed but that commit isn't replicated, you have a > problem. Righ

Re: [GENERAL] Data replication through disk replication

2007-05-20 Thread Hannes Dorbath
Thomas Lopatic wrote: >> So what happens in those cases where the primary node gets in trouble >> but isn't actually dead yet? > > Hmmm. Is this really a problem? Couldn't the secondary DRBD node simply > stop accepting replicated data from the primary node before firing up > postmaster? Then the

Re: [GENERAL] Data replication through disk replication

2007-05-20 Thread Ben
On May 20, 2007, at 3:01 AM, Thomas Lopatic wrote: So what happens in those cases where the primary node gets in trouble but isn't actually dead yet? Hmmm. Is this really a problem? The problem comes when the primary is cannot replicate to the secondary but can, for whatever reason, still

Re: [GENERAL] Data replication through disk replication

2007-05-20 Thread Andrew Sullivan
On Sun, May 20, 2007 at 12:01:46PM +0200, Thomas Lopatic wrote: > Hmmm. Is this really a problem? Couldn't the secondary DRBD node simply > stop accepting replicated data from the primary node before firing up > postmaster? Then the postmaster on the primary DRBD node would only > write locally and

Re: [GENERAL] Data replication through disk replication

2007-05-20 Thread Thomas Lopatic
> So what happens in those cases where the primary node gets in trouble > but isn't actually dead yet? Hmmm. Is this really a problem? Couldn't the secondary DRBD node simply stop accepting replicated data from the primary node before firing up postmaster? Then the postmaster on the primary DRBD n

Re: [GENERAL] Data replication through disk replication

2007-05-19 Thread Joshua D. Drake
Alvaro Herrera wrote: Ben wrote: If you're just looking for a way to have high availability and you're ok being tied to linux, DRBD is a good way to go. It keeps things simple in that all changes are replicated, it won't say an fsync is finished until it's finished on the remote host too, Oh

Re: [GENERAL] Data replication through disk replication

2007-05-19 Thread Ben
Er, yes, sorry, I didn't mean to imply that you should run without some kind of STONITH solution, to catch the case when the link DRDB uses goes down but the other network links are still working fine. It's in the common case, when everything is working, that DRBD won't accidentally let you

Re: [GENERAL] Data replication through disk replication

2007-05-19 Thread Joris Dobbelsteen
>-Original Message- >From: [EMAIL PROTECTED] >[mailto:[EMAIL PROTECTED] On Behalf Of >Andrew Sullivan >Sent: zaterdag 19 mei 2007 15:28 >To: pgsql-general@postgresql.org >Subject: Re: [GENERAL] Data replication through disk replication > >On Fri, May 18, 20

Re: [GENERAL] Data replication through disk replication

2007-05-19 Thread Andrew Sullivan
On Fri, May 18, 2007 at 05:03:30PM -0700, Ben wrote: > that all changes are replicated, it won't say an fsync is finished until > it's finished on the remote host too, and it won't let you mount the block > device on the slave system (at least with 0.7x). How can it guarantee these things? Th

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Ben
You pay a price writes, but with write caching enabled on your (battery-backed, of course) RAID card and using gigabit, it's easy to get >100MB/s throughput. It's also easy to replicate different block devices over separate network links, if that becomes your bottleneck. On May 18, 2007, a

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Alvaro Herrera
Ben wrote: > If you're just looking for a way to have high availability and you're ok > being tied to linux, DRBD is a good way to go. It keeps things simple in > that all changes are replicated, it won't say an fsync is finished until > it's finished on the remote host too, Oh, so that's how i

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Ben
If you're just looking for a way to have high availability and you're ok being tied to linux, DRBD is a good way to go. It keeps things simple in that all changes are replicated, it won't say an fsync is finished until it's finished on the remote host too, and it won't let you mount the block d

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Andrew Sullivan
On Fri, May 18, 2007 at 07:55:24PM +0200, Thomas Lopatic wrote: > For Slony-I it seems to me that my risk is losing a couple of rows in my > database, which is something that I could live with. For disk-level > replication it seems to me that, in case of a master failure, I could > easily end up w

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Thomas Lopatic
[Disk-level replication instead of using Slony-I] > What are the reasons they recommend this? (See my blathering in > another thread about how often the hand-wavy recommendations that are > made on this topic can really bite you hard if you don't know all the > intimate details underneath.) The

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Andrew Sullivan
On Fri, May 18, 2007 at 02:48:03PM +0200, Thomas Lopatic wrote: > I am currently looking into replicated two-node master/slave PostgreSQL > environments. Lately I've heard more and more people recommend > replicating data from the master to the slave at the disk device level > as opposed to the DB

Re: [GENERAL] Data replication through disk replication

2007-05-18 Thread Martijn van Oosterhout
On Fri, May 18, 2007 at 02:48:03PM +0200, Thomas Lopatic wrote: > What I keep wondering: Isn't there substantial risk involved? > I mean, suppose the master fails in the middle of a write. Isn't there > the possibility that this corrupts the database? How robust is > PostgreSQL's on-disk file forma

[GENERAL] Data replication through disk replication

2007-05-18 Thread Thomas Lopatic
Hi there, I am currently looking into replicated two-node master/slave PostgreSQL environments. Lately I've heard more and more people recommend replicating data from the master to the slave at the disk device level as opposed to the DBMS level (Slony-I). On Linux, usually DRBD is recommended for