Hi Attila 2011/11/28 Attila Megyeri <amegy...@minerva-soft.com>: > Hi Takatoshi, > > I understand your point and I agree that the correct behavior is not to start > replication when data consistency exists. > The only thing I do not really understand is how it could have happened: > > 1) nodes were in sync (psql1=PRI, psql2=STREAMING|SYNC) > 2) I shut down node psql1 (by placing it into standby) > 3) At this moment psql1's baseline became higher by 20? What could cause > this? Probably the demote operation itself? There were no clients connected - > and there was definitively no write operation to the db (except if not from > system side).
Yes, PostgreSQL executes a CHECKPOINT when it is shut down normally on demote. > On the other hand - thank you very much for your contribution, the RA works > very well and I really appreciate your work and help! Not at all. Don't mention it. Regards, Takatoshi MATSUO > Bests, > > Attil > > -----Original Message----- > From: Takatoshi MATSUO [mailto:matsuo....@gmail.com] > Sent: 2011. november 28. 2:10 > To: The Pacemaker cluster resource manager > Subject: Re: [Pacemaker] Postgresql streaming replication failover - RA needed > > Hi Attila > > Primary can not send all wals to HotStandby whether primary is shut down > normally. > These logs validate it. > >> Nov 27 16:03:27 psql1 pgsql[12204]: INFO: My Timeline ID and >> Checkpoint : 14:0000000023000020 Nov 27 16:03:27 psql1 pgsql[12204]: >> INFO: psql2 master baseline : 14:0000000023000000 > > psql1's location was "0000000023000020" when it was demoted. > OTOH psql2's location was "0000000023000000" when it was promoted. > > It means that psql1's data was newer than psql2's one at that time. > The gap is 20. > > As you said you can start psql1's PostgreSQL manually, but PostgreSQL can't > realize this occurrence. > If you start HotStandby at psql1, data is replicated after 0000000023000020. > It's inconsistency. > > Thanks, > Takatoshi MATSUO > > > 2011/11/28 Attila Megyeri <amegy...@minerva-soft.com>: >> Hi Takatoshi, >> >> I don't think it is inconsistency problem - for me it looks like some RA bug. >> I think so, because postgres starts properly outside pacemaker. >> >> When pacemaker starts node psql1 I see only: >> >> postgresql:0_start_0 (node=psql1, call=9, rc=1, status=complete): >> unknown error >> >> and the postgres log is empty - so I suppose that it does not even try to >> start it. >> >> What I tested was: >> - I had a stable cluster, where psql1 was the master, psql2 was the >> slave >> - I put psql1 into standby mode. ("node psql1 standby") to test >> failover >> - After a while psql2 became the PRI, which is very good >> - When I put psql1 back online, postgres wouldn't start anymore from >> pacemaker (unknown error). >> >> >> I tried to start postgres manually from the shell it worked fine, even the >> monitor was able to see that it became in SYNC (obviously the master/slave >> group was showing improper state as psql was started outside pacemaker. >> >> I don't think data inconsistency is the case, partially because there are no >> clients connected, partially because psql starts properly outside pacemaker. >> >> Here is what is relevant from the log: >> >> Nov 27 16:02:50 psql1 pgsql[11021]: DEBUG: PostgreSQL is running as a >> primary. >> Nov 27 16:02:51 psql1 pgsql[11021]: DEBUG: node=psql2, >> state=STREAMING, sync_state=SYNC Nov 27 16:02:53 psql1 pgsql[11142]: DEBUG: >> PostgreSQL is running as a primary. >> Nov 27 16:02:53 psql1 pgsql[11142]: DEBUG: node=psql2, >> state=STREAMING, sync_state=SYNC Nov 27 16:02:55 psql1 pgsql[11272]: DEBUG: >> PostgreSQL is running as a primary. >> Nov 27 16:02:55 psql1 pgsql[11272]: DEBUG: node=psql2, >> state=STREAMING, sync_state=SYNC Nov 27 16:02:57 psql1 pgsql[11368]: DEBUG: >> PostgreSQL is running as a primary. >> Nov 27 16:02:57 psql1 pgsql[11368]: DEBUG: node=psql2, >> state=STREAMING, sync_state=SYNC Nov 27 16:03:00 psql1 pgsql[11463]: DEBUG: >> PostgreSQL is running as a primary. >> Nov 27 16:03:00 psql1 pgsql[11463]: DEBUG: node=psql2, >> state=STREAMING, sync_state=SYNC Nov 27 16:03:00 psql1 pgsql[11556]: >> DEBUG: notify: pre for demote Nov 27 16:03:00 psql1 pgsql[11590]: INFO: >> Stopping PostgreSQL on demote. >> Nov 27 16:03:02 psql1 pgsql[11590]: INFO: waiting for server to shut >> down..... done server stopped Nov 27 16:03:02 psql1 pgsql[11590]: INFO: >> Removing /var/lib/pgsql/PGSQL.lock. >> Nov 27 16:03:02 psql1 pgsql[11590]: INFO: PostgreSQL is down Nov 27 >> 16:03:02 psql1 pgsql[11590]: INFO: Changing pgsql-status on psql1 : >> PRI->STOP. >> Nov 27 16:03:02 psql1 pgsql[11590]: DEBUG: Created recovery.conf. >> host=10.12.1.28, user=postgres Nov 27 16:03:02 psql1 pgsql[11590]: INFO: >> Setup all nodes as an async. >> Nov 27 16:03:02 psql1 pgsql[11732]: DEBUG: notify: post for demote Nov >> 27 16:03:02 psql1 pgsql[11732]: DEBUG: post-demote called. Demote >> uname is psql1 Nov 27 16:03:02 psql1 pgsql[11732]: INFO: My Timeline >> ID and Checkpoint : 14:0000000023000020 Nov 27 16:03:02 psql1 pgsql[11732]: >> WARNING: Can't get psql2 master baseline. Waiting... >> Nov 27 16:03:03 psql1 pgsql[11732]: INFO: psql2 master baseline : >> 14:0000000023000000 Nov 27 16:03:03 psql1 pgsql[11732]: ERROR: My data is >> inconsistent. >> Nov 27 16:03:03 psql1 pgsql[11867]: DEBUG: notify: pre for stop Nov 27 >> 16:03:03 psql1 pgsql[11969]: INFO: PostgreSQL is already stopped. >> Nov 27 16:03:12 psql1 pgsql[12053]: INFO: Don't check >> /var/lib/postgresql/9.1/main during probe Nov 27 16:03:12 psql1 >> pgsql[12053]: INFO: PostgreSQL is down Nov 27 16:03:27 psql1 pgsql[12204]: >> INFO: Changing pgsql-status on psql1 : ->STOP. >> Nov 27 16:03:27 psql1 pgsql[12204]: DEBUG: Created recovery.conf. >> host=10.12.1.28, user=postgres Nov 27 16:03:27 psql1 pgsql[12204]: INFO: >> Setup all nodes as an async. >> Nov 27 16:03:27 psql1 pgsql[12204]: INFO: My Timeline ID and >> Checkpoint : 14:0000000023000020 Nov 27 16:03:27 psql1 pgsql[12204]: >> INFO: psql2 master baseline : 14:0000000023000000 Nov 27 16:03:27 psql1 >> pgsql[12204]: ERROR: My data is inconsistent. >> Nov 27 16:03:27 psql1 pgsql[12339]: DEBUG: notify: post for start Nov >> 27 16:03:27 psql1 pgsql[12373]: DEBUG: notify: pre for stop Nov 27 >> 16:03:27 psql1 pgsql[12407]: INFO: PostgreSQL is already stopped. >> >> >> Thanks, >> >> Attila >> >> >> -----Original Message----- >> From: Takatoshi MATSUO [mailto:matsuo....@gmail.com] >> Sent: 2011. november 27. 11:07 >> To: The Pacemaker cluster resource manager >> Subject: Re: [Pacemaker] Postgresql streaming replication failover - >> RA needed >> >> Hi Attila >> >> 2011/11/27 Attila Megyeri <amegy...@minerva-soft.com>: >>> Hi Takatoshi, >>> >>> You were right, changing the shell to bash resolved the problem. >>> The cluster now started in sync mode - thank you very much. >> >> You're very welcome. >> >>> I will be testing it in the next couple of days. I did just a very >>> quick test - it seems that psql master failed over to psql2 properly, >>> but when I tried to move it back to psql1 there was some problems starting >>> psql on node 1. >> >> If master(psql1) is failed, its data may be inconsistency. >> A PostgreSQL developer says that it's a feature. >> Therefore my RA prevent it from starting automatically if data is >> inconsistency. >> Please backup psql2' data and restore it to psql1, and remove >> /var/lib/pgsql/PGSQL.lock file before clearing failcount. >> >> I use rsync to backup and restore in the following way. >> ----- >> # psql -h 192.168.2.114 -U postgres -c "SELECT pg_start_backup('label')" >> # rsync -avr --delete --exclude=postmaster.pid >> 192.168.2.114:/var/lib/pgsql/9.1/data/ /var/lib/pgsql/9.1/data/ # psql >> -h 192.168.2.114 -U postgres -c "SELECT pg_stop_backup()" >> ----- >> >> >> BTW I fixed some bugs 2 days ago. >> Please use the newest version. >> >> Thanks, >> Takatoshi MATSUO >> >> >>> >>> Does it work fine for you in both directions? >>> >>> Thank you very much. >>> >>> Have a nice weekend, >>> >>> Attila >>> >>> >>> >>> -----Original Message----- >>> From: Takatoshi MATSUO [mailto:matsuo....@gmail.com] >>> Sent: 2011. november 27. 6:12 >>> To: The Pacemaker cluster resource manager >>> Subject: Re: [Pacemaker] Postgresql streaming replication failover - >>> RA needed >>> >>> Hi Attila >>> >>> 2011/11/27 Attila Megyeri <amegy...@minerva-soft.com>: >>>> Hi Takatoshi, >>>> >>>> Thank you for coming back to me so quickly. >>>> >>>> In the /var/lib/pgsql there are the following files: >>>> >>>> PSQL1: >>>> ===== >>>> root@psql1:/var/lib/pgsql# ls -la >>>> total 16 >>>> drwxr-xr-x 2 postgres postgres 4096 Nov 26 18:04 . >>>> drwxr-xr-x 35 root root 4096 Nov 25 22:21 .. >>>> -rw-r--r-- 1 postgres postgres 1 Nov 26 00:17 rep_mode.conf >>>> -rw-r--r-- 1 root root 49 Nov 26 18:04 xlog_note.0 >>>> >>>> root@psql1:/var/lib/pgsql# cat xlog_note.0 -e psql1 0000000019000000 >>>> psql2 0000000019000000 >>>> root@psql1:/var/lib/pgsql# >>>> >>>> PSQL2: >>>> ======= >>>> root@psql2:/var/lib/pgsql# ls -la >>>> total 16 >>>> drwxr-xr-x 2 postgres postgres 4096 Nov 26 18:05 . >>>> drwxr-xr-x 33 root root 4096 Nov 26 00:10 .. >>>> -rw-r--r-- 1 postgres postgres 1 Nov 26 00:24 rep_mode.conf >>>> -rw-r--r-- 1 root root 49 Nov 26 18:05 xlog_note.0 >>>> root@psql2:/var/lib/pgsql# cat xlog_note.0 -e psql1 0000000019000000 >>>> psql2 0000000019000000 >>>> root@psql2:/var/lib/pgsql# >>> >>> It seems that dash's bultin echo command is used because echo with "-e" >>> option dose not function. >>> >>> Perhaps my RA also depends on bash. >>> Can you use a bash instead of a dash? >>> >>>> BTW, postgres is installed under /var/lib/postgresql , but I noticed that >>>> some parts of the RA are referring to the /var/lib/pgsql directory, so I >>>> created that directory and i keep some of the files there. >>> >>> It's no ploblem. >>> If you want to change this path, please specify it using "tmpdir" parameter. >>> >>> Regards, >>> Takatoshi MATSUO >>> >>>> >>>> Thanks, >>>> Attila >>>> >>>> >>>> >>>> -----Original Message----- >>>> From: Takatoshi MATSUO [mailto:matsuo....@gmail.com] >>>> Sent: 2011. november 26. 18:27 >>>> To: The Pacemaker cluster resource manager >>>> Subject: Re: [Pacemaker] Postgresql streaming replication failover - >>>> RA needed >>>> >>>> Hi Attila >>>> >>>> 1. Are there /var/lib/pgsql/xlog_note.0 , xlog_note.1, xlog_note.2 .... >>>> files? >>>> These files are created while checking a xlog location on monitor. >>>> >>>> 2. Do these files include lines as below? >>>> --------- >>>> pgsql1 0000000019000000 >>>> pgsql2 0000000019000000 >>>> --------- >>>> >>>> Regards. >>>> Takatoshi MATSUO >>>> >>>> >>>> 2011年11月26日22:44 Attila Megyeri <amegy...@minerva-soft.com>: >>>>> Hi Yoshiharu, Takatoshi, >>>>> >>>>> Spent another day, without success. :( >>>>> >>>>> I started from scratch and synchronous replications works nicely when >>>>> nodes are started outside pacemaker. >>>>> My PostgreSQL version is 9.1.1. >>>>> >>>>> When I start from pacemaker, after a while it gets into the following >>>>> state: >>>>> >>>>> Online: [ psql1 psql2 ] >>>>> >>>>> Master/Slave Set: msPostgresql [postgresql] >>>>> Slaves: [ psql1 psql2 ] >>>>> Clone Set: clnPingCheck [pingCheck] >>>>> Started: [ psql1 psql2 ] >>>>> >>>>> Node Attributes: >>>>> * Node psql1: >>>>> + default_ping_set : 100 >>>>> + master-postgresql:0 : -INFINITY >>>>> + pgsql-status : HS:alone >>>>> + pgsql-xlog-loc : 0000000019000000 >>>>> * Node psql2: >>>>> + default_ping_set : 100 >>>>> + master-postgresql:1 : -INFINITY >>>>> + pgsql-status : HS:alone >>>>> + pgsql-xlog-loc : 0000000019000000 >>>>> >>>>> >>>>> The psql status queries return the following: >>>>> >>>>> PSQL1 >>>>> ====== >>>>> postgres@psql1:/root$ psql -c "select >>>>> application_name,upper(state),upper(sync_state) from pg_stat_replication" >>>>> application_name | upper | upper >>>>> ------------------+-------+------- >>>>> (0 rows) >>>>> >>>>> postgres@psql1:/root$ psql -Atc "select >>>>> pg_last_xlog_replay_location(),pg_last_xlog_receive_location()" >>>>> 0/19000020|0/19000000 >>>>> >>>>> PSQL2 >>>>> ====== >>>>> postgres@psql2:~$ psql -c "select >>>>> application_name,upper(state),upper(sync_state) from pg_stat_replication" >>>>> application_name | upper | upper >>>>> ------------------+-------+------- >>>>> (0 rows) >>>>> >>>>> postgres@psql2:~$ psql -Atc "select >>>>> pg_last_xlog_replay_location(),pg_last_xlog_receive_location()" >>>>> 0/19000000|0/19000000 >>>>> >>>>> >>>>> Neither server can connect (obviously) to the master, as the vip_repl Is >>>>> not brought up. >>>>> >>>>> >>>>> Could you help me understand WHAT is the action/state/event that sould >>>>> promote one of the nodes? I see that pacemaker monitors the servers every >>>>> X seconds, but nothing else happens. >>>>> >>>>> In the log (limited to pgsql) the following sequence is repeated >>>>> forewer >>>>> >>>>> Nov 26 13:36:19 psql1 pgsql[19829]: INFO: Master is not exist. >>>>> Nov 26 13:36:19 psql1 pgsql[19829]: DEBUG: Checking right of master. >>>>> Nov 26 13:36:19 psql1 pgsql[19829]: INFO: My data status=. >>>>> Nov 26 13:36:19 psql1 pgsql[19829]: INFO: psql1 xlog location : >>>>> 0000000019000000 Nov 26 13:36:19 psql1 pgsql[19829]: INFO: psql2 >>>>> xlog location : 0000000019000000 Nov 26 13:36:26 psql1 pgsql[19993]: >>>>> DEBUG: PostgreSQL is running as a hot standby. >>>>> Nov 26 13:36:26 psql1 pgsql[19993]: INFO: Master is not exist. >>>>> Nov 26 13:36:26 psql1 pgsql[19993]: DEBUG: Checking right of master. >>>>> Nov 26 13:36:26 psql1 pgsql[19993]: INFO: My data status=. >>>>> Nov 26 13:36:26 psql1 pgsql[19993]: INFO: psql1 xlog location : >>>>> 0000000019000000 Nov 26 13:36:26 psql1 pgsql[19993]: INFO: psql2 >>>>> xlog location : 0000000019000000 Nov 26 13:36:33 psql1 pgsql[20176]: >>>>> DEBUG: PostgreSQL is running as a hot standby. >>>>> Nov 26 13:36:33 psql1 pgsql[20176]: INFO: Master is not exist. >>>>> Nov 26 13:36:33 psql1 pgsql[20176]: DEBUG: Checking right of master. >>>>> Nov 26 13:36:33 psql1 pgsql[20176]: INFO: My data status=. >>>>> Nov 26 13:36:33 psql1 pgsql[20176]: INFO: psql1 xlog location : >>>>> 0000000019000000 Nov 26 13:36:33 psql1 pgsql[20176]: INFO: psql2 >>>>> xlog location : 0000000019000000 Nov 26 13:36:41 psql1 pgsql[20343]: >>>>> DEBUG: PostgreSQL is running as a hot standby. >>>>> >>>>> >>>>> Any help is appreciated! >>>>> >>>>> Regards, >>>>> Attila >>>>> >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Yoshiharu Mori [mailto:y-m...@sraoss.co.jp] >>>>> Sent: 2011. november 25. 14:17 >>>>> To: The Pacemaker cluster resource manager >>>>> Cc: Attila Megyeri >>>>> Subject: Re: [Pacemaker] Postgresql streaming replication failover >>>>> - RA needed >>>>> >>>>> Hi Attila >>>>> >>>>>> A quick snippet from the corosync.log >>>>>> >>>>>> Nov 23 05:43:05 psql1 pgsql[2845]: DEBUG: Checking right of master. >>>>>> Nov 23 05:43:05 psql1 pgsql[2845]: INFO: My data status=. >>>>>> Nov 23 05:43:05 psql1 pgsql[2845]: INFO: psql1 xlog location : >>>>>> 000000000D000000 Nov 23 05:43:05 psql1 pgsql[2845]: INFO: psql2 >>>>>> xlog location : 0000000008000000 >>>>>> >>>>>> As you see, the "my data status" returns an empty string. >>>>> >>>>> My log is same. but it works. >>>>> >>>>> Nov 18 19:28:26 osspc24-1 pgsql[17350]: INFO: Master is not exist. >>>>> Nov 18 19:28:26 osspc24-1 pgsql[17350]: INFO: Checking right of master. >>>>> Nov 18 19:28:19 osspc24-1 pgsql[17138]: INFO: My data status=. >>>>> Nov 18 19:28:19 osspc24-1 pgsql[17138]: INFO: pm01 xlog location : >>>>> 0000000005000020 Nov 18 19:28:19 osspc24-1 pgsql[17138]: INFO: pm02 >>>>> xlog location : 0000000005000000 >>>>> >>>>> In my log, the following logs are outputted and started after checking >>>>> xlog location(3 times). >>>>> >>>>> Nov 18 19:29:39 osspc24-1 pgsql[18720]: INFO: I have a master right. >>>>> >>>>> Please show us more corosync.log. >>>>> >>>>> >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Attila Megyeri [mailto:amegy...@minerva-soft.com] >>>>>> Sent: 2011. november 25. 9:28 >>>>>> To: The Pacemaker cluster resource manager >>>>>> Subject: Re: [Pacemaker] Postgresql streaming replication failover >>>>>> - RA needed >>>>>> >>>>>> Hi Takatoshi, >>>>>> >>>>>> I have restored the PSQL to run without corosync so I cannot send you >>>>>> the crm_mon output now. >>>>>> >>>>>> What I can tell for sure: >>>>>> - RA never promoted any of the nodes, no matter what the status was. It >>>>>> also did not promote the node, when it was the only one. >>>>>> - I believe the issue is in the comparison of the xlogs. How could I >>>>>> troubleshoot that? I see from the logs that crm NEVER tried to invoke >>>>>> pgsql with "promote" >>>>>> - I tried previously the crm_mon -A option, but there was never a " >>>>>> pgsql-data-status" attribute. The other attribs were there, >>>>>> including the HS:alone >>>>>> - In the corosync log the only relevant RA message I see is " Master is >>>>>> not exist. " I never saw a message like "My data is out-of-date" >>>>>> >>>>>> Thank you! >>>>>> >>>>>> Attila >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Takatoshi MATSUO [mailto:matsuo....@gmail.com] >>>>>> Sent: 2011. november 25. 8:56 >>>>>> To: The Pacemaker cluster resource manager >>>>>> Subject: Re: [Pacemaker] Postgresql streaming replication failover >>>>>> - RA needed >>>>>> >>>>>> Hi Attila >>>>>> >>>>>> 2011/11/24 Attila Megyeri <amegy...@minerva-soft.com>: >>>>>> > Hi Takatoshi, All, >>>>>> > >>>>>> > Thanks for your reply. >>>>>> > I see that you have invested significant effort in the development of >>>>>> > the RA. I spent the last day trying to set up the RA, but without much >>>>>> > success. >>>>>> > >>>>>> > My infrastructure is very similar to yours, except for the fact that >>>>>> > currently I am testing with a single network adapter. >>>>>> > >>>>>> > Replication works nicely when I start the databases manually, not >>>>>> > using corosync. >>>>>> > >>>>>> > When I try to start using corosync,I see that the ping resources start >>>>>> > normally, but the msPostgresql starts on both nodes in slave mode, and >>>>>> > I see "HS:alone" >>>>>> >>>>>> To see "HS:alone" is normal. >>>>>> And RA compares xlog locations and promote the postgresql having new >>>>>> data. >>>>>> >>>>>> > In the Wiki you state, the if I start on a signle node only, PSQL >>>>>> > should start in Master mode (PRI), but this is not the case. >>>>>> >>>>>> If the data is old, the node can't be master. >>>>>> To be master needs pgsql-data-status="LATEST" or "STREAMING|SYNC". >>>>>> Plese check it using "crm_mon -A". >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> And to become a master from stopped takes a few minutes because the RA >>>>>> compares xlog location on monitor. >>>>>> >>>>>> >>>>>> > The recovery.conf file is created immediately, and from the logs I see >>>>>> > no attempt at all to promote the node. >>>>>> > In the postgres logs I see that node1, which is supposed to be a >>>>>> > master, tries to connect to the vip-rep IP address, which is NOT >>>>>> > brought up, because it depends on the Master role... >>>>>> > >>>>>> > Do you have any idea? >>>>>> >>>>>> Please check HA log. >>>>>> My RA outputs "My data is out-of-date. status=********" to log if the >>>>>> data is old. >>>>>> >>>>>> Regards, >>>>>> Takatoshi MATSUO >>>>>> >>>>>> _______________________________________________ >>>>>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>>>>> >>>>>> Project Home: http://www.clusterlabs.org Getting started: >>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>>> Bugs: http://bugs.clusterlabs.org >>>>>> >>>>>> _______________________________________________ >>>>>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>>>>> >>>>>> Project Home: http://www.clusterlabs.org Getting started: >>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>>> Bugs: http://bugs.clusterlabs.org >>>>>> >>>>>> _______________________________________________ >>>>>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>>>>> >>>>>> Project Home: http://www.clusterlabs.org Getting started: >>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>>> Bugs: http://bugs.clusterlabs.org >>>>> >>>>> >>>>> -- >>>>> Yoshiharu Mori <y-m...@sraoss.co.jp> SRA OSS, Inc Japan >>>>> http://www.sraoss.co.jp >>>>> TEL: 03-5979-2701 >>>>> FAX: 03-5979-2702 >>>>> >>>>> _______________________________________________ >>>>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>>>> >>>>> Project Home: http://www.clusterlabs.org Getting started: >>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>> Bugs: http://bugs.clusterlabs.org >>>>> >>>> >>>> _______________________________________________ >>>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>>> >>>> Project Home: http://www.clusterlabs.org Getting started: >>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>> Bugs: http://bugs.clusterlabs.org >>>> _______________________________________________ >>>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>>> >>>> Project Home: http://www.clusterlabs.org Getting started: >>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>> Bugs: http://bugs.clusterlabs.org >>>> >>> >>> _______________________________________________ >>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>> >>> Project Home: http://www.clusterlabs.org Getting started: >>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>> Bugs: http://bugs.clusterlabs.org >>> _______________________________________________ >>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>> >>> Project Home: http://www.clusterlabs.org Getting started: >>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>> Bugs: http://bugs.clusterlabs.org >>> >> >> _______________________________________________ >> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >> >> Project Home: http://www.clusterlabs.org Getting started: >> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org >> _______________________________________________ >> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >> >> Project Home: http://www.clusterlabs.org Getting started: >> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org >> > > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org Getting started: > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
