Long story short... the buffer is created by the Corosync IPC code. See: gboolean ais_dispatch(int sender, gpointer user_data) { int rc = CS_OK; char *buffer = NULL; gboolean good = TRUE; gboolean (*dispatch)(AIS_Message*,char*,int) = user_data;
rc = coroipcc_dispatch_get (ais_ipc_handle, (void**)&buffer, 0); // passing in a pointer to a NULL pointer, nothing allocated yet ... } Angus is the corosync IPC specialist, perhaps he has some insights as to your alignment issues. I know shared memory is involved somehow, perhaps that is complicating things. I also note that AIS_Message should in theory already be aligned, at least to my reading of it. struct crm_ais_msg_s { coroipc_response_header_t header __attribute__((aligned(8))); On Thu, Dec 16, 2010 at 10:44 PM, Greg Walton <pacema...@gwalton.net> wrote: > I've been slowly working at getting pacemaker(stable-1.0) with > corosync(flatiron) to run on arm little endian arch. > > I'm having alignment issues that break everything, crm status never reports > any nodes joining. I've simplified things down to starting corosync on a > single node, and expecting it to contact crmd and join the cluster via > ipc(which it's not doing). > > > I have isolated one alignment case I've been trying to fix: > > The AIS_Message struct pointed to by 'msg" in in ais_dispatch() (from > lib/common/ais.c) shows up with odd alignment sometimes which causes access > to msg->is_compressed to throw a bus error. > > What I've been doing is switching the kernel to signal on unaligned access > "echo 5 > /proc/cpu/alignment" which gets me a core file.. 99 times out of > 100 crmd is the first to core out, within 2 or 3 seconds at most. > > I should add that I can switch on kernel alignment fixups via "echo 3 > > /proc/cpu/alignment" and the single node will sign on with crmd... at least > initially, i haven't actually tested beyond "crm status" > > With some debugging lines in ais.c, I can see that crmd does NOT core when > the AIS_Message struct starts in an address that is a multiple of 4. I've > fiddled with various __attrib__ align statements in the AIS_Message struct > in include/crm/ais.h and regardless of what align attributes I apply to > struct members or the struct, the entire struct shows up starting at > different alignments, 4 first then some odd/non alignment and cores out. > > So I'm trying to track down where the memory for the AIS_Message > struct/buffer is getting allocated at. I've backed into corosync with > coroipcc_dispatch_get(), then socket_recv(), and finally to recvmsg(). I > only see 1 char at a time being read from the socket, and returned back to > ais_dispatch()... but I got getting fairly confused by all the dispatches > and handles being passed around and I'm not sure how the dispatch scheme is > supposed to work anyway. > > So, the big question I have is: In ais_dispatch(), the memory that "msg" > points to get's allocated by what/where... and/or is it in corosync or > pacemaker? > > > Here's an example of what I'm seeing in gdb( that source line #455 is > something I added for debugging, a few lines later the code normally uses > msg->is_compressed and you'd get the bus error there instead.. > > Core was generated by `/usr/lib/heartbeat/crmd'. > Program terminated with signal 7, Bus error. > #0 0x40105048 in ais_dispatch (sender=7, user_data=0xfebc) at ais.c:455 > 455 crm_debug("msg->is_compressed:%d", msg->is_compressed); > (gdb) print msg > $1 = (AIS_Message *) 0x40b6832b > (gdb) print &msg->is_compressed > $2 = (int *) 0x40b6834b > (gdb) print *msg > $3 = {header = {size = 778, id = 0, error = CS_OK}, id = 1, is_compressed = > 0, > host = {id = 0, pid = 0, local = 1, type = crm_msg_crmd, size = 0, > uname = '\000' <repeats 255 times>}, sender = {id = 855754762, > pid = 24720, local = 0, type = crm_msg_crmd, size = 5, > uname = "serva", '\000' <repeats 250 times>}, size = 178, > compressed_size = 0, > data = 0x40b6857f "<create_request_adv origin=\"post_cache_update\" > t=\"crmd\" version=\"3.0.1\" subt=\"request\" > reference=\"noop-crmd-1292462365-0\" crm_task=\"noop\" crm_sys_to=\"crmd\" > crm_sys_from=\"crmd\"/>"} > (gdb) bt > #0 0x40105048 in ais_dispatch (sender=7, user_data=0xfebc) at ais.c:455 > #1 0x401ced6c in G_fd_dispatch (source=0x5a4f0, callback=0, user_data=0x0) > at GSource.c:303 > #2 0x405308b4 in g_main_context_dispatch () from /lib/libglib-2.0.so.0 > #3 0x405346b4 in ?? () from /lib/libglib-2.0.so.0 > Cannot access memory at address 0x0 > #4 0x405346b4 in ?? () from /lib/libglib-2.0.so.0 > Cannot access memory at address 0x0 > Backtrace stopped: previous frame identical to this frame (corrupt stack?) > (gdb) > > > Note that msg=0x40b6832b is not 4 byte aligned by my math. In some cases msg > does point to 4 byte aligned memory. I added another debug line to print out > %p for msg each time ais_dispatch() is called. > > Here is part of a corosync.log with my debug prints for msg, you can see > crmd cores and re-registers with lrmd after each bus error, then it's > aligned good 1 time, then cores again after that, over and over.... 4 byte > aligned 1 time then the next it's not. > > root@serva:/var/lib/heartbeat/cores/hacluster# grep msg /tmp/corosync.log > Dec 15 20:19:13 corosync [TOTEM ] downcheck (1000 ms) fail to recv const (50 > msgs) > Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: msg %p:0x40b26000 > Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: (char*)msg:+ > Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: > msg->is_compressed:0 > Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: Test alignment of > ais_data_len(msg):211 > Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: Test alignment of > msg->size():211 > Dec 15 20:19:15 serva cib: [24692]: debug: ais_dispatch: Test alignment of > msg->compressed_size():0 > Dec 15 20:19:16 serva lrmd: [24693]: debug: on_msg_register:client crmd > [24696] registered > Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: msg %p:0x40b68000 > Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: (char*)msg:+ > Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: > msg->is_compressed:0 > Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: Test alignment of > ais_data_len(msg):211 > Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: Test alignment of > msg->size():211 > Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: Test alignment of > msg->compressed_size():0 > Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: msg %p:0x40b6832b > Dec 15 20:19:16 serva crmd: [24696]: debug: ais_dispatch: (char*)msg: > Dec 15 20:19:17 serva lrmd: [24693]: debug: on_msg_register:client crmd > [24702] registered > Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: msg %p:0x40b68000 > Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: (char*)msg:+ > Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: > msg->is_compressed:0 > Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: Test alignment of > ais_data_len(msg):211 > Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: Test alignment of > msg->size():211 > Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: Test alignment of > msg->compressed_size():0 > Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: msg %p:0x40b6832b > Dec 15 20:19:17 serva crmd: [24702]: debug: ais_dispatch: (char*)msg: > Dec 15 20:19:18 serva lrmd: [24693]: debug: on_msg_register:client crmd > [24704] registered > Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: msg %p:0x40b68000 > Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: (char*)msg:+ > Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: > msg->is_compressed:0 > Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: Test alignment of > ais_data_len(msg):211 > Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: Test alignment of > msg->size():211 > Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: Test alignment of > msg->compressed_size():0 > Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: msg %p:0x40b6832b > Dec 15 20:19:18 serva crmd: [24704]: debug: ais_dispatch: (char*)msg: > Dec 15 20:19:19 serva lrmd: [24693]: debug: on_msg_register:client crmd > [24706] registered > Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: msg %p:0x40b68000 > Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: (char*)msg:+ > Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: > msg->is_compressed:0 > Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: Test alignment of > ais_data_len(msg):211 > Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: Test alignment of > msg->size():211 > Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: Test alignment of > msg->compressed_size():0 > Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: msg %p:0x40b6832b > Dec 15 20:19:19 serva crmd: [24706]: debug: ais_dispatch: (char*)msg: > Dec 15 20:19:20 serva lrmd: [24693]: debug: on_msg_register:client crmd > [24708] registered > Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: msg %p:0x40b68000 > Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: (char*)msg:+ > Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: > msg->is_compressed:0 > Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: Test alignment of > ais_data_len(msg):211 > Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: Test alignment of > msg->size():211 > Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: Test alignment of > msg->compressed_size():0 > Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: msg %p:0x40b6832b > Dec 15 20:19:20 serva crmd: [24708]: debug: ais_dispatch: (char*)msg: > Dec 15 20:19:21 serva lrmd: [24693]: debug: on_msg_register:client crmd > [24710] registered > Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: msg %p:0x40b68000 > Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: (char*)msg:+ > Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: > msg->is_compressed:0 > Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: Test alignment of > ais_data_len(msg):211 > Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: Test alignment of > msg->size():211 > Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: Test alignment of > msg->compressed_size():0 > Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: msg %p:0x40b6832b > Dec 15 20:19:21 serva crmd: [24710]: debug: ais_dispatch: (char*)msg: > Dec 15 20:19:22 serva lrmd: [24693]: debug: on_msg_register:client crmd > [24712] registered > Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: msg %p:0x40b68000 > Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: (char*)msg:+ > Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: > msg->is_compressed:0 > Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: Test alignment of > ais_data_len(msg):211 > Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: Test alignment of > msg->size():211 > Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: Test alignment of > msg->compressed_size():0 > Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: msg %p:0x40b6832b > Dec 15 20:19:22 serva crmd: [24712]: debug: ais_dispatch: (char*)msg: > Dec 15 20:19:23 serva lrmd: [24693]: debug: on_msg_register:client crmd > [24715] registered > Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: msg %p:0x40b68000 > Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: (char*)msg:+ > Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: > msg->is_compressed:0 > Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: Test alignment of > ais_data_len(msg):211 > Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: Test alignment of > msg->size():211 > Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: Test alignment of > msg->compressed_size():0 > Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: msg %p:0x40b6832b > Dec 15 20:19:23 serva crmd: [24715]: debug: ais_dispatch: (char*)msg: > Dec 15 20:19:24 serva lrmd: [24693]: debug: on_msg_register:client crmd > [24717] registered > Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: msg %p:0x40b68000 > Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: (char*)msg:+ > Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: > msg->is_compressed:0 > Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: Test alignment of > ais_data_len(msg):211 > Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: Test alignment of > msg->size():211 > Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: Test alignment of > msg->compressed_size():0 > Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: msg %p:0x40b6832b > Dec 15 20:19:24 serva crmd: [24717]: debug: ais_dispatch: (char*)msg: > Dec 15 20:19:25 serva lrmd: [24693]: debug: on_msg_register:client crmd > [24720] registered > Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: msg %p:0x40b68000 > Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: (char*)msg:+ > Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: > msg->is_compressed:0 > Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: Test alignment of > msg->is_compressed:0 > Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: Test alignment of > ais_data_len(msg):211 > Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: Test alignment of > msg->size():211 > Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: Test alignment of > msg->compressed_size():0 > Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: msg %p:0x40b6832b > Dec 15 20:19:25 serva crmd: [24720]: debug: ais_dispatch: (char*)msg: > Dec 15 20:19:26 serva lrmd: [24693]: debug: on_msg_register:client crmd > [24722] registered > Dec 15 20:19:26 serva crmd: [24722]: debug: ais_dispatch: msg %p:0x40b68000 > Dec 15 20:19:26 serva crmd: [24722]: debug: ais_dispatch: (char*)msg:+ > root@serva:/var/lib/heartbeat/cores/hacluster# > > > here's a short tail of dmesg (which is the "warn" that you can configure > /proc/cpu/alignment for) that shows the various PIDs of crmd whenthey bus > error. > Notice the address 0c40b6834b is the same as my gdb ouptut above shows for > print &msg->is_compressed. > > [728065.861236] Alignment trap: crmd (18435) PC=0x40105008 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > [728066.902288] Alignment trap: crmd (18438) PC=0x40105008 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > [728067.944698] Alignment trap: crmd (18440) PC=0x40105008 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > [735468.169460] Alignment trap: crmd (24696) PC=0x40105048 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > [735469.296659] Alignment trap: crmd (24702) PC=0x40105048 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > [735470.337662] Alignment trap: crmd (24704) PC=0x40105048 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > [735471.403065] Alignment trap: crmd (24706) PC=0x40105048 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > [735472.395324] Alignment trap: crmd (24708) PC=0x40105048 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > [735473.426591] Alignment trap: crmd (24710) PC=0x40105048 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > [735474.462234] Alignment trap: crmd (24712) PC=0x40105048 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > [735475.505742] Alignment trap: crmd (24715) PC=0x40105048 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > [735476.542277] Alignment trap: crmd (24717) PC=0x40105048 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > [735477.587241] Alignment trap: crmd (24720) PC=0x40105048 Instr=0xe5933020 > Address=0x40b6834b FSR 0x001 > root@serva:/var/lib/heartbeat/cores/hacluster# > > > Sorry for the massive cut and paste in my first email to the list, I just > wanted to throw in all the details I've got and hope that someone can > enlighten me further and/or tell me what I missed. > > Thanks, > Greg > > oh yeah this should cover details on the arch/build some on the arch/build: > > root@serva:/var/lib/heartbeat/cores/hacluster# uname -a > Linux serva 2.6.32-5-kirkwood #1 Fri Nov 26 07:01:06 UTC 2010 armv5tel > GNU/Linux > > > root@serva:/var/lib/heartbeat/cores/hacluster# readelf -h > /usr/lib/heartbeat/crmd > ELF Header: > Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 > Class: ELF32 > Data: 2's complement, little endian > Version: 1 (current) > OS/ABI: UNIX - System V > ABI Version: 0 > Type: EXEC (Executable file) > Machine: ARM > Version: 0x1 > Entry point address: 0xc1b4 > Start of program headers: 52 (bytes into file) > Start of section headers: 4217016 (bytes into file) > Flags: 0x5000002, has entry point, Version5 > EABI > Size of this header: 52 (bytes) > Size of program headers: 32 (bytes) > Number of program headers: 8 > Size of section headers: 40 (bytes) > Number of section headers: 41 > Section header string table index: 38 > root@serva:/var/lib/heartbeat/cores/hacluster# > > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: > http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker > _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker