Hi!
I ran into the issue of ocfs2_controld.pcmk consuming vast CPU again -
twice, actually. The most recent happenstance was after a multi-node
failure. One node stayed alive, two nodes had to be rebooted. After
the reboots, one of the two came back without issue, and was able to
mount the OCFS2 stores. The second node exhibited high-cpu usage on the
ocfs2_controld.pcmk process, and could not mount the OCFS2 stores. The
logs were being voraciously filled with the following message:
ocfs2_controld: Unable to open checkpoint "ocfs2:controld": Object
does not exist
This message was being output so frequently that syslogd was starting to
rate-limit it. I suspect this accounts for the high CPU usage. After
restarting the troubled node several times, I found the solution was to
order the OCFS2/DLM resource group to stop, cluster-wide, and then
restart it. Normal behavior followed. (In a prior post to the list, I
referenced hard-killing the ocfs2_controld.pcmk process. This was a
more graceful shutdown.)
Attached are two strace outputs. I'm sorry I'm not very familiar with
strace, so the value of these files may be questionable. If there is
anything else I can provide the next time this happens, I'd be happy to
do so! The log-f.txt file was generated with the -f option, and the
log-fc.txt file was generated with -f -c.
Here also is a snippet from the syslog, during the cluster-wide shutdown
of the OCFS2/DLM group:
May 14 15:22:13 gw05 ocfs2_controld: Unable to open checkpoint
"ocfs2:controld": Object does not exist
May 14 15:22:14 ocfs2_controld: last message repeated 199 times
May 14 15:22:15 gw05 o2cb[4134]: INFO: Stopping ocfs2_controld.pcmk
May 14 15:22:16 gw05 dlm_controld.pcmk: [3411]: notice:
terminate_ais_connection: Disconnecting from AIS
May 14 15:22:16 gw05 lrmd: [2993]: info: RA output:
(p_dlm:2:stop:stderr) dlm_controld.pcmk: no process found
May 14 15:22:19 gw05 ocfs2_controld: Unable to open checkpoint
"ocfs2:controld": Object does not exist
May 14 15:22:20 ocfs2_controld: last message repeated 199 times
May 14 15:22:25 gw05 ocfs2_controld: Unable to open checkpoint
"ocfs2:controld": Object does not exist
May 14 15:22:26 ocfs2_controld: last message repeated 199 times
May 14 15:22:31 gw05 ocfs2_controld: Unable to open checkpoint
"ocfs2:controld": Object does not exist
May 14 15:22:32 ocfs2_controld: last message repeated 199 times
May 14 15:22:37 gw05 ocfs2_controld: Unable to open checkpoint
"ocfs2:controld": Object does not exist
May 14 15:22:38 ocfs2_controld: last message repeated 199 times
One other interesting bit of log (well, to me), was this bit that
occurred when I tried to manually mount the OCFS2 store on the afflicted
server:
mount.ocfs2: Unable to access cluster service while trying to join
the group
One other note - I discovered I had not specified a monitor for either
the pacemaker:o2cb or the pacemaker:controld RA. Could that have
possibly triggered this issue?
--
Sincerely,
Matthew O'Connor
-----------------------------------------------------------------
Sr. Software Engineer
PGP/GPG Key: 0x55F981C4
Fingerprint: E5DC A0F8 5A40 E4DA 2CE6 B5A2 014C 2CBF 55F9 81C4
Engineering and Computer Simulations, Inc.
11825 High Tech Ave Suite 250
Orlando, FL 32817
Tel: 407-823-9991 x315
Fax: 407-823-8299
Email: m...@ecsorl.com
Web: www.ecsorl.com
-----------------------------------------------------------------
CONFIDENTIAL NOTICE: The information contained in this electronic
message is legally privileged, confidential and exempt from disclosure
under applicable law. It is intended only for the use of the individual
or entity named above. If the reader of this message is not the intended
recipient, you are hereby notified that any dissemination, distribution
or copying of this message is strictly prohibited. If you have received
this communication in error, please notify the sender immediately by
return e-mail and delete the original message and any copies of it from
your computer system. Thank you.
Process 8940 attached
Process 8940 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 0.000159 159 1 clone
0.00 0.000000 0 31 read
0.00 0.000000 0 1 write
0.00 0.000000 0 38 5 open
0.00 0.000000 0 34 close
0.00 0.000000 0 4 3 stat
0.00 0.000000 0 30 fstat
0.00 0.000000 0 77 mmap
0.00 0.000000 0 58 mprotect
0.00 0.000000 0 2 munmap
0.00 0.000000 0 3 brk
0.00 0.000000 0 2 rt_sigaction
0.00 0.000000 0 1 rt_sigprocmask
0.00 0.000000 0 29 29 access
0.00 0.000000 0 3 dup2
0.00 0.000000 0 1 execve
0.00 0.000000 0 1 1 fcntl
0.00 0.000000 0 1 chdir
0.00 0.000000 0 1 umask
0.00 0.000000 0 1 getrlimit
0.00 0.000000 0 1 setsid
0.00 0.000000 0 1 statfs
0.00 0.000000 0 1 arch_prctl
0.00 0.000000 0 2 1 futex
0.00 0.000000 0 1 set_tid_address
0.00 0.000000 0 1 set_robust_list
------ ----------- ----------- --------- --------- ----------------
100.00 0.000159 326 39 total
execve("/usr/sbin/ocfs2_controld.pcmk", ["ocfs2_controld.pcmk"], [/* 14 vars
*/]) = 0
brk(0) = 0x184f000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf7909000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=27111, ...}) = 0
mmap(NULL, 27111, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f2cf7902000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcom_err.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\23\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14544, ...}) = 0
mmap(NULL, 2109864, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf74e7000
mprotect(0x7f2cf74ea000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf76e9000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f2cf76e9000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libSaCkpt.so.3", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\21\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=35176, ...}) = 0
mmap(NULL, 2130352, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf72de000
mprotect(0x7f2cf72e6000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf74e5000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f2cf74e5000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libcpg.so.4", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\21\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=26928, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf7901000
mmap(NULL, 2122104, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf70d7000
mprotect(0x7f2cf70dd000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf72dc000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f2cf72dc000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libdlmcontrol.so.3", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\v\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=10232, ...}) = 0
mmap(NULL, 2105480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf6ed4000
mprotect(0x7f2cf6ed6000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf70d5000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f2cf70d5000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libcrmcluster.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\2203\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=64120, ...}) = 0
mmap(NULL, 2159736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf6cc4000
mprotect(0x7f2cf6cd3000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf6ed2000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe000) = 0x7f2cf6ed2000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libcrmcommon.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360w\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=204912, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf7900000
mmap(NULL, 2300864, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf6a92000
mprotect(0x7f2cf6ac3000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf6cc2000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x30000) = 0x7f2cf6cc2000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \24\2\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1685816, ...}) = 0
mmap(NULL, 3801960, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf66f1000
mprotect(0x7f2cf6888000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf6a87000, 20480, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x196000) = 0x7f2cf6a87000
mmap(0x7f2cf6a8c000, 21352, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f2cf6a8c000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0Pl\0\0\0\0\0\0"..., 832)
= 832
fstat(3, {st_mode=S_IFREG|0755, st_size=135500, ...}) = 0
mmap(NULL, 2212920, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf64d4000
mprotect(0x7f2cf64ec000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf66eb000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f2cf66eb000
mmap(0x7f2cf66ed000, 13368, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f2cf66ed000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libcoroipcc.so.4", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\23\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=22872, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf78ff000
mmap(NULL, 2118040, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf62ce000
mprotect(0x7f2cf62d3000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf64d2000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f2cf64d2000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libhbclient.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\35\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=43792, ...}) = 0
mmap(NULL, 2145088, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf60c2000
mprotect(0x7f2cf60cc000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf62cb000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) = 0x7f2cf62cb000
mmap(0x7f2cf62cd000, 2880, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f2cf62cd000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libquorum.so.4", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\v\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14480, ...}) = 0
mmap(NULL, 2109656, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf5ebe000
mprotect(0x7f2cf5ec1000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf60c0000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f2cf60c0000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libplumb.so.2", O_RDONLY) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\216\0\0\0\0\0\0"..., 832) =
832
fstat(3, {st_mode=S_IFREG|0644, st_size=223952, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf78fe000
mmap(NULL, 2321832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf5c87000
mprotect(0x7f2cf5cbc000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf5ebb000, 12288, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x34000) = 0x7f2cf5ebb000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libbz2.so.1.0", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\26\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=66784, ...}) = 0
mmap(NULL, 2162064, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf5a77000
mprotect(0x7f2cf5a86000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf5c85000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe000) = 0x7f2cf5c85000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxml2.so.2", O_RDONLY) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\303\2\0\0\0\0\0"..., 832) =
832
fstat(3, {st_mode=S_IFREG|0644, st_size=1417304, ...}) = 0
mmap(NULL, 3517784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf571c000
mprotect(0x7f2cf586d000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf5a6c000, 40960, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x150000) = 0x7f2cf5a6c000
mmap(0x7f2cf5a76000, 3416, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f2cf5a76000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libglib-2.0.so.0", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300w\1\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=1003848, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf78fd000
mmap(NULL, 3101496, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf5426000
mprotect(0x7f2cf551a000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf5719000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf3000) = 0x7f2cf5719000
mmap(0x7f2cf571b000, 824, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f2cf571b000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libgnutls.so.26", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360M\1\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=688328, ...}) = 0
mmap(NULL, 2783560, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf517e000
mprotect(0x7f2cf521f000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf541e000, 32768, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa0000) = 0x7f2cf541e000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxslt.so.1", O_RDONLY) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\222\0\0\0\0\0\0"..., 832) =
832
fstat(3, {st_mode=S_IFREG|0644, st_size=241504, ...}) = 0
mmap(NULL, 2336792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf4f43000
mprotect(0x7f2cf4f7d000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf517c000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x39000) = 0x7f2cf517c000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf78fc000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libpils.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\30\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=39512, ...}) = 0
mmap(NULL, 2135008, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf4d39000
mprotect(0x7f2cf4d42000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf4f41000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f2cf4f41000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libuuid.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\24\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=18888, ...}) = 0
mmap(NULL, 2113936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf4b34000
mprotect(0x7f2cf4b38000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf4d37000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f2cf4d37000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\r\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14768, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf78fb000
mmap(NULL, 2109704, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf4930000
mprotect(0x7f2cf4932000, 2097152, PROT_NONE) = 0
mmap(0x7f2cf4b32000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f2cf4b32000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libz.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P \0\0\0\0\0\0"..., 832)
= 832
fstat(3, {st_mode=S_IFREG|0644, st_size=96816, ...}) = 0
mmap(NULL, 2191920, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf4718000
mprotect(0x7f2cf472f000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf492e000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f2cf492e000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360>\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=538928, ...}) = 0
mmap(NULL, 2633960, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf4494000
mprotect(0x7f2cf4517000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf4716000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x82000) = 0x7f2cf4716000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\25\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=243800, ...}) = 0
mmap(NULL, 2338984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf4258000
mprotect(0x7f2cf4293000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf4492000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3a000) = 0x7f2cf4492000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200!\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=31752, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf78fa000
mmap(NULL, 2129016, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf4050000
mprotect(0x7f2cf4057000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf4256000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f2cf4256000
close(3) = 0
open("/usr/lib/x86_64-linux-gnu/tls/x86_64/libtasn1.so.3", O_RDONLY) = -1
ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/tls/x86_64", 0x7fffe3b1d300) = -1 ENOENT (No
such file or directory)
open("/usr/lib/x86_64-linux-gnu/tls/libtasn1.so.3", O_RDONLY) = -1 ENOENT (No
such file or directory)
stat("/usr/lib/x86_64-linux-gnu/tls", 0x7fffe3b1d300) = -1 ENOENT (No such file
or directory)
open("/usr/lib/x86_64-linux-gnu/x86_64/libtasn1.so.3", O_RDONLY) = -1 ENOENT
(No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/x86_64", 0x7fffe3b1d300) = -1 ENOENT (No such
file or directory)
open("/usr/lib/x86_64-linux-gnu/libtasn1.so.3", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\31\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=67864, ...}) = 0
mmap(NULL, 2163496, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf3e3f000
mprotect(0x7f2cf3e4f000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf404e000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf000) = 0x7f2cf404e000
close(3) = 0
open("/usr/lib/x86_64-linux-gnu/libgcrypt.so.11", O_RDONLY) = -1 ENOENT (No
such file or directory)
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libgcrypt.so.11", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0~\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=516056, ...}) = 0
mmap(NULL, 2612480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf3bc1000
mprotect(0x7f2cf3c3b000, 2097152, PROT_NONE) = 0
mmap(0x7f2cf3e3b000, 16384, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7a000) = 0x7f2cf3e3b000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libltdl.so.7", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\36\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=39184, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf78f9000
mmap(NULL, 2134448, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf39b7000
mprotect(0x7f2cf39bf000, 2097152, PROT_NONE) = 0
mmap(0x7f2cf3bbf000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f2cf3bbf000
close(3) = 0
open("/usr/lib/x86_64-linux-gnu/libgpg-error.so.0", O_RDONLY) = -1 ENOENT (No
such file or directory)
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libgpg-error.so.0", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\10\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14352, ...}) = 0
mmap(NULL, 2109528, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f2cf37b3000
mprotect(0x7f2cf37b6000, 2093056, PROT_NONE) = 0
mmap(0x7f2cf39b5000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f2cf39b5000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf78f8000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf78f7000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf78f6000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf78f4000
arch_prctl(ARCH_SET_FS, 0x7f2cf78f4780) = 0
mprotect(0x7f2cf39b5000, 4096, PROT_READ) = 0
mprotect(0x7f2cf3bbf000, 4096, PROT_READ) = 0
mprotect(0x7f2cf3e3b000, 4096, PROT_READ) = 0
mprotect(0x7f2cf404e000, 4096, PROT_READ) = 0
mprotect(0x7f2cf4256000, 4096, PROT_READ) = 0
mprotect(0x7f2cf4492000, 4096, PROT_READ) = 0
mprotect(0x7f2cf4716000, 4096, PROT_READ) = 0
mprotect(0x7f2cf492e000, 4096, PROT_READ) = 0
mprotect(0x7f2cf4b32000, 4096, PROT_READ) = 0
mprotect(0x7f2cf4d37000, 4096, PROT_READ) = 0
mprotect(0x7f2cf4f41000, 4096, PROT_READ) = 0
mprotect(0x7f2cf517c000, 4096, PROT_READ) = 0
mprotect(0x7f2cf541e000, 28672, PROT_READ) = 0
mprotect(0x7f2cf5719000, 4096, PROT_READ) = 0
mprotect(0x7f2cf5a6c000, 32768, PROT_READ) = 0
mprotect(0x7f2cf5c85000, 4096, PROT_READ) = 0
mprotect(0x7f2cf5ebb000, 4096, PROT_READ) = 0
mprotect(0x7f2cf60c0000, 4096, PROT_READ) = 0
mprotect(0x7f2cf62cb000, 4096, PROT_READ) = 0
mprotect(0x7f2cf64d2000, 4096, PROT_READ) = 0
mprotect(0x7f2cf66eb000, 4096, PROT_READ) = 0
mprotect(0x7f2cf6a87000, 16384, PROT_READ) = 0
mprotect(0x7f2cf6cc2000, 4096, PROT_READ) = 0
mprotect(0x7f2cf6ed2000, 4096, PROT_READ) = 0
mprotect(0x7f2cf70d5000, 4096, PROT_READ) = 0
mprotect(0x7f2cf72dc000, 4096, PROT_READ) = 0
mprotect(0x7f2cf74e5000, 4096, PROT_READ) = 0
mprotect(0x7f2cf76e9000, 4096, PROT_READ) = 0
mprotect(0x61a000, 4096, PROT_READ) = 0
mprotect(0x7f2cf790b000, 4096, PROT_READ) = 0
munmap(0x7f2cf7902000, 27111) = 0
set_tid_address(0x7f2cf78f4a50) = 8876
set_robust_list(0x7f2cf78f4a60, 0x18) = 0
futex(0x7fffe3b1f04c, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fffe3b1f04c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL,
7f2cf78f4780) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigaction(SIGRTMIN, {0x7f2cf64da6c0, [], SA_RESTORER|SA_SIGINFO,
0x7f2cf64e4060}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7f2cf64da750, [], SA_RESTORER|SA_RESTART|SA_SIGINFO,
0x7f2cf64e4060}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
brk(0) = 0x184f000
brk(0x1870000) = 0x1870000
open("/sys/fs/ocfs2/cluster_stack", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=4096, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f2cf7908000
read(3, "pcmk\n", 4096) = 5
close(3) = 0
munmap(0x7f2cf7908000, 4096) = 0
open("/sys/fs/o2cb/interface_revision", O_RDONLY) = 3
read(3, "5\n", 15) = 2
read(3, "", 13) = 0
close(3) = 0
stat("/sys/kernel/config", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
statfs("/sys/kernel/config", {f_type=0x62656570, f_bsize=4096, f_blocks=0,
f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255,
f_frsize=4096}) = 0
clone(Process 8877 attached
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7f2cf78f4a50) = 8877
[pid 8877] setsid() = 8877
[pid 8877] chdir("/") = 0
[pid 8877] umask(0) = 022
[pid 8877] close(0) = 0
[pid 8877] close(1 <unfinished ...>
[pid 8876] exit_group(0) = ?
[pid 8877] <... close resumed> ) = 0
[pid 8877] close(2) = 0
[pid 8877] open("/dev/null", O_RDWR) = 0
[pid 8877] dup2(0, 0) = 0
[pid 8877] dup2(0, 1) = 1
[pid 8877] dup2(0, 2) = 2
[pid 8877] open("/var/run/ocfs2_controld.pid", O_WRONLY|O_CREAT, 0644) = 3
[pid 8877] fcntl(3, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0})
= -1 EAGAIN (Resource temporarily unavailable)
[pid 8877] write(2, "ocfs2_controld is already runnin"..., 34) = 34
exit_group(1) = ?
Process 8877 detached
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
